ROMDownloader

The ROMDownloader uses rclone to sync files from a remote (or local) location to be used in other steps. For instructions on how to set up rclone, see their documentation at Rclone.

If using the default filter_then_download mode, ROMDownloader will use rclone copy to copy files and then delete extraneous files from the raw directory at the end. For download_then_filter it will use rclone sync to do a full sync, and then automatically delete files afterwards.

ROMDownloader also has optional Discord integration, which will print out files downloaded or deleted at the end of each run.

For more details on the ROMDownloader arguments, see the config file documentation.

API

class romsearch.ROMDownloader(platform=None, config_file=None, config=None, platform_config=None, rclone_method='sync', copy_files=None, subchannel_dict=None, logger=None, include_filter_wildcard=True, log_line_sep='=', log_line_length=100)

Downloader tool via rclone

This works per-platform, so must be specified here.

For this, we can either sync an entire directory (rclone_method=’sync’), or copy individual files (rclone_method=’copy’). If the method is copy, then copy_files must be set as a list

Parameters:

• platform (str, optional) – Platform name. Defaults to None, which will throw a ValueError

• config_file (str, optional) – Configuration file. Defaults to None

• config (dict, optional) – Configuration dictionary. Defaults to None

• platform_config (dict, optional) – Platform configuration dictionary. Defaults to None

• rclone_method (str, optional) – Should be one of ‘sync’ or ‘copy’. Defaults to ‘sync’

• copy_files (list, optional) – Must be set if rclone_method is ‘copy’. Determines the filenames to copy over. Defaults to None

• subchannel_dict (dict, optional) – Dictionary of subchannel files. Defaults to None

• logger (logging.Logger, optional) – Logger instance. Defaults to None

• include_filter_wildcard (bool, optional) – If set, will include wildcards in rclone filters. Defaults to True.

• log_line_length (int, optional) – Line length of log. Defaults to 100

log_items_added_deleted(items_added, items_deleted)

Log a summary of items added and removed

Parameters:

• items_added (list) – list of items added

• items_deleted (list) – list of items deleted

post_to_discord(items_added, items_deleted, name, max_per_message=10)

Create a discord post summarising files added and removed

Parameters:

• items_added (list) – list of files added

• items_deleted (list) – list of files deleted

• name (string) – Name of the post title

• max_per_message (int, optional) – Maximum number of items per post. Defaults to 10.

rclone_copy(remote_dir, out_dir=None, subchannel=None, subchannel_original_dir=None, max_retries=5)

Use rclone to copy files one-by-one

Parameters:

• remote_dir – rclone remote path

• out_dir – directory to download to

• subchannel – subchannel to copy to

• subchannel_original_dir – Original directory for subchannel-related files. Defaults to None

• max_retries – maximum number of retries

rclone_download(remote_dir, out_dir=None, subchannel=None, subchannel_original_dir=None, max_retries=5)

Download from rclone, either via sync or copy

Parameters:

• remote_dir – rclone remote path

• out_dir – directory to download to

• subchannel – subchannel to download. Defaults to None

• subchannel_original_dir – Original directory for subchannel-related files. Defaults to None

• max_retries – maximum number of retries

rclone_sync(remote_dir, out_dir=None, transfers=5, max_retries=5)

Use rclone to sync an entire directory

Parameters:

• remote_dir – rclone remote path

• out_dir – directory to download to

• transfers – number of simultaneous transfers

• max_retries – maximum number of retries

run()

Run Rclone downloader tool