Home

Steven Selph's Scraper

This scraper downloads images and metadata primarily from online databases like thegamesdb for use with EmulationStation It uses hashes of roms to provide the most accurate results.

Supported Systems:

A complete list is HERE.

Platforms Supported:

• Windows
• Mac
• Linux
• Raspberry Pi

Note that if you are using RetroPie, the scraper can easily be used and installed through the RetroPie Setup Script - you can see the documentation for it HERE

Installing:

You can install compiled binaries from the releases page HERE

you can also compile from source as described in the README

Usage

$ cd <rom directory>
$ scraper

ROMs will be scanned and a gamelist.xml file will be created. All images will be placed inside the images folder.

All Systems

$ scraper -scrape_all -thumb_only

MAME/FBA

The scraper now supports MAME/FBA but using file names instead of hashing. Since it uses a different DB and lookup method, several of the command line flags no longer apply. When the -mame flag is used it disables all other databases and the mamedb only has the one size of image so the flags about thumbnails, gdb, ovgdb, etc don't do anything.

$ scraper -mame

You can choose your preference of image type with the mame_img flag. If you prefer marquees but want to fallback to titles then snapshots you can do:

$ scraper -mame -mame_img "m,t,s"

Command Line Flags

There are several command flags you can pass. To see a full list use -help

$ scraper -help

Full List:

  -add_not_found=false: If true, add roms that are not found as an empty gamelist entry.
  -append=false: If the gamelist file already exist skip files that are already listed and only append new files.
  -download_images=true: If false, don't download any images, instead see if the expected file is stored locally already.
  -extra_ext="": Comma separated list of extensions to also include in the scraper.
  -gdb_img="b": Comma seperated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo.
  -hash_file="": The `file` containing hash information.
  -image_dir="images": The `directory` to place downloaded images to locally.
  -image_path="images": The `path` to use for images in gamelist.xml.
  -image_suffix="-image": The `suffix` added after rom name when creating image files.
  -img_format="jpg": `jpg or png`, the format to write the images.
  -img_workers=0: Use `N` worker threads to process images. If 0, then use the same value as workers.
  -mame=false: If true we want to run in MAME mode.
  -mame_img="s,t,m,c": Comma separated order to prefer images, s=snap, t=title, m=marquee, c=cabniet.
  -max_width=400: The max `width` of images. Larger images will be resized.
  -missing="": The `file` where information about ROMs that weren't scraped is added.
  -nested_img_dir=false: Use a nested img directory structure that matches rom structure.
  -no_thumb=false: Don't add thumbnails to the gamelist.
  -output_file="gamelist.xml": The XML `file` to output to.
  -overview_len=0: If set it will truncate the overview of roms to `N` characters + ellipsis.
  -refresh=false: Information will be attempted to be downloaded again but won't remove roms that are not scraped.
  -retries=2: Retry a rom `N` times on an error.
  -rom_dir=".": The `directory` containing the roms file to process.
  -rom_path=".": The `path` to use for roms in gamelist.xml.
  -scrape_all=false: If true, scrape all systems listed in es_systems.cfg. All dir/path flags will be ignored.
  -skip_check=false: Skip the check if thegamesdb.net is up.
  -start_pprof=false: If true, start the pprof service used to profile the application.
  -strip_unicode=true: If true, remove all non-ascii characters.
  -thumb_only=false: Download the thumbnail for both the image and thumb (faster).
  -thumb_suffix="-thumb": The `suffix` added after rom name when creating thumb files.
  -use_filename=false: If true, use the filename minus the extension as the game title in xml.
  -use_gdb=true: Use the hash.csv and theGamesDB metadata.
  -use_nointro_name=true: Use the name in the No-Intro DB instead of the one in the GDB.
  -use_ovgdb=false: Use the OpenVGDB if the hash isn't in hash.csv.
  -version=false: Print the release version and exit.
  -workers=1: Use `N` worker threads to process roms.