Redesign of DataSet API

[hughleat]

Redesign the Dataset class to not depend on tar balls and particular data structures.

Currently the datasets are hard coded as tarballs - https://github.com/facebookresearch/CompilerGym/blob/development/compiler_gym/envs/llvm/datasets.py
these later get unpacked into a particular format where the directory structure is very important.

This means that we have to curate them. E.g. we can't pull benchmarks from Anghabench directly, we have to host a tarball somewhere.

Also, if we had random program generators, e.g. CSmith, CLGen, we couldn't really work with them.

Instead, add methods to the Dataset class to install and extract:

class Dataset:
  ...
  def name() -> String
  def install(path_to_use: Path)    # e.g. ~/.compiler_gym/datasets/<name>
  def benchmark_ids() -> Iterable[Any] # Possibly lazy list of benchmark names
  def benchmarks() -> Iterable[Benchmark] # Possibly lazy list of benchmarks
  def benchmark(id: Any) -> Benchmark

class TarballDataset(Dataset):
  def __init__(name: String, url: Stirng, etc..)
  def install(p): download_unpack(p, url)
  def benchmark_ids(): get_filenames_from_install_dir()
  ...

class Anghabench(Dataset):
  ...
  def name(): return "anghabench"
  def install(p): git_clone_into(p, "https://github.com/brenocfg/AnghaBench.git")
  
class CSmith(Dataset):
  def install(p): install_csmith_binary(p)
  def benchmark_ids(): range_all_ints
  def benchmark(id: int): csmith_for_seed(id)

At the same time, something like this would free datasets to have their own directory structure. This might be useful when considering input data for correctness and performance. Sometimes multiple benchmarks will share things.

At init time, the gym could look in the dataset dir (e.g. ~/.compiler_gym/datasets or [ENV COMPILER_GYM_DIR]). Any python scripts in there could be run to register datasets.
Programmatically, people could register their own datasets, outside of that common mechanism.
A command line tool, install_dataset url, could fetch a script from the url, drop it in the dataset dir, then run install.

[ChrisCummins]

Thanks @hughleat, I think that a python-side dataset API makes a lot of sense, and I like the class hierarchy you're proposing. I think a prerequisite for this is to figure out the role of the backend service in managing datasets as at the moment, the frontend python code and backend service have slightly jumbled and overlapping roles. We could shift the responsibility of managing benchmarks from the service to the frontend by:

1. Removing the list-of-all-benchmarks from the BenchmarkFactory in the backend service.
2. Remove the benchmark management methods from the service API.
3. Make the benchmark a compulsory parameter for starting episodes.

Cheers,
Chris

[ChrisCummins]

I think it would be a good idea also to wrap the Benchmark proto with a python class that can add extra functionality like the ability to validate benchmark behavior etc. To begin with, something simple like:

class Benchmark(object):
  def __init__(self, proto: BenchmarkProto)
  def sha1(self) -> bytes  # Name for caching benchmark+any other attributes
  def program_data(self) -> BenchmarkProto  # The data that the service needs
  def program_data_sha1(self) -> bytes  # Used for caching benchmarks on service-side
  def is_validatable(self) -> bool
  def validation_callbacks(self) -> List[Callback[[CompilerEnv], Optional[str]]  # Run any ad-hoc validation, e.g. difftest, valgrind, etc