Merge pull request #374 from ChrisCummins/release-v0.1.10

Release v0.1.10 (2021-09-08)

.bazelrc

@@ -77,6 +77,10 @@ build:ubsan --linkopt -lubsan
 # --config ci: Continuous Integration tool
 query:ci --noshow_progress
 query:ci --noshow_loading_progress
+fetch:ci --noshow_progress
+fetch:ci --noshow_loading_progress
+fetch:ci --show_task_finish
+fetch:ci --keep_going
 build:ci --noshow_progress
 build:ci --noshow_loading_progress
 build:ci --show_task_finish

.github/actions/install-build-dependencies/action.yml

@@ -1,5 +1,5 @@
 ---
-name: Install the Linux dependencies
+name: Install build dependencies
 description: Install build dependencies
 runs:
     using: composite

.github/workflows/ci.yaml

@@ -37,6 +37,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_TEST_OPTS: --config=ci --test_timeout=300,900,1800,7200
 
     install_test:
@@ -65,6 +66,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_BUILD_OPTS: --config=ci
 
             - name: Test

.github/workflows/coverage.yaml

@@ -29,6 +29,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_BUILD_OPTS: --config=ci
 
             - name: Test

.github/workflows/fuzz.yaml

@@ -39,6 +39,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_TEST_OPTS: --config=ci
 
             - name: Test

.github/workflows/release_test.yaml

@@ -42,3 +42,6 @@ jobs:
 
             - name: Test
               run: make install-test
+              env:
+                  CC: clang
+                  CXX: clang++

.github/workflows/sanitizers.yaml

@@ -29,6 +29,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_BUILD_OPTS: --config=ci
 
             - name: Build address sanitized LLVM compiler service
@@ -37,6 +38,7 @@ jobs:
                   CC: clang
                   CXX: clang++
                   BAZEL_OPTS: --batch
+                  BAZEL_FETCH_OPTS: --config=ci
                   BAZEL_BUILD_OPTS: --config=ci
 
             - name: Install pip package

CHANGELOG.md

@@ -1,3 +1,50 @@
+## Release 0.1.10 (2021-09-08)
+
+This release lays the foundation for several new exciting additions to
+CompilerGym:
+
+- [LLVM] Added experimental support for **optimizing for runtime** and **compile
+  time** ([#307](https://github.com/facebookresearch/CompilerGym/pull/307)).
+  This is still proof of concept and is not yet stable. For now, only the
+  `benchmark://cbench-v1` and `generator://csmith-v0` datasets are supported.
+- [CompilerGym Explorer] Started development of a **web frontend** for the
+  LLVM environments. The work-in-progress Flask API and React website can be
+  found in the `www` directory.
+- [New Backend API] Added a mechanism for sending arbitrary data payloads to the
+  compiler service backends
+  ([#313](https://github.com/facebookresearch/CompilerGym/pull/313)). This
+  allows ad-hoc parameters that do not conform to the usual action space to be
+  set for the duration of an episode. Add support for these parameters in the
+  backend by implementing the optional
+  [handle_session_parameter()](https://github.com/facebookresearch/CompilerGym/blob/63ee95a34157856ca21c392c49d35234e065fa8d/compiler_gym/service/compilation_session.py#L94-L112)
+  method, and then send parameters using the
+  [send_params()](https://github.com/facebookresearch/CompilerGym/blob/63ee95a34157856ca21c392c49d35234e065fa8d/compiler_gym/envs/compiler_env.py#L1317-L1338)
+  method.
+
+Other highlights of this release include:
+
+- [LLVM] The Csmith program generator is now shipped as part of the CompilerGym
+  binary release, removing the need to compile it locally
+  ([#348](https://github.com/facebookresearch/CompilerGym/pull/348)).
+- [LLVM] A new `ProgramlJson` observation space provides the JSON node-link data
+  of a ProGraML graph without parsing to a `nx.MultiDiGraph`
+  ([#332](https://github.com/facebookresearch/CompilerGym/pull/332)).
+- [LLVM] Added a leaderboard submission for a DQN agent
+  ([#292](https://github.com/facebookresearch/CompilerGym/pull/292), thanks
+  @phesse001!).
+- [Backend API Update] The `Reward.reset()` method now receives an observation
+  view that can be used to compute initial states
+  ([#341](https://github.com/facebookresearch/CompilerGym/pull/341), thanks
+  @bwasti!).
+- [Datasets API] The size of infinite datasets has been changed from
+  `float("inf")` to `0`
+  ([#347](https://github.com/facebookresearch/CompilerGym/pull/347)). This is a
+  compatibility fix for `__len__()` which requires integers values.
+- Prevent excessive growth of in-memory caches
+  ([#299](https://github.com/facebookresearch/CompilerGym/pull/299)).
+- Multiple compatibility fixes for `compiler_gym.wrappers`.
+- Numerous other bug fixes and improvements.
+
 ## Release 0.1.9 (2021-06-03)
 
 This release of CompilerGym focuses on **backend extensibility** and adds a

Makefile

@@ -88,6 +88,9 @@ Deployment
         Run the `make install-test` suite against the build artifact generated
         by `make bdist_wheel-linux`.
 
+	make www
+		Run a local instance of the web visualization service. See www/README.md
+		for details.
 
 Tidying up
 -----------
@@ -119,6 +122,7 @@ PYTHON ?= python3
 
 # Bazel build options.
 BAZEL_OPTS ?=
+BAZEL_FETCH_OPTS ?=
 BAZEL_BUILD_OPTS ?= -c opt
 BAZEL_TEST_OPTS ?=
 
@@ -154,7 +158,24 @@ DISTTOOLS_OUTS := dist build compiler_gym.egg-info
 
 BUILD_TARGET ?= //:package
 
-bazel-build:
+BAZEL_FETCH_RETRIES ?= 5
+
+# Run `bazel fetch` in a retry loop due to intermitent failures when fetching
+# remote archives in the CI environment.
+bazel-fetch:
+	@for i in $$(seq 1 $(BAZEL_FETCH_RETRIES)); do \
+		echo "$(BAZEL) $(BAZEL_OPTS) fetch $(BAZEL_FETCH_OPTS) $(BUILD_TARGET)"; \
+		if $(BAZEL) $(BAZEL_OPTS) fetch $(BAZEL_FETCH_OPTS) $(BUILD_TARGET) ; then \
+			break; \
+		else \
+			echo "bazel fetch attempt $$i of $(BAZEL_FETCH_RETRIES) failed" >&2; \
+		fi; \
+		if [ $$i -eq 10 ]; then \
+			false; \
+		fi; \
+	done
+
+bazel-build: bazel-fetch
 	$(BAZEL) $(BAZEL_OPTS) build $(BAZEL_BUILD_OPTS) $(BUILD_TARGET)
 
 bdist_wheel: bazel-build
@@ -178,8 +199,23 @@ bdist_wheel-linux-test:
 
 all: docs bdist_wheel bdist_wheel-linux
 
-.PHONY: bazel-build bdist_wheel bdist_wheel-linux bdist_wheel-linux-shell bdist_wheel-linux-test
+.PHONY: bazel-fetch bazel-build bdist_wheel bdist_wheel-linux bdist_wheel-linux-shell bdist_wheel-linux-test
 
+#################
+# Web interface #
+#################
+
+www: www-build
+	cd www && $(PYTHON) www.py
+
+www-build:
+	cd www/frontends/compiler_gym && npm install && npm run build
+
+www-image: www-build
+	cd www && docker build -t chriscummins/compiler_gym-www .
+	docker run -p 5000:5000 chriscummins/compiler_gym-www
+
+.PHONY: www www-build
 
 #################
 # Documentation #
@@ -237,10 +273,10 @@ TEST_TARGET ?=
 # Extra command line arguments for pytest.
 PYTEST_ARGS ?=
 
-test:
+test: bazel-fetch
 	$(BAZEL) $(BAZEL_OPTS) test $(BAZEL_TEST_OPTS) $(if $(TEST_TARGET),$(TEST_TARGET),//...)
 
-itest:
+itest: bazel-fetch
 	$(IBAZEL) $(BAZEL_OPTS) test $(BAZEL_TEST_OPTS) $(if $(TEST_TARGET),$(TEST_TARGET),//...)
 
 # Since we can't run compiler_gym from the project root we need to jump through

README.md

@@ -69,7 +69,9 @@ Install the latest CompilerGym release using:
 
     pip install -U compiler_gym
 
-See [INSTALL.md](INSTALL.md) for further details.
+See
+[INSTALL.md](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md)
+for further details.
 
 
 ## Usage
@@ -129,6 +131,7 @@ environment on the 23 benchmarks in the `cbench-v1` dataset.
 | Facebook | e-Greedy search (e=0.1) | [write-up](leaderboard/llvm_instcount/e_greedy/README.md), [results](leaderboard/llvm_instcount/e_greedy/results_e10.csv) | 2021-03 | 152.579s | 1.041× |
 | Jiadong Guo | Tabular Q (N=5000, H=10) | [write-up](leaderboard/llvm_instcount/tabular_q/README.md), [results](leaderboard/llvm_instcount/tabular_q/results-H10-N5000.csv) | 2021-04 | 2534.305 | 1.036× |
 | Facebook | Random search (t=10) | [write-up](leaderboard/llvm_instcount/random_search/README.md), [results](leaderboard/llvm_instcount/random_search/results_p125_t10.csv) | 2021-03 | **42.939s** | 1.031× |
+| Patrick Hesse | DQN (N=4000, H=10) | [write-up](leaderboard/llvm_instcount/dqn/README.md), [results](leaderboard/llvm_instcount/dqn/results-instcountnorm-H10-N4000.csv) | 2021-06 | 91.018s | 1.029× |
 | Jiadong Guo | Tabular Q (N=2000, H=5) | [write-up](leaderboard/llvm_instcount/tabular_q/README.md), [results](leaderboard/llvm_instcount/tabular_q/results-H5-N2000.csv) | 2021-04 | 694.105 | 0.988× |
 
 

VERSION

@@ -1 +1 @@
-0.1.9
+0.1.10

WORKSPACE

@@ -40,7 +40,7 @@ http_archive(
     urls = ["https://github.com/google/glog/archive/v0.4.0.tar.gz"],
 )
 
-# C++ subprocess management. https://github.com/arun11299/cpp-subprocess.git
+# C++ subprocess management. https://github.com/arun11299/cpp-subprocess
 
 http_archive(
     name = "subprocess",
@@ -207,9 +207,9 @@ filegroup(
 http_archive(
     name = "ctuning-ai",
     build_file = "//:compiler_gym/third_party/ctuning-ai.BUILD",
-    sha256 = "2e12233a3d898a96a2ce279123ee256bee357f649e3c579f974628b6c35ea53d",
-    strip_prefix = "ai-afaba3e2900523a0813693fd614a114b4a6f6729",
-    urls = ["https://github.com/ctuning/ai/archive/afaba3e2900523a0813693fd614a114b4a6f6729.tar.gz"],
+    sha256 = "a82c13733696c46b5201c614fcf7229c3a74a83ce485cab2fbf17309b7564f9c",
+    strip_prefix = "ck-mlops-406738ad6d1fb2c1da9daa2c09d26fccab4e0938",
+    urls = ["https://github.com/ChrisCummins/ck-mlops/archive/406738ad6d1fb2c1da9daa2c09d26fccab4e0938.tar.gz"],
 )
 
 # Datasets.
@@ -290,6 +290,17 @@ http_archive(
     urls = ["https://github.com/pytorch/cpuinfo/archive/63b254577ed77a8004a9be6ac707f3dccc4e1fd9.tar.gz"],
 )
 
+# === Csmith ===
+# https://embed.cs.utah.edu/csmith/
+
+http_archive(
+    name = "csmith",
+    build_file_content = all_content,
+    sha256 = "ba871c1e5a05a71ecd1af514fedba30561b16ee80b8dd5ba8f884eaded47009f",
+    strip_prefix = "csmith-csmith-2.3.0",
+    urls = ["https://github.com/csmith-project/csmith/archive/refs/tags/csmith-2.3.0.tar.gz"],
+)
+
 # === DeepDataFlow ===
 # https://zenodo.org/record/4122437
 

compiler_gym/bin/service.py

@@ -75,7 +75,6 @@
 import sys
 from typing import Iterable
 
-import humanize
 from absl import app, flags
 
 from compiler_gym.datasets import Dataset
@@ -132,7 +131,8 @@ def summarize_datasets(datasets: Iterable[Dataset]) -> str:
             [
                 (
                     n,
-                    humanize.intcomma(f) if f >= 0 else "∞",
+                    # A size of zero means infinite.
+                    f"{f:,d}" if f > 0 else "∞",
                     l,
                     v,
                 )

compiler_gym/datasets/benchmark.py

@@ -298,11 +298,15 @@ def from_file_contents(cls, uri: str, data: bytes):
     def __eq__(self, other: Union[str, "Benchmark"]):
         if isinstance(other, Benchmark):
             return self.uri == other.uri
-        else:
-            return self.uri == other
+        return self.uri == other
 
-    def __ne__(self, other: Union[str, "Benchmark"]):
-        return not self == other
+    def __lt__(self, other: Union[str, "Benchmark"]):
+        if isinstance(other, Benchmark):
+            return self.uri < other.uri
+        return self.uri < other
+
+    def __le__(self, other: Union[str, "Benchmark"]):
+        return self < other or self == other
 
 
 class BenchmarkInitError(OSError):

compiler_gym/datasets/dataset.py

@@ -233,21 +233,19 @@ def site_data_size_in_bytes(self) -> int:
             )
         return total_size
 
-    # We use Union[int, float] to represent the size because infinite size is
-    # represented by math.inf, which is a float. For all other sizes this should
-    # be an int.
     @property
-    def size(self) -> Union[int, float]:
-        """The number of benchmarks in the dataset. If the number of benchmarks
-        is unbounded, for example because the dataset represents a program
-        generator that can produce an infinite number of programs, the value is
-        :code:`math.inf`.
+    def size(self) -> int:
+        """The number of benchmarks in the dataset.
+
+        If the number of benchmarks is unknown or unbounded, for example because
+        the dataset represents a program generator that can produce an infinite
+        number of programs, the value is 0.
 
-        :type: Union[int, float]
+        :type: int
         """
         return 0
 
-    def __len__(self) -> Union[int, float]:
+    def __len__(self) -> int:
         """The number of benchmarks in the dataset.
 
         This is the same as :meth:`Dataset.size
@@ -256,10 +254,27 @@ def __len__(self) -> Union[int, float]:
             >>> len(dataset) == dataset.size
             True
 
-        :return: An integer, or :code:`math.float`.
+        If the number of benchmarks is unknown or unbounded, for example because
+        the dataset represents a program generator that can produce an infinite
+        number of programs, the value is 0.
+
+        :return: An integer.
         """
         return self.size
 
+    def __eq__(self, other: Union["Dataset", str]) -> bool:
+        if isinstance(other, Dataset):
+            return self.name == other.name
+        return self.name == other
+
+    def __lt__(self, other: Union["Dataset", str]) -> bool:
+        if isinstance(other, Dataset):
+            return self.name < other.name
+        return self.name < other
+
+    def __le__(self, other: Union["Dataset", str]) -> bool:
+        return self < other or self == other
+
     @property
     def installed(self) -> bool:
         """Whether the dataset is installed locally. Installation occurs