bpo-29708: support SOURCE_DATE_EPOCH env var in py_compile

to allow for reproducible builds of python packages

See https://reproducible-builds.org/ for why this is good
and https://reproducible-builds.org/specs/source-date-epoch/
for the definition of this variable.

Background:
In some distributions like openSUSE, binary rpms contain precompiled .pyc files.

And packages like amqp or twisted dynamically generate .py files at build time
so those have the current time and that timestamp gets embedded
into the .pyc file header.
When we then adapt file timestamps in rpms to be constant,
the timestamp in the .pyc header will no more match
the .py timestamp in the filesystem.
The software will still work, but it will not use the .pyc file as it should.

Author: bmwiedemann

Doc/library/py_compile.rst

@@ -57,6 +57,10 @@ byte-code cache files in the directory containing the source code.
    enum and controls how the generated ``.pyc`` files are invalidated at
    runtime.
 
+   If the SOURCE_DATE_EPOCH environment variable is set, the timestamp entry in
+   the .pyc file header, will be limited to this value.
+   See https://reproducible-builds.org/specs/source-date-epoch/ for more info.
+
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
       default was *file* + ``'c'`` (``'o'`` if optimization was enabled).

Lib/py_compile.py

@@ -147,6 +147,14 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1,
         pass
     if invalidation_mode == PycInvalidationMode.TIMESTAMP:
         source_stats = loader.path_stats(file)
+        source_date_epoch = os.environ.get('SOURCE_DATE_EPOCH')
+        if source_date_epoch:
+            try:
+                source_date_epoch = int(source_date_epoch)
+            except ValueError:
+                raise ValueError("SOURCE_DATE_EPOCH is not a valid integer")
+            if source_stats['mtime'] > source_date_epoch:
+                source_stats['mtime'] = source_date_epoch
         bytecode = importlib._bootstrap_external._code_to_timestamp_pyc(
             code, source_stats['mtime'], source_stats['size'])
     else:

Lib/test/test_py_compile.py

@@ -98,6 +98,18 @@ def test_bad_coding(self):
         self.assertFalse(os.path.exists(
             importlib.util.cache_from_source(bad_coding)))
 
+    def test_source_date_epoch(self):
+        testtime = 123456789
+        with support.EnvironmentVarGuard() as env:
+            env["SOURCE_DATE_EPOCH"] = str(testtime)
+            py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+        with open(self.pyc_path, "rb") as f:
+            f.read(4) # Skip the magic number.
+            timebytes = f.read(4) # Read timestamp.
+        self.assertEqual(testtime, int.from_bytes(timebytes, 'little'))
+
     @unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
     def test_double_dot_no_clobber(self):
         # http://bugs.python.org/issue22966

Misc/NEWS.d/next/Build/2017-09-07-04-20-49.bpo-29708.YCaHEx.rst

@@ -0,0 +1,2 @@
+If the SOURCE_DATE_EPOCH environment variable is set,
+py_compile will use it to override the timestamps it puts into .pyc files.