blib2to3 tries to store grammar pickle in global site-packages

[tiran]

blib2to3 tries to store a version specific pickle of the grammar in the global site-package directory. This works fine for user-owned installations in a virtual env but not for OS vendored installation. Users typically don't have permission to write to system-wide site-packages directory.

$ strace black
...
openat(AT_FDCWD, "/usr/lib/python3.6/site-packages/blib2to3/Grammar3.6.5.final.0.pickle", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 EACCES (Permission denied)
stat("/usr/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt", {st_mode=S_IFREG|0644, st_size=793, ...}) = 0
stat("/usr/lib/python3.6/site-packages/blib2to3/PatternGrammar3.6.5.final.0.pickle", 0x7ffc225e5c40) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=793, ...}) = 0
...

I tracked down the write through Grammar.dump, load_grammar and load_packaged_grammar to https://github.com/ambv/black/blob/master/blib2to3/pygram.py. The load_grammar takes a pickle file name as gp argument, but load_packaged_grammar and pygram don't use it.

Some pickle files are checked into git but not installed by setup.py, too.

[tiran]

The cache shoves off about 18ms startup time on my machine:

$ python3 -m timeit -s "from blib2to3.pgen2 import pgen" "pgen.generate_grammar('blib2to3/Grammar.txt')"
100 loops, best of 3: 18.1 msec per loop
$ python3 -m timeit -s "from blib2to3.pgen2 import driver" "driver.load_grammar('blib2to3/Grammar.txt')"
1000 loops, best of 3: 245 usec per loop

[ambv]

We should move this cache to the cache dir in appdirs. @ojii, interested?

[ojii]

I'll try to have a look next week.