Improve the the blib2to3 grammar caching mechanism

[EWouters]

I get an error upon launching Jupyter Lab with the jupyterlab_code_formatter extension enabled:

[I 10:37:25.686 LabApp] Node v10.8.0

[I 10:37:26.075 LabApp] Build is up to date
Generating grammar tables from C:\Users\%USERNAME%\Anaconda3\lib\site-packages\blib2to3\Grammar.txt
Writing grammar tables to C:\Users\%USERNAME%\AppData\Local\black\black\Cache\18.9b0\Grammar3.7.1.final.0.pickle
Writing failed: [Errno 2] No such file or directory: 'C:\\Users\\%USERNAME%\\AppData\\Local\\black\\black\\Cache\\18.9b0\\Grammar3.7.1.final.0.pickle'
Generating grammar tables from C:\Users\%USERNAME%\Anaconda3\lib\site-packages\blib2to3\PatternGrammar.txt
Writing grammar tables to C:\Users\%USERNAME%\AppData\Local\black\black\Cache\18.9b0\PatternGrammar3.7.1.final.0.pickle
Writing failed: [Errno 2] No such file or directory: 'C:\\Users\\%USERNAME%\\AppData\\Local\\black\\black\\Cache\\18.9b0\\PatternGrammar3.7.1.final.0.pickle'

After manually creating the folder C:\\Users\\%USERNAME%\\AppData\\Local\\black\\black\\Cache\\18.9b0 and relaunching Jupyter lab the error disappears (even though the folder seems to stay empty).

The error appears to originate from this line. It might be related to #192. I think this line fails because the cache directory does not exist, but I don't understand why it doesn't exist.

Other users confirmed this happens on linux and with version 19.10b as well in this issue.

I tried to find the cause of this issue in the extension, but it doesn't seem to do anything out of the ordinary with the cache folder, so I was hoping someone here can tell me why the cache folder would not exist?

[EWouters]

Looks like I found the cause, thanks for the fix @AlJohri!

[AlJohri]

Thanks @EWouters! If you the bandwidth you can make a PR to updated the vendored blib2to3 as described in the conversation here: #1224

I haven't gotten the chance.

[copperwiring]

Has there been a PR on this yet?

[ichard26]

I'm reopening this issue since 1. the issue still exists and 2. the fix #1223 proposes isn't ideal.

I see two possible solutions to this issue:

1. Upon importing Black, the cache directory where the grammar table cache is stored is created. (CACHE_DIR created too late when using black as a library #1223 suggests going this route)
2. When Black is called using the CLI and actually finishes doing something, the cache directory is created if needed and the grammar table cache¹ is then written to disk.

While option 1 is simpler, the cache directory would be created when simply importing Black. Applications that use Black as a library might never use the cache, yet the cache dir would still be created, not an ideal side-effect. Option 2 is a bit more involved but it avoids the unnecessary side-effect mentioned above. Anyway the other caches like the cache which records which files are well-formatted already behave this way. It's only the grammar table cache that tries to write to disk when Black is imported.

---

1. which is handled completely differently compared to the other caches Black maintains but still lives in the same directory

[VasuBhog]

I am still seeing this error with
papermill 2.2.2
black 20.8b1
Mac OS: 10.15.7

Is there any update or workaround for this error?
Upon papermill execution:
Generating grammar tables from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blib2to3/Grammar.txt Writing grammar tables to /Users/xxx/Library/Caches/black/20.8b1/Grammar3.6.8.final.0.pickle Writing failed: [Errno 2] No such file or directory: '/Users/xxx/Library/Caches/black/20.8b1/tmpp31361aj' Generating grammar tables from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt Writing grammar tables to /Users/xxx/Library/Caches/black/20.8b1/PatternGrammar3.6.8.final.0.pickle Writing failed: [Errno 2] No such file or directory: '/Users/xxx/Library/Caches/black/20.8b1/tmpcq8qn401'

[AlJohri]

@VasuBhog As a work around, use the black CLI at least once locally or run: python -c "import logging; logging.basicConfig(level='INFO'); import black" once in your python environment (see #1223 for details).

[andyljones]

Had this happen on macos; as with the OP, immediate fix was to mkdir the path it was complaining about,

mkdir /Users/andy/Library/Caches/black/21.5b1

This issue showed up at about the same time as I updated various libs in my environment, and there's a 21.4b2 dir next to the one I just created. So my suspicion is that there's some latch in black's caching that doesn't get un-latched when the lib is updated.

[Qiuzhuang]

just manually mkdir -p to solve this error.

[jodemaey]

Hi,

I got this error too with papermill 2.3.3 and black 21.9b0 on Ubuntu 20.04 LTS.

[darachm]

I get this error trying to run papermill (which apparently imports black, hello) inside of a Singularity container.

• Improve the the blib2to3 grammar caching mechanism  #1143 (comment) does not work, likely because the container is built as root and then run as a user
• similarly, mkdir won't work because the container is supposed to be able to be run by different users - hence different user-specific caches
• similarly, this is being used inside of Nextflow for some pipeline management, so no I do not want it to be interacting or dependent on states in home cache directories across different machines, so I'm not binding those into the container

Before I go and remove black/papermill from my workflow, is there a way to move the CACHE_DIR to a predictable location? Can I set this with an environmental variable? Can the cache go into /tmp or something? Or can I disable it?

[michaelaye]

python -c "import logging; logging.basicConfig(level='INFO'); import black"

@AlJohri This workaround does not work:

$ python -c "import logging; logging.basicConfig(level='INFO'); import black"
INFO:blib2to3.pgen2.driver:Generating grammar tables from /luna4/maye/miniconda3/envs/py38/lib/python3.8/site-packages/blib2to3/Grammar.txt
INFO:blib2to3.pgen2.driver:Writing grammar tables to /u/paige/maye/.cache/black/21.11b1/Grammar3.8.12.final.0.pickle
INFO:blib2to3.pgen2.driver:Writing failed: [Errno 2] No such file or directory: '/u/paige/maye/.cache/black/21.11b1/tmpp8nlg3be'
INFO:blib2to3.pgen2.driver:Generating grammar tables from /luna4/maye/miniconda3/envs/py38/lib/python3.8/site-packages/blib2to3/PatternGrammar.txt
INFO:blib2to3.pgen2.driver:Writing grammar tables to /u/paige/maye/.cache/black/21.11b1/PatternGrammar3.8.12.final.0.pickle
INFO:blib2to3.pgen2.driver:Writing failed: [Errno 2] No such file or directory: '/u/paige/maye/.cache/black/21.11b1/tmp0dmgsyif'

[Paul-Durrant]

This issue is still a problem. Perhaps someone could document how to use black as a library so that this warning is not issued?

[felix-hilden]

Please follow #779 for a discussion about Black's public Python API, which doesn't exist yet. This definitely relates to that, but "importing Black" isn't exactly a thing yet 😄 at least officially.

[hauntsaninja]

I think #3193 removed the logging? There's also e.g. BLACK_CACHE_DIR. Feel free to re-open if you still have issues