Mount python worker files after taking memory snapshot #2806
Conversation
hoodmane
force-pushed
the
hoodmane/python-later-worker-files
branch
from
93f9e76
to
246c248
Compare
| Module.FS.mount(mdFS, {}, '/session/metadata'); | ||
| simpleRunPython( | ||
| Module, | ||
| `import sys; sys.path.append("/session/metadata"); del sys` |
There was a problem hiding this comment.
I might be totally off base here but wouldn't we want sys.path.insert(0, ...) here?
If a user has http.py file with his main.py file we'd want his http.py file to take precedence over the builtin http module when he uses import http no?
There was a problem hiding this comment.
Edit: I can see this was moved so obviously it isn't specific to this PR but just a general question
There was a problem hiding this comment.
Actually, and sorry for being annoying but the move actually does change behaviour where now site-packages are before session/metadata in path in contrast to previous behaviour.
There was a problem hiding this comment.
Good point I put that back, and sys.path.insert(0, ...) would be better. I'll make that change in a separate PR though.
hoodmane
force-pushed
the
hoodmane/python-later-worker-files
branch
2 times, most recently
from
045d770
to
c05f0ea
Compare
| @@ -424,7 +424,7 @@ struct CompatibilityFlags @0x8f8c1b68151b6cef { | |||
| pythonWorkers @43 :Bool | |||
| $compatEnableFlag("python_workers") | |||
| $pythonSnapshotRelease(pyodide = "0.26.0a2", pyodideRevision = "2024-03-01", | |||
| packages = "2024-03-01", backport = 0) | |||
| packages = "2024-03-01", backport = 1) | |||
There was a problem hiding this comment.
Curious why you're bumping this.
There was a problem hiding this comment.
Well I made a new bundle with the updated code with backport number 1. I want the new bundle to be used, so I have to bump the backport number.
There was a problem hiding this comment.
Ahh, because the .ts file changes here need a new bundle in order to be used in prod, right?
It's going to take a bit of a mind shift to keep that in mind.
There was a problem hiding this comment.
Yeah, it's annoying.
There was a problem hiding this comment.
yeah.. this is the thing that I'm concerned is going to end up causing us too much pain. Like even my small PR will need a regenerated bundle, right? #2830
I guess small changes like that can wait. Anyway, we should discuss about how we want to handle this. Started a thread internally for this.
This ensures that the contents of worker files cannot be accessed prior to taking the snapshot and so won't appear in the linear memory.
hoodmane
force-pushed
the
hoodmane/python-later-worker-files
branch
from
c05f0ea
to
08bad5a
Compare
This ensures that the contents of worker files cannot be accessed prior to taking the snapshot and so won't appear in the linear memory.