PATH issues due to incomplete conda environment activation on AWS Batch #734
Comments
|
I will try to compile a minimum example reproducing the problem. |
|
You can also print |
|
@savingoyal |
|
Umm, that shouldn't be the case - In 2.2.10 we have the same snippet - https://github.com/Netflix/metaflow/blob/2.2.10/metaflow/plugins/conda/conda_step_decorator.py#L256 |
|
For 2.2.10, I use: import os
from metaflow import FlowSpec, step, conda, conda_base, batch
@conda_base(libraries={...})
class TestFlow(FlowSpec):
@step
def start(self):
self.next(self.use_node)
@batch
@conda(libraries={"nodejs": ">=16.0.0"})
@step
def use_node(self):
print(os.environ.get('PATH'))
self.next(self.end)
@step
def end(self):
pass
if __name__ == "__main__":
TestFlow()and get: Above, the |
|
As for Metaflow 2.3.6, I observe the same behavior: Note, however, that I cannot test vanilla Metaflow due to how access to our infrastructure is managed. Perhaps it would be good to re-run the above tests using an "official" Metaflow-on-AWS deployment like the Sandboxes (to which I have no access). |
|
Remark: It looks like the |
|
Metaflow 2.4.1 addresses this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The AWS Batch jobs created by Metaflow invoke Python like
metaflow_TrainingFlow_linux-64_ad59afd22c0b2ee67176dd7dc845cf35ff5599f9/bin/python, which does not update thePATHvariable. Therefore, Python packages relying on dependencies like NodeJS fail, even though the corresponding dependencies are installed within the Conda environment (metaflow_TrainingFlow_linux-64_ad59afd22c0b2ee67176dd7dc845cf35ff5599f9in the example).A work-around for this problem is to add
to the flow step run on AWS Batch. However, it appears that a more proper solution would be Metaflow invoking
conda activate 64_ad59afd22c0b2ee67176dd7dc845cf35ff5599f9before startingpythonin the AWS Batch task definition.The text was updated successfully, but these errors were encountered: