seldonio/mlflowserver:1.12.0-dev now unable to server models

[zbloss]

Describe the bug

My models were being served one week ago, but today after redeploying them, I'm seeing the below error.

To reproduce

Deploy a SeldonDeployment using the default MLFLOW_SERVER with image seldonio/mlflowserver:1.12.0-dev

Environment

• Cloud Provider: AWS
• Kubernetes Cluster Version 1.22.3
• Deployed Seldon System Images:
  value: docker.io/seldonio/engine:1.12.0-dev
  value: docker.io/seldonio/seldon-core-executor:1.12.0-dev
  image: docker.io/seldonio/seldon-core-operator:1.12.0-dev

Model Details

• Images of your model:
• Logs of your model:

Executing before-run script
---> Creating environment with Conda...
INFO:root:Copying contents of /mnt/models to local
INFO:root:Reading MLmodel file
INFO:root:Creating Conda environment 'mlflow' from conda.yaml
Collecting package metadata (repodata.json): ...working... failed
NotWritableError: The current user does not have write permissions to a required path.
  path: /opt/conda/pkgs/cache/47929eba.json
  uid: 8888
  gid: 0
If you feel that permissions on this path are set incorrectly, you can manually
change them by executing
  $ sudo chown 8888:0 /opt/conda/pkgs/cache/47929eba.json
In general, it's not advisable to use 'sudo conda'.
Traceback (most recent call last):
  File "./conda_env_create.py", line 151, in <module>
    main(args)
  File "./conda_env_create.py", line 146, in main
    setup_env(model_folder)
  File "./conda_env_create.py", line 54, in setup_env
    create_env(env_file_path)
  File "./conda_env_create.py", line 110, in create_env
    run(cmd, shell=True, check=True)
  File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'conda env create -n mlflow --file /tmp/tmpyfdn9z_4/conda.yaml' returned non-zero exit status 1.```

[igor-stranjanac]

I can confirm that this issue still exists. Can anybody please give some advice regarding it?

[adriangonz]

This should have been fixed by #3754. @zbloss @igor-stranjanac can you try the latest version of the image and reopen if it's still an issue?

[axsaucedo]

Following up on this, these were the changes that were recently added - it would be great if you could try again

seldon-core/wrappers/s2i/python/Dockerfile

Lines 37 to 40 in bd0d4fe

	RUN mkdir -p /.conda && \
	chmod a+rwx /.conda && \
	chown -R 8888 /microservice && \
	conda clean --index-cache

More information can be found in this issue #3670

[igor-stranjanac]

Hi @adriangonz , I've restarted the deployment and it works, thanks!
Hi @axsaucedo , these changes from #3754 fixed that issue, and it was working for a while, but suddenly started to receive the error from above last week.

[adriangonz]

Hey @igor-stranjanac, if the fixes worked initially, could it be that different nodes on your K8s cluster have different cached versions of that image?

[igor-stranjanac]

Hey @adriangonz , yeah, could be the issue. I will redeploy them, just to be safe. Thanks for the hint!