Inconvenience Loading Several Models With Colliding Namespaces

[snakers4]

If you load 2+ models from torch.hub at the same time in the same process, they are treated like one namespace, i.e. if you have a utils.py module in both packages, it raises cryptic errors.

This comment best describes this behavior - snakers4/silero-vad#28 (comment) - a used tried loading 2 models at the same time (I just renamed the utils module in one of them to avoid this).

• https://github.com/snakers4/silero-vad
• https://github.com/snakers4/silero-models

This is not really a problem and I am not sure this is intentional, but very many torch.hub packages have utils.py module.

I understand that this can be solved with more proper packaging / containerization / CI, but since the ideology of torch.hub is to keep things minimal, this may become an issue in future.

[ailzhang]

@snakers4 Good point! btw we have documented a similar behavior in https://pytorch.org/docs/stable/hub.html#known-limitations which is a variant of issue raised here.
We might be able to put proper scope when we load so that we can avoid this limitation, but we didn't do this for the first design due to torch.hub.load is supposed to be used in light workload instead of really heavy training work etc.
But I'd like to say this feature request is totally valid, although we may not have bw to work on it atm. We'd happily take and review PRs that implement it.

[snakers4]

torch.hub.load is supposed to be used in light workload instead of really heavy training work etc.

I totally agree here.
I circumvented this just by renaming the utils.py into utils_vad.py.
Essentially we mostly use pre-build docker images internally and we did not want to support extra pip / conda packages just for our model demos, to keep things simple.
For a simple demo there is little value in making a separate package instead of just loading utils.py via torch hub.

But I'd like to say this feature request is totally valid, although we may not have bw to work on it atm. We'd happily take and review PRs that implement it.

I am not the biggest expert on how python internals work, but so far I have been able to replicate an error on a minimalist example (if this helps anyone):

A minimal set up like this:

tree -L 2
.
├── module1
│   ├── hubconf.py
│   └── utils.py
├── module2
│   ├── hubconf.py
│   └── utils.py
└── test.ipynb

Error can be replicated like this:

import sys
from torch.hub import import_module


MODULE_HUBCONF = 'hubconf.py'

# this works just fine
repo_dir = 'module1'
sys.path.insert(0, repo_dir)
hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + MODULE_HUBCONF)
sys.path.remove(repo_dir)

# this fails if run after the above code
repo_dir = 'module2'
sys.path.insert(0, repo_dir)
hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + MODULE_HUBCONF)
sys.path.remove(repo_dir)

I will report here if I find an easy solution

[snakers4]

A naïve solution I found:

import sys
from torch.hub import import_module


MODULE_HUBCONF = 'hubconf.py'
repo_dir = 'git_repo'

repo_subdir = 'module1'
sys.path.insert(0, repo_dir)
hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + repo_subdir + '/' + MODULE_HUBCONF)
sys.path.remove(repo_dir)

repo_subdir = 'module2'
sys.path.insert(0, repo_dir)
hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + repo_subdir + '/' + MODULE_HUBCONF)
sys.path.remove(repo_dir)

works, but you have to manage imports inside of utils.py as follows:

from module2.utils import example, example2

def return_utils():
    return example, example2

thinking about a backwards compatible one

[snakers4]

I have tried various ideas how to make this work reverse compatible
No luck yet
I do not mind actually doing /testing the PR itself, but maybe anyone just knows a workaround
Hope that these minimal repro scripts help

[Hasankanso]

any news?

I have problem with utils too. in my case one of the utils are in my environment, and I'm trying to load another model from hub, but I always get utils.x not found

editing on sys.path is obviously useless, I don't know why...If I load model first detector = torch.hub.load('ultralytics/yolov3', 'yolov3', pretrained=True).autoshape() it get initialized normally, but then in from config import cfg I get utils.dir not found
if I load the model after all imports, I get utils.datasets not found even if I changed on sys.path nothing works...I hate my existence already...been debugging it since 4 days...

import sys
import os
import os.path as osp
import argparse
import numpy as np
import cv2
import torch
import torchvision.transforms as transforms
from torch.nn.parallel.data_parallel import DataParallel
import torch.backends.cudnn as cudnn


sys.path = ['/root/.cache/torch/hub/ultralytics_yolov3_master/utils/','/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/local/lib/python3.7/dist-packages/jetcam-0.0.0-py3.7.egg', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.7/dist-packages/IPython/extensions']

print(sys.path)

detector = torch.hub.load('ultralytics/yolov3', 'yolov3', pretrained=True).autoshape() 

sys.path.insert(0, osp.join('..', 'main'))
sys.path.insert(0, osp.join('..', 'data'))
sys.path.insert(0, osp.join('..', 'common'))
print(sys.path)
from config import cfg
from model import get_model
from utils.preprocessing import process_bbox, generate_patch_image
from utils.transforms import pixel2cam, cam2pixel
from utils.vis import vis_mesh, save_obj, render_mesh

[snakers4]

In your case you can try 2 obvious hacks:

• Try importing your utils module as another module
• If this does not help then just copy yolo utils to your project under a different name

[Hasankanso]

In your case you can try 2 obvious hacks:

* Try importing your utils module as another module

* If this does not help then just copy yolo utils to your project under a different name

Thank you,
for both solutions I have to change "import utils" manually in the entire project, since I only use jupyter to edit the code...
Anyway I did the second point and it worked(changed the name of utils to yolo_utils).
But for some reason the other project which is this https://github.com/mks0601/I2L-MeshNet_RELEASE by the way, couldn't be executed because the model method expect different number of parameters...which implies the model is using the wrong library again...I gave up on it, but hopefully I find some convenient way to merge multiple projects...