Running docker having code with nemo library, running in an environment without internet?

[deepakdhiman7]

We need to host our code base as a Docker container in an environment where internet is not available. We are using NeMo library. Is it possible to run it in offline mode. Please suggest, with code if possible, how and what do we need to do in such case?

[titu1994]

Nemo itself should not require online but it depends on Huggingface Transformers which does require online - unless you first download the required models and then cache it to some dir and explicitly set environment flag (see jenkinsfile for example).

[deepakdhiman7]

Hello

Thanks for replying. I went through Jenkins file.

pipeline {
agent {
docker {
image 'nvcr.io/nvidia/pytorch:23.06-py3'
args '--device=/dev/nvidia0 --gpus all --user 0:128 -v /home/TestData:/home/TestData -v $HOME/.cache:/root/.cache --shm-size=8g --env TRANSFORMERS_OFFLINE=1'
}
}

I see TRANSFORMERS_OFFLINE=1.

I need to prepare the Docker which is supposed to work on env where there in no internet. This is one of our legacy system that I am fixing for. Now, I am thinking to prepare the .cache of Huggingface in the path which I give, then I want all files to simply refer to this .cache folder. I checked in Python env site-packages files where nemo modules are placed. Unable to understand how to map all this precisely.

I did read Huggingface Documentation. In my project, I have a server.py file which starts the application. I added env variable so that I can get the cache in given directory but when I run this code, it only created nemo_nlp_tmp directory in local_hf_cache folder which is empty. I did below changes in server.py

`import os
from api import app

cache_path = '/home/centos/cair/local_hf_cache/'
os.environ['HF_HOME'] = cache_path

if name == "main":
app.run(host="0.0.0.0", port=5010, debug=True)
`

We need to prepare our application in such a way so that it works in offline mode as well. Currently I tried to run it in an env where we have internet and it is working fine. Now, to test that if it will work in offline mode, how do I setup. I am primarily getting error that 'bert' file not found when I run in env without intenet. What exactly I have to do. Currently I get below error:

`/PunctAPI
starting
[NeMo W 2023-07-11 07:04:43 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-07-11 07:04:43 experimental:27] Module <class 'nemo.collections.nlp.data.language_modeling.megatron.megatron_batch_samplers.MegatronPretrainingRandomBatchSampler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-07-11 07:04:43 experimental:27] Module <class 'nemo.collections.nlp.models.text_normalization_as_tagging.thutmose_tagger.ThutmoseTaggerModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2023-07-11 07:04:46 tokenizer_utils:130] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /tmp/tmp9ncqco8p/tokenizer.vocab_file, merges_files: None, special_tokens_dict: {}, and use_fast: False
[NeMo I 2023-07-11 07:04:52 tokenizer_utils:130] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /tmp/tmp9ncqco8p/tokenizer.vocab_file, merges_files: None, special_tokens_dict: {}, and use_fast: False
[NeMo E 2023-07-11 07:04:58 common:503] Model instantiation failed!
Target class: nemo.collections.nlp.models.token_classification.punctuation_capitalization_model.PunctuationCapitalizationModel
Error(s): Unable to instantiate HuggingFace AUTOTOKENIZER for bert-base-uncased. Exception: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 70, in init
self.tokenizer = AUTOTOKENIZER.from_pretrained(
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 549, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 401, in get_tokenizer_config
resolved_config_file = cached_file(
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 408, in cached_file
resolved_file = hf_hub_download(
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1136, in hf_hub_download
with open(ref_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 482, in from_config_dict
instance = imported_cls(cfg=config, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 102, in init
super().init(cfg=cfg, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 77, in init
self.setup_tokenizer(cfg.tokenizer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 201, in setup_tokenizer
self.tokenizer = get_tokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/modules/common/tokenizer_utils.py", line 134, in get_tokenizer
return AutoTokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 81, in init
raise ValueError(
ValueError: Unable to instantiate HuggingFace AUTOTOKENIZER for bert-base-uncased. Exception: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 70, in init
self.tokenizer = AUTOTOKENIZER.from_pretrained(
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 549, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 401, in get_tokenizer_config
resolved_config_file = cached_file(
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 408, in cached_file
resolved_file = hf_hub_download(
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1136, in hf_hub_download
with open(ref_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "get_model.py", line 3, in
model = PunctuationCapitalizationModel.restore_from('./model/bncc_Punctuation_and_Capitalization.nemo')
File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/modelPT.py", line 311, in restore_from
instance = cls._save_restore_connector.restore_from(
File "/usr/local/lib/python3.8/dist-packages/nemo/core/connectors/save_restore_connector.py", line 235, in restore_from
loaded_params = self.load_config_and_state_dict(
File "/usr/local/lib/python3.8/dist-packages/nemo/core/connectors/save_restore_connector.py", line 158, in load_config_and_state_dict
instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 504, in from_config_dict
raise e
File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 496, in from_config_dict
instance = cls(cfg=config, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 102, in init
super().init(cfg=cfg, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 77, in init
self.setup_tokenizer(cfg.tokenizer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 201, in setup_tokenizer
self.tokenizer = get_tokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/modules/common/tokenizer_utils.py", line 134, in get_tokenizer
return AutoTokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 81, in init
raise ValueError(
ValueError: Unable to instantiate HuggingFace AUTOTOKENIZER for bert-base-uncased. Exception: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'
/PunctAPI
[NeMo W 2023-07-11 07:05:05 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-07-11 07:05:06 experimental:27] Module <class 'nemo.collections.nlp.data.language_modeling.megatron.megatron_batch_samplers.MegatronPretrainingRandomBatchSampler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-07-11 07:05:06 experimental:27] Module <class 'nemo.collections.nlp.models.text_normalization_as_tagging.thutmose_tagger.ThutmoseTaggerModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-07-11 07:05:06 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/celery/platforms.py:800: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(

-------------- celery@42eacf2830ef v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Linux-5.14.0-134.el9.x86_64-x86_64-with-glibc2.29 2023-07-11 07:05:06
-- * - **** ---

• ** ---------- [config]
• ** ---------- .> app: api:0x7f4fc4db65e0
• ** ---------- .> transport: amqp://guest:**@192.168.3.2:5672//
• ** ---------- .> results: redis://:**@192.168.3.2:6379/0
• *** --- * --- .> concurrency: 12 (solo)
  -- ******* ---- .> task events: ON
  --- ***** -----
  -------------- [queues]
  .> punct_api_queue exchange=punct_api_exchange(direct) key=punct_api_queue

[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. tasks.punct_task.punct_task

[NeMo I 2023-07-11 07:05:48 tokenizer_utils:130] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /tmp/tmpawv3jedq/tokenizer.vocab_file, merges_files: None, special_tokens_dict: {}, and use_fast: False
[NeMo I 2023-07-11 07:05:54 tokenizer_utils:130] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /tmp/tmpawv3jedq/tokenizer.vocab_file, merges_files: None, special_tokens_dict: {}, and use_fast: False
[NeMo E 2023-07-11 07:06:00 common:503] Model instantiation failed!
Target class: nemo.collections.nlp.models.token_classification.punctuation_capitalization_model.PunctuationCapitalizationModel
Error(s): Unable to instantiate HuggingFace AUTOTOKENIZER for bert-base-uncased. Exception: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 70, in init
self.tokenizer = AUTOTOKENIZER.from_pretrained(
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 549, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 401, in get_tokenizer_config
resolved_config_file = cached_file(
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 408, in cached_file
resolved_file = hf_hub_download(
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1136, in hf_hub_download
with open(ref_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 482, in from_config_dict
instance = imported_cls(cfg=config, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 102, in init
super().init(cfg=cfg, trainer=trainer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 77, in init
self.setup_tokenizer(cfg.tokenizer)
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 201, in setup_tokenizer
self.tokenizer = get_tokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/modules/common/tokenizer_utils.py", line 134, in get_tokenizer
return AutoTokenizer(
File "/usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 81, in init
raise ValueError(
ValueError: Unable to instantiate HuggingFace AUTOTOKENIZER for bert-base-uncased. Exception: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--bert-base-uncased/refs/main'`