requests.exceptions.SSLError: HTTPSConnectionPool

[starlitsky2010]

Hi NVIDIA,

Environment:

enroot 3.4.1
pyxis 0.7.0
slurm slurm-wlm 19.05.5
Ubuntu 20.04
NeMo docker image: nvcr.io+ea-bignlp+nemofw-training+23.04.1-py3.sqsh

When I do the pile dataset preprocessing. The following error occurs.

Traceback (most recent call last):
  File "/opt/NeMo/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py", line 74, in __init__
    self.tokenizer = AUTOTOKENIZER.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 643, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 487, in get_tokenizer_config
    resolved_config_file = cached_file(
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 417, in cached_file
    resolved_file = hf_hub_download(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download
    metadata = get_hf_file_metadata(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1532, in get_hf_file_metadata
    r = _request_wrapper(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 407, in _request_wrapper
    response = _request_wrapper(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 442, in _request_wrapper
    return http_backoff(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_http.py", line 212, in http_backoff
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /gpt2/resolve/main/tokenizer_config.json (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2635)')))

Thanks
Aaron