Imports interfere with AWS boto3

[thehappycheese]

Describe the bug

Does arcgis monkey patch ssl, or do any non standard import behavior?
Importing acrgis first creates an error when trying to use boto3.
One of the two libraries is doing some badly behaved import side effects. I don't know which?

To Reproduce

Simply import FeatureLayer before trying to import and create a boto3 s3 client.
I realize this is tricky unless you happen to have some AWS profile configured;

import boto3
from arcgis.features import FeatureLayer
session = boto3.Session(
    profile_name="..."
)
client = session.client('s3')
# Fails... see error message further below

A workaround is to swap the order of imports:

# Swapping the order of imports; and it works again:
from arcgis.features import FeatureLayer
import boto3
session = boto3.Session(profile_name="...")
client = session.client('s3')
# Succeeds

error:

{
	"name": "RecursionError",
	"message": "maximum recursion depth exceeded",
	"stack": "---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
c:\\Users\\...\\LOCAL\\GIT\\pta_api\\test_rtop_archive.ipynb Cell 2 line 1
----> <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=0'>1</a> pta_api.fetch_rtop_archive(
      <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=1'>2</a>     aws_access_key_id     = keyring.get_password(\"...-key\",        \"user\"),
      <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=2'>3</a>     aws_secret_access_key = keyring.get_password(\"...-access-key\", \"user\"),
      <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=3'>4</a>     year                  = 2023,
      <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=4'>5</a> )

File ~\\LOCAL\\GIT\\pta_api\\src\\pta_api\\_fetch_rtop_archive.py:51, in fetch_rtop_archive(aws_access_key_id, aws_secret_access_key, year, month, day, region_name, parse_dates, parse_geometry)
     48 elif day is not None:
     49     raise ValueError('day specified without month')
---> 51 client = session.client('s3')
     53 # download all files (one by one; not ideal at all, but this is the only way)
     54 results = []

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\boto3\\session.py:299, in Session.client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    217 def client(
    218     self,
    219     service_name,
   (...)
    228     config=None,
    229 ):
    230     \"\"\"
    231     Create a low-level service client by name.
    232 
   (...)
    297 
    298     \"\"\"
--> 299     return self._session.create_client(
    300         service_name,
    301         region_name=region_name,
    302         api_version=api_version,
    303         use_ssl=use_ssl,
    304         verify=verify,
    305         endpoint_url=endpoint_url,
    306         aws_access_key_id=aws_access_key_id,
    307         aws_secret_access_key=aws_secret_access_key,
    308         aws_session_token=aws_session_token,
    309         config=config,
    310     )

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\session.py:997, in Session.create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    980 self._add_configured_endpoint_provider(
    981     client_name=service_name,
    982     config_store=config_store,
    983 )
    985 client_creator = botocore.client.ClientCreator(
    986     loader,
    987     endpoint_resolver,
   (...)
    995     user_agent_creator=user_agent_creator,
    996 )
--> 997 client = client_creator.create_client(
    998     service_name=service_name,
    999     region_name=region_name,
   1000     is_secure=use_ssl,
   1001     endpoint_url=endpoint_url,
   1002     verify=verify,
   1003     credentials=credentials,
   1004     scoped_config=self.get_scoped_config(),
   1005     client_config=config,
   1006     api_version=api_version,
   1007     auth_token=auth_token,
   1008 )
   1009 monitor = self._get_internal_component('monitor')
   1010 if monitor is not None:

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\client.py:159, in ClientCreator.create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config, auth_token)
    146 region_name, client_config = self._normalize_fips_region(
    147     region_name, client_config
    148 )
    149 endpoint_bridge = ClientEndpointBridge(
    150     self._endpoint_resolver,
    151     scoped_config,
   (...)
    157     ),
    158 )
--> 159 client_args = self._get_client_args(
    160     service_model,
    161     region_name,
    162     is_secure,
    163     endpoint_url,
    164     verify,
    165     credentials,
    166     scoped_config,
    167     client_config,
    168     endpoint_bridge,
    169     auth_token,
    170     endpoints_ruleset_data,
    171     partition_data,
    172 )
    173 service_client = cls(**client_args)
    174 self._register_retries(service_client)

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\client.py:490, in ClientCreator._get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge, auth_token, endpoints_ruleset_data, partition_data)
    466 def _get_client_args(
    467     self,
    468     service_model,
   (...)
    479     partition_data,
    480 ):
    481     args_creator = ClientArgsCreator(
    482         self._event_emitter,
    483         self._user_agent,
   (...)
    488         user_agent_creator=self._user_agent_creator,
    489     )
--> 490     return args_creator.get_client_args(
    491         service_model,
    492         region_name,
    493         is_secure,
    494         endpoint_url,
    495         verify,
    496         credentials,
    497         scoped_config,
    498         client_config,
    499         endpoint_bridge,
    500         auth_token,
    501         endpoints_ruleset_data,
    502         partition_data,
    503     )

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\args.py:137, in ClientArgsCreator.get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge, auth_token, endpoints_ruleset_data, partition_data)
    134 new_config = Config(**config_kwargs)
    135 endpoint_creator = EndpointCreator(event_emitter)
--> 137 endpoint = endpoint_creator.create_endpoint(
    138     service_model,
    139     region_name=endpoint_region_name,
    140     endpoint_url=endpoint_config['endpoint_url'],
    141     verify=verify,
    142     response_parser_factory=self._response_parser_factory,
    143     max_pool_connections=new_config.max_pool_connections,
    144     proxies=new_config.proxies,
    145     timeout=(new_config.connect_timeout, new_config.read_timeout),
    146     socket_options=socket_options,
    147     client_cert=new_config.client_cert,
    148     proxies_config=new_config.proxies_config,
    149 )
    151 serializer = botocore.serialize.create_serializer(
    152     protocol, parameter_validation
    153 )
    154 response_parser = botocore.parsers.create_parser(protocol)

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\endpoint.py:409, in EndpointCreator.create_endpoint(self, service_model, region_name, endpoint_url, verify, response_parser_factory, timeout, max_pool_connections, http_session_cls, proxies, socket_options, client_cert, proxies_config)
    406 endpoint_prefix = service_model.endpoint_prefix
    408 logger.debug('Setting %s timeout as %s', endpoint_prefix, timeout)
--> 409 http_session = http_session_cls(
    410     timeout=timeout,
    411     proxies=proxies,
    412     verify=self._get_verify_value(verify),
    413     max_pool_connections=max_pool_connections,
    414     socket_options=socket_options,
    415     client_cert=client_cert,
    416     proxies_config=proxies_config,
    417 )
    419 return Endpoint(
    420     endpoint_url,
    421     endpoint_prefix=endpoint_prefix,
   (...)
    424     http_session=http_session,
    425 )

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:323, in URLLib3Session.__init__(self, verify, proxies, timeout, max_pool_connections, socket_options, client_cert, proxies_config)
    321     self._socket_options = []
    322 self._proxy_managers = {}
--> 323 self._manager = PoolManager(**self._get_pool_manager_kwargs())
    324 self._manager.pool_classes_by_scheme = self._pool_classes_by_scheme

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:340, in URLLib3Session._get_pool_manager_kwargs(self, **extra_kwargs)
    336 def _get_pool_manager_kwargs(self, **extra_kwargs):
    337     pool_manager_kwargs = {
    338         'timeout': self._timeout,
    339         'maxsize': self._max_pool_connections,
--> 340         'ssl_context': self._get_ssl_context(),
    341         'socket_options': self._socket_options,
    342         'cert_file': self._cert_file,
    343         'key_file': self._key_file,
    344     }
    345     pool_manager_kwargs.update(**extra_kwargs)
    346     return pool_manager_kwargs

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:349, in URLLib3Session._get_ssl_context(self)
    348 def _get_ssl_context(self):
--> 349     return create_urllib3_context()

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:139, in create_urllib3_context(ssl_version, cert_reqs, options, ciphers)
    133     # TLSv1.2 only. Unless set explicitly, do not request tickets.
    134     # This may save some bandwidth on wire, and although the ticket is encrypted,
    135     # there is a risk associated with it being on wire,
    136     # if the server is not rotating its ticketing keys properly.
    137     options |= OP_NO_TICKET
--> 139 context.options |= options
    141 # Enable post-handshake authentication for TLS 1.3, see GH #1634. PHA is
    142 # necessary for conditional client cert authentication with TLS 1.3.
    143 # The attribute is None for OpenSSL <= 1.1.0 or does not exist in older
    144 # versions of Python.  We only enable on Python 3.7.4+ or if certificate
    145 # verification is enabled to work around Python issue #37428
    146 # See: https://bugs.python.org/issue37428
    147 if (
    148     cert_reqs == ssl.CERT_REQUIRED or sys.version_info >= (3, 7, 4)
    149 ) and getattr(context, \"post_handshake_auth\", None) is not None:

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
    622 @options.setter
    623 def options(self, value):
--> 624     super(SSLContext, SSLContext).options.__set__(self, value)

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
    622 @options.setter
    623 def options(self, value):
--> 624     super(SSLContext, SSLContext).options.__set__(self, value)

    [... skipping similar frames: SSLContext.options at line 624 (1478 times)]

File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
    622 @options.setter
    623 def options(self, value):
--> 624     super(SSLContext, SSLContext).options.__set__(self, value)

RecursionError: maximum recursion depth exceeded"
}

Expected behavior

It should not matter what order these two libraries are imported.

Platform (please complete the following information):

• OS: Windows
• Python API Version: 2.2.0.1
• Python 3.11.5 (Freshly re-installed to try fix this issue, still doesn't work!)

Additional context

I raised the same issue on the boto3 repo: boto/boto3#3912

[nanaeaubry]

@jtroe or @achapkowski Can you advise?

[achapkowski]

@thehappycheese we do not modify the ssl module at all. It looks like the issue for you is in the botocore library, which powers the boto3 library.

[nateprewitt]

Hi @achapkowski, I added an update to the boto3 ticket. This does seem to be stemming from how arcgis is mutating the ssl module with truststore in Python 3.10+ by calling inject_into_ssl in your arcgis/auth/api.py file. I'll start a thread with Seth to see if this can be improved.

[nateprewitt]

Created sethmlarson/truststore#121. After talking with Seth offline, it seems the current usage in arcgis may not be what's intended. We can use that issue for tracking upstream, but the remedy for this will probably require adjustment to the current logic in arcgis.

It may be useful to reopen this to track that portion.

[sethmlarson]

Indeed this will require some action on arcgis, we weren't clear enough in the docs about who should be using inject_into_ssl() by only hinting at "you must run this asap in your program" and by definition libraries don't control when they run their code since users can rearrange imports arbitrarily and expect things to work.

Arcgis should change its usage from inject_into_ssl to use truststore.SSLContext and then the issue will resolve itself.

[achapkowski]

@sethmlarson we are using truststore for the python 3.10+ version of python, but this issue is for python 3.7-3.9.

[nateprewitt]

I think from the details that were provided above, the original report is for Python 3.11.5.

Python 3.11.5 (Freshly re-installed to try fix this issue, still doesn't work!)

I've been able to reproduce this with just arcgis and urllib3 for all versions of Python 3.10+.

[felixleong]

@achapkowski I can also confirm the issue for Python 3.11.7.

[felixleong]

Having scratched the surface a bit, the situation does seem much more gnarly, and as I understand it:

1. Based on the latest truststore user guide and @sethmlarson's comment above, Truststore advises against using inject_into_ssl()
2. Arcgis's currently uses requests as the library to interact with the API (see arcgis/gis/_impl/_con/_connection.py) and Truststore's user guide doesn't provide an example to use truststore.SSLContext with requests
3. This is due to the fact that requests being resistant into allowing users to use their own SSLContext (dates as far back as 2016, see Let the user provide an SSLContext object psf/requests#2118, Consider using system trust stores by default in 3.0.0. psf/requests#2966)

As an end user of this library and having used arcgis within our Django system, currently the only easy option I have is revert to Python 3.9 for now, though would not be an ideal solution in the long run. Would appreciate some means to have a band-aid hack to maintain at Python 3.11+ if possible.

---

In terms of how arcgis could improve the codebase, my surface level research would lead to two possible solutions:

1. If it were to maintain to use the requests library, the recommended path would be to use TransportAdapters (e.g. requests.adapters.HTTPAdapter - it is to be noted that it may not be fool-proof (see HTTPAdapter with SSLContext specified does not use SSLContext's ca_certs on Windows psf/requests#5316)
2. Possibly need to refactor the code to use another library. urllib3 perhaps?

[martimpassos]

Just checking in to see if there's a fix for this. For now downgrading to Python < 3.10 should work? Can this issue be reopened as it hasn't been solved yet?

[trahloff]

Hi folks, I would like to re-open this issue.

The workaround is only available if you work in highly manual environments. And even if you can apply the workaround, it will introduce unnecessary complexity.

Is it planned to address the underlying root cause?

Minimal reproducible example:

import boto3
from arcgis.gis import GIS

s3_client = boto3.client("s3")

# => RecursionError: maximum recursion depth exceeded while calling a Python object