Wrapper around smart_open to cache transport_params for every call #335
Conversation
There is no nice way how to pass transport_params to the reader and writer The user has to call SmartOpen.setup separately
oplatek
force-pushed
the
transport-params-smart-open-everywhere
branch
from
b623b3b to
c75832d
Compare
lhotse/utils.py
Outdated
| client = session.client('s3') | ||
| from lhotse.utils import SmartOpen | ||
|
|
||
| if not slow: # switch between 44s vs 18.9 Wall time |
There was a problem hiding this comment.
Sorry to be nit-picking -- this 44s vs 18.9s seems very specific to your use-case, could we remove it?
There was a problem hiding this comment.
Also -- can you format it like:
Example::
>>> import boto3
>>> session = boto3.Session()
...
This will render well with sphinx.
There was a problem hiding this comment.
Can you pls check that it is ok now?
Or just tell me how to generate (and check) the docs myself.
There was a problem hiding this comment.
The auto generated docs for this PR are here https://lhotse--335.org.readthedocs.build/en/335/
RTD used to add it as a CI test result in GH but the integration broke when I moved Lhotse to lhotse-speech organization… and I couldn’t fix it.
Unfortunately it seems that the API reference doesn’t list lhotse.utils so you won’t see it there anyway.. I haven’t gotten to re-working this part of the docs yet.
|
Looking good, thanks! |
Motivation
The
smart_open.openfunction when used with default value setups all necessary parameters to access e.g. S3 clouduri.However, it typically means that it creates a session e.g.
boto3.sessionwithclient('s3')for AWS S3 access.Creating such a session for every call is slow.
If we reused a single session for AWS S3 access for 1000 calls the Wall time was reduced from 44s to 18.9s.
This PR introduces SmartOpen singleton class
which caches transport_params and thus e.g. the
client('s3')In addition, it imports smart_open.open only once and does not import it in every call.