-
-
Notifications
You must be signed in to change notification settings - Fork 382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use warnings.warn instead of logger.warning #321
Changes from all commits
6433b62
88f213f
948838b
c57ae73
9a85d05
88618e0
beb3857
5877193
5485c86
69fa8de
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -255,9 +255,114 @@ Since going over all (or select) keys in an S3 bucket is a very common operation | |
>>> prefix = 'annual/monthly_rain/' | ||
>>> for key, content in s3_iter_bucket(bucket, prefix=prefix, accept_key=lambda key: '/201' in key, workers=1, key_limit=3): | ||
... print(key, round(len(content) / 2**20)) | ||
annual/monthly_rain/2010.monthly_rain.nc 14 | ||
annual/monthly_rain/2011.monthly_rain.nc 14 | ||
annual/monthly_rain/2012.monthly_rain.nc 14 | ||
annual/monthly_rain/2010.monthly_rain.nc 13 | ||
annual/monthly_rain/2011.monthly_rain.nc 13 | ||
annual/monthly_rain/2012.monthly_rain.nc 13 | ||
|
||
|
||
Migrating to the New `open` Function | ||
------------------------------------ | ||
|
||
Since 1.8.1, there is a `smart_open.open` function that replaces `smart_open.smart_open`. | ||
The new function offers several advantages over the old one: | ||
|
||
- 100% compatible with the built-in `open` function (aka `io.open`): it accepts all | ||
the parameters that the built-in `open` accepts. | ||
- The default open mode is now "r", the same as for the built-in `open`. | ||
The default for the old `smart_open.smart_open` function used to be "rb". | ||
- Fully documented keyword parameters (try `help("smart_open.open")`) | ||
|
||
The instructions below will help you migrate to the new function painlessly. | ||
|
||
First, update your imports: | ||
|
||
.. code-block:: python | ||
|
||
>>> from smart_open import smart_open # before | ||
>>> from smart_open import open # after | ||
|
||
In general, `smart_open` uses `io.open` directly, where possible, so if your | ||
code already uses `open` for local file I/O, then it will continue to work. | ||
If you want to continue using the built-in `open` function for e.g. debugging, | ||
then you can `import smart_open` and use `smart_open.open`. | ||
|
||
**The default read mode is now "r" (read text) by default.** | ||
If your code was implicitly relying on the default mode being "rb" (read | ||
binary), then you'll need to update it and pass "r" explicitly. | ||
|
||
Before: | ||
|
||
.. code-block:: python | ||
|
||
>>> buf = b'' | ||
>>> buf += smart_open('s3://commoncrawl/robots.txt').read(32) | ||
|
||
After: | ||
|
||
.. code-block:: python | ||
|
||
>>> buf = b'' | ||
>>> buf += open('s3://commoncrawl/robots.txt', 'rb').read(32) | ||
|
||
The `ignore_extension` keyword parameter is now called `ignore_ext`. | ||
It behaves identically otherwise. | ||
|
||
The most significant change is in the handling on keyword parameters for the | ||
transport layer, e.g. HTTP, S3, etc. The old function accepted these directly: | ||
|
||
.. code-block:: python | ||
|
||
>>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
>>> session = boto3.Session(profile_name='smart_open') | ||
>>> smart_open(url, 'r', session=session).read(32) | ||
'first line\nsecond line\nthird lin' | ||
|
||
The new function accepts a `transport_params` keyword argument. It's a dict. | ||
Put your transport parameters in that dictionary. | ||
|
||
.. code-block:: python | ||
|
||
>>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
>>> session = boto3.Session(profile_name='smart_open') | ||
>>> params = {'session': session} | ||
>>> open(url, 'r', transport_params=params).read(32) | ||
'first line\nsecond line\nthird lin' | ||
|
||
Renamed parameters: | ||
|
||
- `s3_upload` ➡ `multipart_upload_kwargs` | ||
- `s3_session` ➡ `session` | ||
|
||
Removed parameters: | ||
|
||
- `profile_name` | ||
|
||
**The `profile_name` parameter has been removed.** | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that's just the way github renders RST. I don't know what we can do about that. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Aaah, this is RST, not Markdown! That explains it. Literal blocks in RST use double-backticks IIRC, not single-backtick. Do we need to use RST though? MD easier? Or is it something to do with PyPI relying on RST… There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it's because of PyPI |
||
Pass an entire boto3.Session object instead. | ||
|
||
Before: | ||
|
||
.. code-block:: python | ||
|
||
>>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
>>> smart_open(url, 'r', profile_name='smart_open').read(32) | ||
'first line\nsecond line\nthird lin' | ||
|
||
After: | ||
|
||
.. code-block:: python | ||
|
||
>>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
>>> session = boto3.Session(profile_name='smart_open') | ||
>>> params = {'session': session} | ||
>>> open(url, 'r', transport_params=params).read(32) | ||
'first line\nsecond line\nthird lin' | ||
|
||
See `help("smart_open.open")` for the full list of acceptable parameter names, | ||
or view the help online `here <https://github.com/RaRe-Technologies/smart_open/blob/master/help.txt>`__. | ||
|
||
If you pass an invalid parameter names, the `open` function will warn you about it. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. typo:
|
||
Keep an eye on your logs for WARNING messages from smart_open. | ||
|
||
Comments, bug reports | ||
--------------------- | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -380,9 +380,23 @@ def open( | |||||
} | ||||||
|
||||||
|
||||||
_MIGRATION_NOTES_URL = ( | ||||||
'https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst' | ||||||
'#migrating-to-the-new-open-function' | ||||||
) | ||||||
|
||||||
|
||||||
def smart_open(uri, mode="rb", **kw): | ||||||
"""Deprecated, use smart_open.open instead.""" | ||||||
logger.warning('this function is deprecated, use smart_open.open instead') | ||||||
"""Deprecated, use smart_open.open instead. | ||||||
|
||||||
See the migration instructions: %s | ||||||
|
||||||
""" % _MIGRATION_NOTES_URL | ||||||
|
||||||
warnings.warn( | ||||||
'This function is deprecated, use smart_open.open instead. ' | ||||||
'See the migration notes for details: %s' % _MIGRATION_NOTES_URL | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
) | ||||||
|
||||||
# | ||||||
# The new function uses a shorter name for this parameter, handle it separately. | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mpenkov what happened here? I remember commenting on this line in a review, and you fixing it. But now it's back in the
master
README.