Skip to content
This repository has been archived by the owner on Aug 27, 2023. It is now read-only.

Summary sent to s3 to be normalized and stripped of non-ASCII characters #197

Closed
wants to merge 1 commit into from
Closed

Summary sent to s3 to be normalized and stripped of non-ASCII characters #197

wants to merge 1 commit into from

Conversation

fizyk
Copy link
Contributor

@fizyk fizyk commented Nov 13, 2018

Some unicode characters might pose problem for boto, which in case
of validators package summary 'Python Data Validation for Humans™'
threw a valdiation error

Stack trace:

Parameter validation failed:
                                        Non ascii characters found in S3 metadata for key "summary", value: "Python Data Validation for Humans™.".  
                                        S3 metadata can only contain ASCII characters. 
                                        Traceback (most recent call last):
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/handlers.py", line 567, in validate_ascii_metadata
                                            value.encode('ascii')
                                        UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in position 33: ordinal not in range(128)
                                        
                                        During handling of the above exception, another exception occurred:
                                        
                                        Traceback (most recent call last):
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/tweens.py", line 39, in excview_tween
                                            response = handler(request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/router.py", line 156, in handle_request
                                            view_name
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/view.py", line 642, in _call_view
                                            response = view_callable(context, request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/config/views.py", line 181, in __call__
                                            return view(context, request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/viewderivers.py", line 390, in attr_view
                                            return view(context, request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/viewderivers.py", line 368, in predicate_wrapper
                                            return view(context, request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid/viewderivers.py", line 439, in rendered_view
                                            result = view(context, request)
                                          File "/opt/pypi/lib/python3.5/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
                                            return fxn(**scope)
                                          File "/opt/pypi/lib/python3.5/site-packages/pypicloud/views/simple.py", line 37, in upload
                                            version=version, summary=summary)
                                          File "/opt/pypi/lib/python3.5/site-packages/pypicloud/cache/base.py", line 118, in upload
                                            self.storage.upload(new_pkg, data)
                                          File "/opt/pypi/lib/python3.5/site-packages/pypicloud/storage/s3.py", line 229, in upload
                                            key.put(Metadata=metadata, Body=datastream, **kwargs)
                                          File "/opt/pypi/lib/python3.5/site-packages/boto3/resources/factory.py", line 520, in do_action
                                            response = action(self, *args, **kwargs)
                                          File "/opt/pypi/lib/python3.5/site-packages/boto3/resources/action.py", line 83, in __call__
                                            response = getattr(parent.meta.client, operation_name)(**params)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/client.py", line 314, in _api_call
                                            return self._make_api_call(operation_name, kwargs)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/client.py", line 586, in _make_api_call
                                            api_params, operation_model, context=request_context)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/client.py", line 619, in _convert_to_request_dict
                                            api_params, operation_model, context)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/client.py", line 648, in _emit_api_params
                                            params=api_params, model=operation_model, context=context)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/hooks.py", line 227, in emit
                                            return self._emit(event_name, kwargs)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
                                            response = handler(**kwargs)
                                          File "/opt/pypi/lib/python3.5/site-packages/botocore/handlers.py", line 575, in validate_ascii_metadata
                                            report=error_msg)
                                        botocore.exceptions.ParamValidationError: Parameter validation failed:
                                        Non ascii characters found in S3 metadata for key "summary", value: "Python Data Validation for Humans™.".  
                                        S3 metadata can only contain ASCII characters.

Context: we're running pypicloud 1.0.5 at the moment on python 3.5, and are uploading other wheels for speedier production/tests installation. (happens only in case we haven't found a wheel)

@fizyk
Copy link
Contributor Author

fizyk commented Nov 13, 2018

relevant information: boto/boto3#478

    Some unicode characters might pose problem for boto, which in case
    of validators package summary 'Python Data Validation for Humans™'
    threw a validation error.
@fizyk fizyk changed the title Send summary to s3 as bytes. Summary sent to s3 to be normalized and stripped of non-ASCII characters Nov 13, 2018
@fizyk
Copy link
Contributor Author

fizyk commented Nov 14, 2018

test failing due to tox/travis issue?

@stevearc
Copy link
Owner

Test failures are because I had to do some weird boto version pinning in the last release. I pulled down the patch and confirmed that it all works. Merged into the 1.0 branch and fix will go out in next release. Thanks for the PR!

@fizyk
Copy link
Contributor Author

fizyk commented Nov 20, 2018

Oh.... there's 1.0 branch.... sorry, haven't noticed it.

@fizyk fizyk closed this Nov 20, 2018
@stevearc
Copy link
Owner

Oh no worries. I was just letting you know so you didn't think I was ignoring the PR. Github doesn't show when it's merged into a non-master branch.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants