Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Double Encoding of URLs #91

Open
thorge opened this issue Sep 4, 2023 · 0 comments · May be fixed by #92
Open

Double Encoding of URLs #91

thorge opened this issue Sep 4, 2023 · 0 comments · May be fixed by #92

Comments

@thorge
Copy link

thorge commented Sep 4, 2023

Issue Description:

Currently, the CKAN database stores resource URLs in percent-encoded form. When the archiver extension attempts to download a file using these URLs, there's an unintended double percent-encoding that can lead to download errors.

The problematic code responsible for this issue is located in the tidy_url() function within the tasks.py file:

# ckanext/archiver/tasks.py:653

# Find out if it has unicode characters, and if it does, quote them
# so we are left with an ASCII string
try:
    url = url.decode('ascii')
except Exception:
    parts = list(urlparse(url))
    parts[2] = quote(parts[2].encode('utf-8'))
    url = urlunparse(parts)
url = str(url)

In Python 3, the attempt to decode the URL into ASCII is unnecessary and, in fact, causes the except block to be applied resulting in the double encoding issue. Therefore, the entire try-except block for ASCII handling can be safely removed in Python 3 environments.

Furthermore, it's crucial to prioritize upgrading the archiver extension to Python 3 as soon as possible to eliminate other py2 issues and ensure compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant