Prevent duplicate `POST` keys in requests #16732

di · 2024-09-17T21:05:25Z

Fixes https://python-software-foundation.sentry.io/share/issue/40756e6481274d3babdc620efac508a9/

Generally, duplicate POST keys in a request are an error for us. Rather than sanity-checking this everywhere we use request.POST, this PR adds a view deriver that returns an error if duplicate keys are present.

di · 2024-09-17T21:15:16Z

warehouse/forklift/legacy.py

@@ -487,6 +487,7 @@ def _sort_releases(request: Request, project: Project):
    require_csrf=False,
    require_methods=["POST"],
    has_translations=True,
+    permit_duplicate_post_keys=True,


As far as I'm aware, this is the only view where we actually depend on and permit duplicate keys.

sdb9696 · 2024-09-19T11:21:42Z

Hi there, I think this PR has somehow broken the twine --skip-existing feature.

When I'm publishing to testpypi and I know there's a file there already, I am getting the error message POST body may not contain duplicate keys from this PR instead of the error message twine expects : A file named "{0}" already exists for'.format(filename).

I can't say 100% that it's this PR but it's the message I'm now getting and I can publish successfully when I bump the project version.

N.B. I run a build and test publish after every push to master as per the Python Packaging user guide but only really care if the test publish succeeds just before a proper release. This now causes the CI to fail on the interim commits.

MOmarMiraj · 2024-09-19T13:21:59Z

Hi guys,

Also running into this error as I am trying to upload some packages on TestPyPi and no matter what I do (version bump, change name, new project, etc) its giving me this POST body may not contain duplicate keys. Was working yesterday but now its not working no matter what I do.

di · 2024-09-19T13:29:05Z

Thanks! Looking into this now.

marcinn · 2024-09-19T13:30:52Z

Same for me:

INFO     Response from https://upload.pypi.org/legacy/:                                                                                                                                                            
         400 Bad Request                                                                                                                                                                                           
INFO     <html>                                                                                                                                                                                                    
          <head>                                                                                                                                                                                                   
           <title>400 Bad Request</title>                                                                                                                                                                          
          </head>                                                                                                                                                                                                  
          <body>                                                                                                                                                                                                   
           <h1>400 Bad Request</h1>                                                                                                                                                                                
           The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>                                                                                             
         POST body may not contain duplicate keys                                                                                                                                                                  
                                                                                                                                                                                                                   
                                                                                                                                                                                                                   
          </body>                                                                                                                                                                                                  
         </html>                                                                                                                                                                                                   
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/                                                                                                                                           
         Bad Request

twine version 5.1.1 (importlib-metadata: 8.5.0, keyring: 25.4.0, pkginfo: 1.10.0, requests: 2.32.3, requests-toolbelt: 1.0.0, urllib3: 2.2.3)

command:
twine upload --verbose --skip-existing dist/*

MOmarMiraj · 2024-09-19T13:34:00Z

Hey so actually I figured out my reason which was an issue in my setup.py. This has been resolved for me now and I am confident that it actually didn't have to do with anything here but just had the same error.

Sorry for that!

di · 2024-09-19T13:38:54Z

@MOmarMiraj Can you give us some more detail about what the issue in your setup.py was?

marcinn · 2024-09-19T14:24:45Z

I have classifiers defined. This is a part of a dump of data_to_send var (in twine's repository.upload()):

    ('classifiers', 'Development Status :: 4 - Beta'),
    ('classifiers', 'Environment :: Web Environment'),
    ('classifiers', 'Framework :: Django'),
    ('classifiers', 'Framework :: Django :: 3.1'),
    ('classifiers', 'Framework :: Django :: 3.2'),
    ('classifiers', 'Framework :: Django :: 4.0'),
    ('classifiers', 'License :: OSI Approved :: ISC License (ISCL)'),
    ('classifiers', 'Operating System :: OS Independent'),
    ('classifiers', 'Programming Language :: Python'),
    ('classifiers', 'Programming Language :: Python :: 3.9'),
    ('classifiers', 'Programming Language :: Python :: 3.10'),
    ('classifiers', 'Programming Language :: Python :: 3.11'),
    ('classifiers', 'Topic :: Internet :: WWW/HTTP'),
    ('classifiers', 'Topic :: Software Development :: Libraries :: Python Modules'),
    ('classifiers', 'Intended Audience :: Developers'),

After removing classifiers the error disappeared (now pypi complains about not enabled 2FA)

di · 2024-09-19T16:34:49Z

I still can't reproduce this myself but I've merged #16755 which might help, can anyone reporting issues here retry and see if this is still happening?

MOmarMiraj · 2024-09-19T16:41:05Z

@MOmarMiraj Can you give us some more detail about what the issue in your setup.py was?

In my setup.py it was the following:

    classifiers = [
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "License  :: MIT License"
    ],

The license was missing the OSI Approved section and I was getting that error. The error message should've been more clear but this is what fixed mines

di · 2024-09-19T17:22:32Z

We've identified the issue, #16759 will fix this once it's merged & deployed.

sdb9696 · 2024-09-19T17:26:45Z

I still can't reproduce this myself but I've merged #16755 which might help, can anyone reporting issues here retry and see if this is still happening?

To reproduce just try uploading a file with the same name and different hash twice should do it.

sdb9696 · 2024-09-19T17:36:19Z

Edited the reproduction steps above to specify using a different hash.

sdb9696 · 2024-09-19T18:20:31Z

Just tested with the latest deployment, all fixed many thanks!

marcinn · 2024-09-19T18:34:25Z

Seems to be ok now, thanks

suzukimain · 2024-09-20T02:18:20Z

It worked. Thank you!

davidlougheed · 2024-09-24T20:05:33Z

is anyone else still getting this error where they didn't before? for me with Maturin - worked last night, but today seems to have stopped working.

EDIT: looks to have been fixed in Maturin 1.7.3: https://github.com/PyO3/maturin/releases/tag/v1.7.3

di added 7 commits September 17, 2024 19:32

Add form IDs and update functional test

17fd990

Reformat comment

6d7df1f

Remove unnecessary cast to MultiDict

b5c0bf1

Add a failing test

dc97860

Add view deriver to prohibit duplicate POST keys

cd90cfd

Permit duplicate POST keys for file_upload view

9e8819c

Update translations

5be4bc9

di requested a review from a team as a code owner September 17, 2024 21:05

di commented Sep 17, 2024

View reviewed changes

ewdurbin approved these changes Sep 18, 2024

View reviewed changes

Merge branch 'main' into prevent-multiple-post-keys

4bf4e66

di enabled auto-merge (squash) September 18, 2024 15:09

di merged commit a908ffd into pypi:main Sep 18, 2024
18 checks passed

di deleted the prevent-multiple-post-keys branch September 18, 2024 15:16

di mentioned this pull request Sep 18, 2024

Ensure formdata gets a multidict #16741

Merged

This was referenced Sep 19, 2024

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/ pypa/twine#713

Closed

Debug issues with duplicate POST keys #16755

Merged

TimoDiepers mentioned this pull request Sep 19, 2024

Broken deploy workflow brightway-lca/dynamic_characterization#4

Closed

thibmonsel mentioned this pull request Sep 19, 2024

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ Bad Request for Github Actions on Test PyPI #16756

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent duplicate `POST` keys in requests #16732

Prevent duplicate `POST` keys in requests #16732

di commented Sep 17, 2024

di Sep 17, 2024

sdb9696 commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024

di commented Sep 19, 2024

marcinn commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024

di commented Sep 19, 2024

marcinn commented Sep 19, 2024 •

edited

Loading

di commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024 •

edited

Loading

di commented Sep 19, 2024

sdb9696 commented Sep 19, 2024 •

edited

Loading

sdb9696 commented Sep 19, 2024

sdb9696 commented Sep 19, 2024 •

edited

Loading

marcinn commented Sep 19, 2024

suzukimain commented Sep 20, 2024

davidlougheed commented Sep 24, 2024 •

edited

Loading

Prevent duplicate POST keys in requests #16732

Prevent duplicate POST keys in requests #16732

Conversation

di commented Sep 17, 2024

di Sep 17, 2024

Choose a reason for hiding this comment

sdb9696 commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024

di commented Sep 19, 2024

marcinn commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024

di commented Sep 19, 2024

marcinn commented Sep 19, 2024 • edited Loading

di commented Sep 19, 2024

MOmarMiraj commented Sep 19, 2024 • edited Loading

di commented Sep 19, 2024

sdb9696 commented Sep 19, 2024 • edited Loading

sdb9696 commented Sep 19, 2024

sdb9696 commented Sep 19, 2024 • edited Loading

marcinn commented Sep 19, 2024

suzukimain commented Sep 20, 2024

davidlougheed commented Sep 24, 2024 • edited Loading

Prevent duplicate `POST` keys in requests #16732

Prevent duplicate `POST` keys in requests #16732

marcinn commented Sep 19, 2024 •

edited

Loading

MOmarMiraj commented Sep 19, 2024 •

edited

Loading

sdb9696 commented Sep 19, 2024 •

edited

Loading

sdb9696 commented Sep 19, 2024 •

edited

Loading

davidlougheed commented Sep 24, 2024 •

edited

Loading