Minimal setup to allow testing upload tools integration

[dnicolodi]

Working on twine it would be very helpful to be able to test against a minimal instance of warehouse. AFAIK so far the only easy way to do that is to test against TestPyPI. However, it is reasonable to do that only in manual tests as it is not advisable to spam TestPyPI with uploads from CI jobs.

I had a quick look at what it would take to spin up a minimal instance of warehouse that does legacy API upload processing and very little more. Unfortunately, I am not familiar with the stack used to run warehouse and making sense of the different components may take more time than what I can invest in this. However, I would be happy to spend some time working on this if someone more familiar with the components involved could point me in the right direction.

Testing against a somewhat simplified version of warehouse as part of integration tests on developer machines may require a bit too much setup. Therefore, the goal would be to find a reasonably complex way to spin up a service on Github Actions CI and be able to run integration tests against it. Just checking that the requests are handled without error would be a good first step, being able to check that the upload data is handled as expected checking the data in the warehouse database would be an added bonus.

Would something like this be possible?

[di]

However, it is reasonable to do that only in manual tests as it is not advisable to spam TestPyPI with uploads from CI jobs.

This is kind of the purpose of TestPyPI, so I think it would be OK, but I would advise against this being a long term plan because we eventually want to shut TestPyPI down: #15226.

I had a quick look at what it would take to spin up a minimal instance of warehouse that does legacy API upload processing and very little more.

I think this might be possible by spinning up our developer container image in CI, but I think the hangup might be all the additional containers/services that interact with our main web container: https://warehouse.pypa.io/development/getting-started.html#running-the-warehouse-container-and-services

It's also quite heavyweight: we recommend 4GB of RAM when running Warehouse for development, which might not be available in CI: https://warehouse.pypa.io/development/cloud.html#cloud-server

That said, this project has limited resources and thus is pretty firmly in the business of "being PyPI" and not much else (e.g. we don't support running this project as a local package index, etc.) so I'm not sure how much support we can lend towards adapting the codebase for this use case.

[webknjaz]

FWIW we just use devpi in pypi-publish. It's literally just a single container invocation — https://github.com/pypa/gh-action-pypi-publish/blob/916e576/.github/workflows/reusable-smoke-test.yml#L59-L64.

[dnicolodi]

Thanks @di. Is the reason to shut down TestPyPI driven by misuse or by other considerations? If it is just misuse maybe having the package repository wiped every 24 hours or so would avoid the possibility for most misuse while still allowing the possibility to smoke test uploads. It would also resolve concerns of spamming TestPyPI with uploads from CI.

However, if the plan is to dismiss it, I think finding an easy way to test upload tools with the de-facto standard implementation of the upload service is important. Unfortunately, AFAIK the upload web API is not standardized, thus testing against warehouse is the only way to make sure that things work as they should.

It would be great to have something similar to the developer container image that uses mocked services for the non-essential parts (anything that is not the handling of the upload request and maybe the database). I'll add investigating how feasible this is to my (constantly growing) TODO list 😃

@webknjaz Thanks for the hint. twine already tests against devpi but apparently devpi is much more permissive with validation of the upload data. For example, a recent and seemingly innocuous change in twine broke upload to PyPI but the integration tests against devpi were happy.

[di]

Sorry, I meant to link to #918 here, #15226 is the wrong issue.

The general goal is to have less infrastructure. Additionally, the separation between PyPI/TestPyPI is often confusing for users (they do not share any data) and TestPyPI is often used in an insecure way (trying to install something from TestPyPI might result in installing a subdependency you meant to get from PyPI but instead is controlled by an attacker on TestPyPI) and we provide little to no TestPyPI support or monitoring. Ideally PyPI would serve the needs of current TestPyPI users in the future (e.g. with draft releases).