Stop publishing to PyPI

[chadwhitacre]

Is anyone using the https://pypi.org/project/sentry/ package?

We would like to stop publishing it for the following reasons:

1. To move build stuff out of setup.py to make the development flow less painful.
2. To make it less likely that setuptools mysteriously breaks us (like 64.0.0 did).

Migration Options

If you have been depending on the sentry PyPI package, here are your options:

1. Install directly from GitHub.
2. Switch to self-hosted.

To Do

• Yank existing versions.
• Stop publishing to PyPI sentry#53980

[chadwhitacre]

@asottile-sentry Can you elaborate on (1) and (2) for us? It's not obvious to me the connection between publishing to PyPI and these two reasons. 🐭

[asottile-sentry]

for (1) a lot of our javascript build and packaging depends on setuptools command which are deprecated. not only that, invoking them through pip is fairly slow and difficult to debug. splitting the parts out of setup.py into actual build scripts would give us more control over how things run without having to debug setuptools internals

as far as I can tell from the history, being pypi-installable was how we used to deploy sentry but that's no longer the case -- it gets very few public downloads and presumably are all either mistakes for sentry-sdk or are mirrors / bots: https://pypistats.org/packages/sentry

as for the setuptools breakage -- the js build and layout currently makes a bunch of assumptions about disk layout and the changes in setuptools 64 break those assumptions (and the build)

if we could stop publishing it would eliminate a bunch of the complexity around the build, have some ~small speedup to development setup time, and potentially allow us to reorganize our dockerfiles to build faster

---

my proposal for deprecation is:

1. "yank" all of the versions of sentry on pypi with some sort of deprecation message (this will display when someone attempts to install things) -- this is entirely reversible and we can workshop the message here
2. simply stop publishing to pypi for future releases

[chadwhitacre]

If we yank and people install will they still be able to install but with our deprecation message, or (as I suspect) will they not be able to install at all? What alternative would we guide them towards?

[asottile-sentry]

the last time I tried this the following happens when you yank every version:

pip install whatever==1.2.3  # succeeds, shows message
pip install whatever>=1.2.3  # errors, no versions avaialable
pip install whatever  # errors, no versions available

[asottile-sentry]

for example:

$ pip install --no-deps  -i https://test.pypi.org/simple aspy.yaml
Looking in indexes: https://test.pypi.org/simple
ERROR: Could not find a version that satisfies the requirement aspy.yaml (from versions: 0.2.0, 1.3.0)
ERROR: No matching distribution found for aspy.yaml
$ pip install --no-deps  -i https://test.pypi.org/simple aspy.yaml>=1.3.0
ERROR: Could not find a version that satisfies the requirement aspy.yaml (from versions: 0.2.0, 1.3.0)
ERROR: No matching distribution found for aspy.yaml
$ pip install --no-deps  -i https://test.pypi.org/simple aspy.yaml==1.3.0
Looking in indexes: https://test.pypi.org/simple
Collecting aspy.yaml==1.3.0
  Downloading https://test-files.pythonhosted.org/packages/99/ce/78be097b00817ccf02deaf481eb7a603eecee6fa216e82fa7848cd265449/aspy.yaml-1.3.0-py2.py3-none-any.whl (3.5 kB)
WARNING: The candidate selected for download or install is a yanked version: 'aspy-yaml' candidate (version 1.3.0 at https://test-files.pythonhosted.org/packages/99/ce/78be097b00817ccf02deaf481eb7a603eecee6fa216e82fa7848cd265449/aspy.yaml-1.3.0-py2.py3-none-any.whl#sha256=463372c043f70160a9ec950c3f1e4c3a82db5fca01d334b6bc89c7164d744bdc (from https://test.pypi.org/simple/aspy-yaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*))
Reason for being yanked: hi there
Installing collected packages: aspy.yaml
Successfully installed aspy.yaml-1.3.0

[chadwhitacre]

Interesting. So we could guide people to use the pinned version as their workaround. Seems reasonable. If we wanted to be really gentle with this we could continue publishing packages for a while (three months?) but update our automation to immediately yank after publishing.

What's the long-term migration path for anyone still using this package? Switch to self-hosted? Install from GitHub?

[asottile-sentry]

either of those, yes -- I actually don't think the pypi package is even usable on its own right now 😆

[chadwhitacre]

How much space do we get for the warning message? What does the input look like? Output?

[asottile-sentry]

in the above example the "hi there" is the yank message -- afaict there's no limit on it: https://github.com/pypi/warehouse/blob/a3d6221d4c692569e2290f6d75a75b4db747335b/warehouse/packaging/models.py#L468

[chadwhitacre]

Copy draft. Thoughts?

Warning

This is for if we go the publish-and-immediately-yank route for a time. We would update the pin version in the tl;dr.

Greetings, earthling. 👋 You're one of a kind. We didn't think anyone was
actually using this package, but you, apparently, are. Sorry for making your
life harder, but we want to stop publishing this package in order to make our
life easier. Check this GitHub ticket for rationale and alternatives:

    https://github.com/getsentry/self-hosted/issues/1654

tl;dr Pin the version and you should be good: `sentry == 22.8.0`.

We will stop publishing new versions of this package after 22.12.0.

Error

Greetings, earthling. 👋 You're one of a kind. We didn't think anyone was
actually using this package, but you, apparently, are. Sorry for making your
life harder, but we have stopped publishing this package in order to make our
lives easier. Check this GitHub ticket for rationale and alternatives:

    https://github.com/getsentry/self-hosted/issues/1654

[emmatyping]

The data on pypistats.org seem to indicate that non-mirror traffic is pretty low, almost nothing:

I'm not really sure what the peaks are but I believe they are noise, one of them coincides with a peak of installs from Windows which seems.. not likely to be someone running sentry 😁

[asottile-sentry]

I'm not sure we'll have much control over the newlines and whitespace -- and we only get one message for the warning case but otherwise looks good

[chadwhitacre]

we only get one message for the warning case

Meaning we can't update the message with the new version numbers as they come out?

[asottile-sentry]

we get an error message per package so we can use the right one for each

[chadwhitacre]

Okay it sounds like we've gone far enough with this for now, and we can revisit during the next calver release. Ya?

[asottile-sentry]

oh I want to set the yank messages now so that we're ready to turn it off for the next calver release :D

[chadwhitacre]

From @Disservin in Discord, who uses the PyPI package:

the firm im working at wishes to switch from rollbar to sentry to deploy on premise solutions for customers but some customers dont want to have docker

[chadwhitacre]

Where do we enter the message? Is it after I click "Yank"? I'm afraid to click it. 😅

Also, there are 354 versions of sentry. I presume you're scripting this, @asottile-sentry? Do you have that ready?

[chadwhitacre]

I added brief notes to the ticket description about migration options.

Is there an unyank operation? Is it possible to change the yank message after yanking once?

[asottile-sentry]

yeah the message will be entered after that -- yanking is entirely reversible which is why we're approaching from that angle rather than deleting. the message can be changed yes (idk if there's a modify but you can unyank and reyank)

I can write up a script for this 👍

as for using the current pypi bundle, I'm curious how because as far as I can tell it doesn't have all the parts needed to execute

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[chadwhitacre]

I logged into PyPI and used Chrome's Copy > as cURL feature to prepare the script below. I copied and pasted from PyPI's web UI to get a list of version numbers and then cleaned it up in Vim as all-versions.txt. Yay for :'<,'>!tac. :)

#!/usr/bin/env zsh
for version in $(cat all-versions.txt); do
  echo "🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺🌺"
  echo "Yanking $version ..."
  curl "https://pypi.org/manage/project/sentry/release/$version/" \
    -H 'authority: pypi.org' \
    -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
    -H 'accept-language: en-US,en;q=0.9' \
    -H 'cache-control: max-age=0' \
    -H 'content-type: application/x-www-form-urlencoded' \
    -H 'cookie: nerp' \
    -H 'origin: https://pypi.org' \
    -H 'referer: https://pypi.org/manage/project/sentry/releases/' \
    -H 'sec-ch-ua: "Not/A)Brand";v="99", "Google Chrome";v="115", "Chromium";v="115"' \
    -H 'sec-ch-ua-mobile: ?0' \
    -H 'sec-ch-ua-platform: "macOS"' \
    -H 'sec-fetch-dest: document' \
    -H 'sec-fetch-mode: navigate' \
    -H 'sec-fetch-site: same-origin' \
    -H 'sec-fetch-user: ?1' \
    -H 'upgrade-insecure-requests: 1' \
    -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36' \
    --data-raw 'csrf_token=derp&yanked_reason=https%3A%2F%2Fgithub.com%2Fgetsentry%2Fself-hosted%2Fissues%2F1654&confirm_yank_version='"$version" \
    --compressed
done;

[chadwhitacre]

Done.