Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: working POST API for Project Observations #15228

Merged
merged 15 commits into from
Jan 30, 2024

Conversation

miketheman
Copy link
Member

@miketheman miketheman commented Jan 18, 2024

Please review commit-by-commit to make reviewing easier.

Notes:

  • There's a bunch of TODOs to highlight conversations or choices.
  • Does have a GET /danger-api/echo endpoint
  • Does have a POST /danger-api/projects/{name}/observations endpoint that validates the inbound payload (kinda sorta) and persists in the database
  • Does NOT include the database migration to add the new column on the User table - I'll submit that as a separate PR if we want to proceed this way (it's a kludge). Here: feat: migration for User.is_observer #15238
  • Does NOT leverage Macaroon Caveats - so a User-scoped API Token from a user that is_observer will be allowed to POST. This is not a good long-term solution, but can likely work well enough to gather some data.
  • Does NOT include any tests yet - I wanted to get some structural approaches done here first. Tests! But they rely on the migration PR being merged first.

@miketheman miketheman added needs discussion a product management/policy issue maintainers and users should discuss APIs/feeds labels Jan 18, 2024
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch from ae9c8e8 to 45d016c Compare January 23, 2024 18:01
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch from 45d016c to 14aec56 Compare January 23, 2024 19:49
Copy link
Member

@ewdurbin ewdurbin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really nice and straightforward PR thanks to a lot of previous legwork. Thanks for that @miketheman.

Overall I will be confident in approving this for merge once we have a couple of things inplace:

  • Sufficiently annoying/obvious signs to users that we are not committing to any API ergonomics or features. Think "danger" and "unstable" and "preview" and "dragons here" type verbiage in versions and or urls.
  • A confidence in our malware observation payloads such that we feel comfortable that they will be actionable immediately, and not too likely to be discarded or useless long term. I see that [DRAFT] PyPI Observation Reporting Payload #14503 has some good updates on approaches here. Thanks for that.
  • A side-effect to committing a malware observation that creates a conversation in the PyPI Security inbox. This ensures that any reporters that adopt the API won't have their reports "lost". It will integrate this effort into our current pipeline fairly seamlessly! I recommend attaching the observation as a .json attachment given...
  • Some documentation that observations associated with deleted objects will also be deleted until we implement soft-deletes.

warehouse/api/echo.py Outdated Show resolved Hide resolved
warehouse/routes.py Outdated Show resolved Hide resolved
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch 2 times, most recently from f5336f9 to 4551d1c Compare January 23, 2024 23:55
Copy link
Member Author

@miketheman miketheman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review!

I've addressed the first couple of bullets so far, and will continue to work on the next two.

warehouse/api/echo.py Outdated Show resolved Hide resolved
warehouse/routes.py Outdated Show resolved Hide resolved
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch 2 times, most recently from 767420a to c4185e9 Compare January 26, 2024 20:41
@miketheman miketheman removed the needs discussion a product management/policy issue maintainers and users should discuss label Jan 26, 2024
@miketheman
Copy link
Member Author

This is pretty much ready to go, while I work out some more details in a follow up, unless reviewers see them as merge blockers:

  • add danger-api endpoint for Releases
  • add Observations and API to Files, Users

Keep in mind: Only superusers or folks we've set User.is_observer flag will be able to post with a user-scoped API Token, until we do more on the caveats/permissions front (soon).

@miketheman miketheman marked this pull request as ready for review January 26, 2024 20:48
@miketheman miketheman requested a review from a team as a code owner January 26, 2024 20:48
Migration to ship separately, before this code is active.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Includes many TODOs, but works.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
So that it's clearer these are dangerous.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch from c4185e9 to a6f76fa Compare January 29, 2024 14:30
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
@miketheman miketheman force-pushed the miketheman/api-basic-observation branch from a6f76fa to 5a5acea Compare January 29, 2024 14:35
Copy link
Member

@ewdurbin ewdurbin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I am very impressed/happy with the iteration on our API infrastructure/machinery here!

I have one concern regarding local development that I think would be wise to consider before merge just so that the feature can be worked on by anyone, not just those of us with HS access.

warehouse/api/echo.py Show resolved Hide resolved
warehouse/observations/tasks.py Show resolved Hide resolved
warehouse/routes.py Show resolved Hide resolved
A simple bail-out clause that prints instead of makes HTTP calls if the
user hasn't provided credentials.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
@miketheman miketheman merged commit 68c1db9 into pypi:main Jan 30, 2024
17 checks passed
@miketheman miketheman deleted the miketheman/api-basic-observation branch January 30, 2024 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants