Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pywb: Init at 2.7.4 #189950

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

pywb: Init at 2.7.4 #189950

wants to merge 4 commits into from

Conversation

anpandey
Copy link
Contributor

@anpandey anpandey commented Sep 6, 2022

Description of changes

pywb described as a "full-featured, advanced web archiving capture and replay framework for python". More info can be found at the webrecorder homepage.

This is partially based on #52814, which was never merged. I've verified on x86_64 Linux that web archive playback and recording works with the following commands:

wb-manager init test-archive
wayback --live --record --enable-auto-fetch -a -t 8

and navigating to the UI at localhost:8080. I did not test any of the redis-based features.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Sep 6, 2022
@anpandey anpandey force-pushed the pywb2 branch 2 times, most recently from db52433 to 114916c Compare September 19, 2022 06:05
@anpandey anpandey marked this pull request as draft September 21, 2022 18:16
@anpandey anpandey force-pushed the pywb2 branch 2 times, most recently from 7e4c882 to 1e4dd94 Compare September 29, 2022 02:58
@anpandey anpandey marked this pull request as ready for review October 2, 2022 14:53
Copy link
Member

@pbsds pbsds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impressive work! This truly must have been a lot of effort.

The nixpkgs way is generally to relax the dependency requirement, then only override versions if really needed.

pkgs/development/python-modules/py3amf/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Show resolved Hide resolved
pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
@anpandey anpandey force-pushed the pywb2 branch 3 times, most recently from 932bc9a to 37a4f40 Compare September 25, 2023 06:00
pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved
@anpandey anpandey changed the title pywb: Init at 2.6.8 pywb: Init at 2.7.4 Sep 26, 2023
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please submit this patch upstream, and then fetch it with fetchpatch?
We'd also love to have an explanation of what it does and why it is needed. Something to the tune of "support updated jinja" i assume

pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
@@ -12054,6 +12054,8 @@ with pkgs;

pywal = with python3Packages; toPythonApplication pywal;

pywb = callPackage ../tools/misc/pywb {};
Copy link
Member

@pbsds pbsds Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pywb = callPackage ../tools/misc/pywb {};
pywb = python3.pkgs.callPackage ../tools/misc/pywb {};

Please also avoid using with python3.pkgs; inside the derivation as it is considered an anti-pattern. Use the function arguments at the top of the derivation file instead. This makes the derivation more overridable

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

pkgs/tools/misc/pywb/default.nix Outdated Show resolved Hide resolved
--replace "gevent==21.12.0" "gevent"
'';

propagatedBuildInputs = with python3.pkgs; [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These don't seem to match upstream. pytest for example should usually not be propagated. Many maintainers in nixpkgs prefer to sort the propagated inputs the same way as upstream, making it easier to spot differences. If any packages have been added that were not referenced in upstream then

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extras is commonly listed in passthru.optional-dependencies

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump, please don't propagate pytest

Copy link
Member

@pbsds pbsds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builds and runs fine, but some of my prior comments has not been addressed nor argued

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a pythonImportsCheck?

--replace "gevent==21.12.0" "gevent"
'';

propagatedBuildInputs = with python3.pkgs; [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump, please don't propagate pytest

fetchSubmodules = true;
};

patches = [ ./pywb.patch ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment. Please comment why this is needed and why it is not submitted upstream

@@ -12054,6 +12054,8 @@ with pkgs;

pywal = with python3Packages; toPythonApplication pywal;

pywb = callPackage ../tools/misc/pywb {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

@pbsds
Copy link
Member

pbsds commented Oct 9, 2023

seems like my "bump" comments don't show context unless you're in the files view

@anpandey anpandey marked this pull request as draft October 9, 2023 10:11
@anpandey
Copy link
Contributor Author

anpandey commented Oct 9, 2023

Sorry about that - I think I finished the review round when I tried to merge the suggestions via the UI. I'll unmark this as a draft once it's ready for review.

Co-authored-by: Peder Bergebakken Sundt <pbsds@hotmail.com>
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Nov 9, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: python 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 11.by: package-maintainer This PR was created by the maintainer of the package it changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants