Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

binwalk, diffoscope: update after Python 3.12 #325623

Closed
wants to merge 2 commits into from

Conversation

philiptaron
Copy link
Contributor

@philiptaron philiptaron commented Jul 8, 2024

Description of changes

related to #325657 (comment)
cc @Mag1cByt3s

Result of nixpkgs-review run on x86_64-linux 1

15 packages built:
  • binwalk (python312Packages.binwalk)
  • binwalk.dist (python312Packages.binwalk.dist)
  • diffoscope
  • diffoscope.dist
  • diffoscope.man
  • diffoscopeMinimal
  • diffoscopeMinimal.dist
  • diffoscopeMinimal.man
  • ghidra-extensions.ghidraninja-ghidra-scripts
  • python311Packages.binwalk
  • python311Packages.binwalk-full
  • python311Packages.binwalk-full.dist
  • python311Packages.binwalk.dist
  • python312Packages.binwalk-full
  • python312Packages.binwalk-full.dist

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@philiptaron
Copy link
Contributor Author

@trofi fyi

@emilazy
Copy link
Member

emilazy commented Jul 8, 2024

Is that behavioural change in binwalk benign? I’m curious as to why a Python upgrade and an import internals change would lead to it finding more binaries.

@philiptaron
Copy link
Contributor Author

Is that behavioural change in binwalk benign? I’m curious as to why a Python upgrade and an import internals change would lead to it finding more binaries.

I was quite surprised to see it as well. You can see from the patch what the detected differences are.

I checked out upstream (https://salsa.debian.org/reproducible-builds/diffoscope) locally and ran master's copy of binwalk (for me, /nix/store/ckv7akdj4cjl8qkiij8vraiisw9n6mvj-python3.11-binwalk-2.3.4):

$ /nix/store/ckv7akdj4cjl8qkiij8vraiisw9n6mvj-python3.11-binwalk-2.3.4/bin/binwalk tests/data/test{1,2}.binwalk

Scan Time:     2024-07-08 13:07:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test1.binwalk
MD5 Checksum:  03bb6be017bd9db1586ee5cc291dd992
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
356           0x164           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
924           0x39C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
1900          0x76C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
2692          0xA84           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"


Scan Time:     2024-07-08 13:07:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test2.binwalk
MD5 Checksum:  d47335a3e695dc2b3dd8bfc56ba476a5
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
364           0x16C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
1156          0x484           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
1892          0x764           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
2460          0x99C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"

Then, with this PR (note the Python 3.12 in the derivation name):

$ /nix/store/2c7kkgy4hbvxjh1rj0cvcvc0r1yb136l-python3.12-binwalk-2.3.4/bin/binwalk tests/data/test{1,2}.binwalk

Scan Time:     2024-07-08 13:08:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test1.binwalk
MD5 Checksum:  03bb6be017bd9db1586ee5cc291dd992
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
356           0x164           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
924           0x39C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
1900          0x76C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
2692          0xA84           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"


Scan Time:     2024-07-08 13:08:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test2.binwalk
MD5 Checksum:  d47335a3e695dc2b3dd8bfc56ba476a5
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
364           0x16C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
1156          0x484           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
1892          0x764           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
2460          0x99C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"

What I don't understand is how there were ever two emitted items. Master appears to have 10 as well.

def test_listing(comparison):
differences = comparison.details
- assert comparison.comments == ["comprises of 2 embedded members"]
+ assert comparison.comments == ["comprises of 10 embedded members"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this by design i.e. a number that we can rely on to continue to be 10? or might this change in the future?

@dotlambda dotlambda removed the backport release-24.05 Backport PR automatically label Jul 10, 2024
@dotlambda dotlambda marked this pull request as draft July 10, 2024 14:23
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-nose-1-3-7-not-supported-for-interpreter-python3-12/48703/22

@emilazy emilazy mentioned this pull request Jul 11, 2024
13 tasks
@superherointj
Copy link
Contributor

superherointj commented Jul 11, 2024

Please rebase. (And whatever new improvement/fix that isn't in 326295 can be integrated.)

@eclairevoyant
Copy link
Contributor

eclairevoyant commented Jul 11, 2024

Both binwalk and diffoscope appear to build on current master (6bf4ef3)

@superherointj
Copy link
Contributor

Both binwalk and diffoscope appear to build on current master (6bf4ef3)

Yes, because I merged #326295.

@emilazy
Copy link
Member

emilazy commented Jul 11, 2024

But there’s nothing to rebase now; #326295 replaces this.

@eclairevoyant

This comment was marked as duplicate.

@philiptaron
Copy link
Contributor Author

But there’s nothing to rebase now; #326295 replaces this.

Quite so.

@philiptaron philiptaron deleted the fix-python-3.12/binwalk branch July 13, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants