Packaging for Nix #391

chinyeungli · 2025-10-14T05:52:44Z

Fixes #339

Generates a Nix expression (default.nix) from the pyproject.toml file that is used to build the Python package for NixOS.

Once the default.nix is generated, one can build/install the package by
using:

Build the package

nix-build default.nix

The above command will create a symlink named result in the current
directory pointing to the build output in the Nix store.
Run the binary directly

./result/bin/dejacode

Signed-off-by: Chin Yeung Li tli@nexb.com

Signed-off-by: Chin Yeung Li <tli@nexb.com>

- Added --generate and --test options - Updated code structure/comments Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez · 2025-10-29T10:00:51Z

@chinyeungli mockldap and funcparserlib were removed in #419
Could you update this branch to remove the workaround related to those libs?

tdruez

$ python build_nix_docker.py
nix-prefetch-url is NOT installed.

The nix-prefetch-url requirement is not documented.

Also, you can remove all references to funcparserlib.

chinyeungli · 2025-10-30T22:50:39Z

Yes. I am on it. For some reasons, it's not building at the moment and I am working on it.

- Updated documentation for the installation process - Added special handling for "python_ldap", which requires specific dependencies when building from source - Refactored and updated helper functions Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli · 2025-10-31T06:06:28Z

@tdruez script updated.

tdruez

I was able to use the script to generate the Nix package.
Could you add a short section in the script doc on how to run the app from this build?

chinyeungli · 2025-11-03T06:54:56Z

@tdruez
one can run the built binary directly with ./result/bin/dejacode

https://github.com/aboutcode-org/dejacode/pull/391/files#diff-c3f471d98a33714e40e5449d317af18dea952c483ac5db9ee41911d01436a08eR68

but it only shows the CLI

$ ./result/bin/dejacode

Type 'dejacode help <subcommand>' for help on a specific subcommand.

Available subcommands:

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    migrate
    optimizemigration
    runserver
    sendtestemail
    shell
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver
Note that only Django core commands are listed as settings are not properly configured (error: Set the SECRET_KEY environment variable).

Should the binary be designed to launch the web server automatically when executed? Similar to what docker compose up does, but without relying on Docker?

tdruez · 2025-11-03T08:33:36Z

@chinyeungli I should be able to run the test suite to validate the package.
Although there are missing packages:

$ SECRET_KEY=secret ./result/bin/dejacode test

Traceback (most recent call last):
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/db/backends/postgresql/psycopg_any.py", line 5, in <module>
    from psycopg import ClientCursor, IsolationLevel, adapt, adapters, errors, sql
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/__init__.py", line 9, in <module>
    from . import pq  # noqa: F401 import early to stabilize side effects
    ^^^^^^^^^^^^^^^^
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/pq/__init__.py", line 116, in <module>
    import_from_libpq()
    ~~~~~~~~~~~~~~~~~^^
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/pq/__init__.py", line 108, in import_from_libpq
            raise ImportError(
    ...<4 lines>...
            )
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: libpq library not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/4r5wwvk1n5rpks6vm1df4nlfhlkxb3j8-dejacode/bin/.dejacode-wrapped", line 9, in <module>
    sys.exit(command_line())
             ~~~~~~~~~~~~^^
  File "/nix/store/4r5wwvk1n5rpks6vm1df4nlfhlkxb3j8-dejacode/lib/python3.13/site-packages/dejacode/__init__.py", line 83, in command_line
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
    ~~~~~~~~~~~~^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
  File "/nix/store/cfapjd2rvqrpry4grb0kljnp8bvnvfxz-python3-3.13.8/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1027, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/contrib/postgres/apps.py", line 4, in <module>
    from django.db.backends.postgresql.psycopg_any import RANGE_TYPES
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/db/backends/postgresql/psycopg_any.py", line 77, in <module>
    from psycopg2 import errors, extensions, sql  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'psycopg2'

…at runtime #339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli · 2025-11-04T04:58:28Z

@tdruez I've updated the script and able to run the test with the following steps

Create and configure a PostgreSQL user named dejacode

psql -h localhost -U postgres -c "CREATE USER dejacode; ALTER USER dejacode CREATEDB; GRANT ALL PRIVILEGES ON DATABASE dejacode TO dejacode;" 2>/dev/null || psql -h localhost -U postgres -c "ALTER USER dejacode CREATEDB; GRANT ALL PRIVILEGES ON DATABASE dejacode TO dejacode;"

Sets up environment variables and runs a test

PGPASSWORD="password" PYTHONPATH=/home/user/project/dejacode SECRET_KEY=secret ./result/bin/dejacode test

Replace the "password" with the PostgreSQL password
Replace the PYTHONPATH with where the sources located

Signed-off-by: Chin Yeung Li <tli@nexb.com>

pombredanne · 2025-11-04T13:29:00Z

etc/scripts/build_nix_docker.py

+                    if component.get("packagetype") == "bdist_wheel":
+                        whl_url = component.get("url")
+                        if (
+                            ("cp313" not in whl_url and "py3" not in whl_url)


You may want to look at https://github.com/aboutcode-org/python-inspector/blob/2a9f1287f33790cefa9f3e9599861f67b5305c85/src/python_inspector/utils_pypi.py#L1118 for tag matching ....

…lity #339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez · 2025-11-18T10:10:50Z

etc/scripts/build_nix_docker.py

+                )
+                nix_content += "        };\n"
+                nix_content += "    };\n"
+            elif name == "python_ldap" and version == "3.4.5":


This would looks like a pain to maintain, what happens when python_ldap version is updated?

refactored the code at e1a8ba9

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez

I'm still getting the psycopg2 module error on Ubuntu 22.04.5 LTS:

Traceback (most recent call last):
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/db/backends/postgresql/psycopg_any.py", line 5, in <module>
    from psycopg import ClientCursor, IsolationLevel, adapt, adapters, errors, sql
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/__init__.py", line 9, in <module>
    from . import pq  # noqa: F401 import early to stabilize side effects
    ^^^^^^^^^^^^^^^^
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/pq/__init__.py", line 116, in <module>
    import_from_libpq()
    ~~~~~~~~~~~~~~~~~^^
  File "/nix/store/ykw9fd32pnfpsri98145wacqi2qlwhn2-python3.13-psycopg-3.2.10/lib/python3.13/site-packages/psycopg/pq/__init__.py", line 108, in import_from_libpq
            raise ImportError(
    ...<4 lines>...
            )
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: libpq library not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/dzjq5n06jjk826yjxjgvlhxzs36vnwf7-dejacode/bin/.dejacode-wrapped", line 9, in <module>
    sys.exit(command_line())
             ~~~~~~~~~~~~^^
  File "/nix/store/dzjq5n06jjk826yjxjgvlhxzs36vnwf7-dejacode/lib/python3.13/site-packages/dejacode/__init__.py", line 83, in command_line
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
    ~~~~~~~~~~~~^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
  File "/nix/store/cfapjd2rvqrpry4grb0kljnp8bvnvfxz-python3-3.13.8/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1027, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/contrib/postgres/apps.py", line 4, in <module>
    from django.db.backends.postgresql.psycopg_any import RANGE_TYPES
  File "/nix/store/qpc72vbqazyciaa9ia81gg5d961r2ir3-python3.13-Django-5.2.7/lib/python3.13/site-packages/django/db/backends/postgresql/psycopg_any.py", line 77, in <module>
    from psycopg2 import errors, extensions, sql  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'psycopg2'

tdruez · 2025-11-19T05:05:16Z

etc/scripts/build_nix_docker.py

+    if "cp313" in url:
+        tags.add("cp313")
+    if "py3" in url:
+        tags.add("py3")


The Python version should be extracted from the pyproject.toml
In general, everything that is currently hardcoded should be extracted from there so the script keeps working after upgrading the Python version or any dependency version.

tdruez · 2025-11-19T05:06:25Z

etc/scripts/build_nix_docker.py

+    # Parse arguments
+    args = parser.parse_args()
+
+    if args.generate or not Path("default.nix").exists():


Path("default.nix").exists()
Report a clear error message because if you re-run the script, you get no feedback and are left to figure out why.

- Better error handling - Reduce the hardcoded code Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

…339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli · 2025-11-20T12:03:55Z

@tdruez
e016700 should resolve the psycopg2 issue

The script no longer requires setting PYTHONPATH to run tests. However, you still need to initialize the database user with:

psql -h localhost -U postgres -c "CREATE USER dejacode; ALTER USER dejacode CREATEDB; GRANT ALL PRIVILEGES ON DATABASE dejacode TO dejacode;" 2>/dev/null || psql -h localhost -U postgres -c "ALTER USER dejacode CREATEDB; GRANT ALL PRIVILEGES ON DATABASE dejacode TO dejacode;"

Then run the tests using:

PGPASSWORD="your_password" SECRET_KEY=secret ./result/bin/dejacode test

Make sure to replace "your_password" with the actual password for the dejacode user.

chinyeungli added 5 commits October 9, 2025 18:05

Created default.nix for build-nix packaging #339

dc303a0

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Add Docker script to run build_nix and update default.nix comments #339

ae2cb2c

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Remove default.nix and add script to generate the default.nix #339

86aca15

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Reformat the code to comply with Ruff's check #339

ed52e44

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Move build_nix_docker.py to etc/scripts #339

201f435

- Added --generate and --test options - Updated code structure/comments Signed-off-by: Chin Yeung Li <tli@nexb.com>

Merge branch 'main' into 339_packaging_for_nix

85a02e5

tdruez requested changes Oct 30, 2025

View reviewed changes

Remove "funcparserlib" references #339

ec8c81a

- Updated documentation for the installation process - Added special handling for "python_ldap", which requires specific dependencies when building from source - Refactored and updated helper functions Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez requested changes Nov 3, 2025

View reviewed changes

Use architecture-independent wheels and confirm libpq is available …

6c27141

…at runtime #339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

Specify the wheel file built for the x86_64 architecture #339

6df3455

Signed-off-by: Chin Yeung Li <tli@nexb.com>

pombredanne reviewed Nov 4, 2025

View reviewed changes

chinyeungli added 2 commits November 5, 2025 17:34

Implement a better approach for conditional check for wheel compatibi…

269073d

…lity #339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

Code reformatted #339

b83542f

Signed-off-by: Chin Yeung Li <tli@nexb.com>

aboutcode-org deleted a comment from Josemaldonadodiosmasmalo Nov 18, 2025

tdruez reviewed Nov 18, 2025

View reviewed changes

Refactor code to better handle "python_ldap" #339

e1a8ba9

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez requested changes Nov 19, 2025

View reviewed changes

chinyeungli added 3 commits November 20, 2025 00:58

Refactor code #339

c92f830

- Better error handling - Reduce the hardcoded code Signed-off-by: Chin Yeung Li <tli@nexb.com>

Remove non-used code and updated hardcoded code #339

863898e

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Fix the "psycopg" issue and make tests run without setting PYTHONPATH #…

e016700

…339 Signed-off-by: Chin Yeung Li <tli@nexb.com>

Uh oh!

Packaging for Nix #391

Are you sure you want to change the base?

Packaging for Nix #391

Uh oh!

Conversation

chinyeungli commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tdruez commented Oct 29, 2025

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

chinyeungli commented Oct 30, 2025

Uh oh!

chinyeungli commented Oct 31, 2025

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

chinyeungli commented Nov 3, 2025

Uh oh!

tdruez commented Nov 3, 2025

Uh oh!

chinyeungli commented Nov 4, 2025

Create and configure a PostgreSQL user named dejacode

Sets up environment variables and runs a test

Uh oh!

pombredanne Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

tdruez Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

chinyeungli Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

tdruez Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

tdruez Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

chinyeungli commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chinyeungli commented Oct 14, 2025 •

edited

Loading