weblate: init at 4.13.1, add module #169797

erictapen · 2022-04-22T14:03:10Z

Description of changes

Back in the Summer of Nix 2021 I created a Flake for Weblate: https://github.com/ngi-nix/weblate

Now I want to merge this into Nixpkgs in order to increase discoverability and likelihood of maintenance (even though I kept the flake uptodate since the summer and also plan to do so in the future.

Older attempt: #48726

Things done

erictapen · 2022-07-25T13:31:45Z

Hi @wiredhikari, thanks for your review. Could you add a bit on what exactly you checked about the code? Did you just look at it or did you also run the tests, try the installation?

Answering that might help other people to determine wether this is mergeable.

wiredhikari · 2022-07-26T04:08:06Z

I went through the code and tests, seemed fine to me

RaitoBezarius

Can I do something to help you move this PR forward?

RaitoBezarius · 2022-08-21T11:12:39Z

nixos/modules/services/web-apps/weblate.nix

+  environment = {
+    PYTHONPATH = "${settings_py}";
+    DJANGO_SETTINGS_MODULE = "settings";
+    GI_TYPELIB_PATH = "${pkgs.pango.out}/lib/girepository-1.0:${pkgs.harfbuzz}/lib/girepository-1.0";


makeLibPath [ pkgs.pango pkgs.harfbuzz ]

lib.makeLibraryPath [ pkgs.pango pkgs.harfbuzz ] produces /nix/store/a83690v9dsx2l2bafc1bqs17nsa8x4i8-pango-1.50.8/lib:/nix/store/jr685bs5jjs3rb9r297hma145wc5jbsn-harfbuzz-5.1.0/lib, but I think we need the girepository-1.0 bit to be included.

This should be ideally done by wrapper (e.g. gobject-introspection + wrapGAppsNoGuiHook) in the package itself so that it works outside the context of the module: https://nixos.org/manual/nixpkgs/stable/#ssec-gnome-typelibs

You're right @erictapen

I'm currently trying to move the the gobject-introspection and PATH logic into the package, but I'm having a hard time properly doing that inside mkPoetryApplication and buildPythonApplication. Will try some more and report back.

But note that we can't completely remove the wrapping, as the settings.py is generated in the module. So the package itself will probably never be of much use outside the module.

If it's a Django app, we can still rewire DJANGO_SETTINGS_MODULE to something else (I have already done it multiple times)

RaitoBezarius · 2022-08-21T11:13:19Z

nixos/modules/services/web-apps/weblate.nix

+
+        locations = {
+          "= /favicon.ico".alias = "${pkgs.weblate}/lib/${pkgs.python3.libPrefix}/site-packages/weblate/static/favicon.ico";
+          "/static/".alias = "${pkgs.weblate}/lib/${pkgs.python3.libPrefix}/site-packages/weblate/static/";


I think static should be refactored in the module system and let the default be the path you're using.

I'm not sure I understand your proposition? The / route must already be served by the uwsgi unix domain socket, so it's necessary to explicitly route only /static/.

@erictapen

I meant it would be nice to have pkgs.weblate.static which references to ${pkgs.weblate}/lib/${pkgs.python3.libPrefix}/site-packages/weblate/static/ and have in the module system cfg.staticPackage which points to pkgs.weblate.static and here you would reference ${cfg.staticPackage}/favicon.ico for example.

So (1) someone can access to the static part of Weblate (here, it means to build the weblate derivation, so few benefits) (2) someone can replace the static part of Weblate with a patch or something easily.

Now I get what you are aiming at, but I don't see the benefits yet.

Is this pattern used anywhere else in Nixpkgs you could point me to?

Is there any existing replacement for the static part of Weblate? I don't see the use case.

A rapid grep shows at least:

pkgs/servers/misc/navidrome/default.nix 26: ui = callPackage ./ui { pkgs/servers/gotify/default.nix 34: ui = callPackage ./ui.nix { };

I imagine those are more involved, as they are most likely Node-generated static files.
Maybe static is not a good name as it could get confused with a statically-linked binary from what I see in nixpkgs, but ui is often used.

Is there any existing replacement for the static part of Weblate? I don't see the use case.

There may be situations where you want to do pkgs.runCommand "patch-static" ''cp -r ${pkgs.weblate.static} $out && do something with $out to patch it, e.g. a robots.txt, etc. and pass it to cfg.staticPackage rather than patching the whole pkgs.weblate.

I have no strong feeling on this, it can be added later on if anyone has a real need for it, but most of the time, I see that people would appreciate having override options rather than locking them into a let expression ; in this case, overlaying the original weblate package would do the job as long as it is a reasonable patch.

erictapen · 2022-09-01T14:49:17Z

Can I do something to help you move this PR forward?

There are no open issues from my side with this one, so reviewing it helps definetly! Thanks.

nixos/modules/services/web-apps/weblate.nix

jtojnar · 2022-09-01T19:41:48Z

nixos/modules/services/web-apps/weblate.nix

+    DATABASES = {
+      "default": {
+        "ENGINE": "django.db.backends.postgresql",
+        "HOST": "/run/postgresql",


Maybe we can use sockets here as well so that we do not need user and password.

Hm, sockets are used here?

Yes, the unix socket is already used. I removed the PASSWORD and PORT option to not cause any confusion.

jtojnar · 2022-09-01T19:44:29Z

nixos/modules/services/web-apps/weblate.nix

+  environment = {
+    PYTHONPATH = "${settings_py}";
+    DJANGO_SETTINGS_MODULE = "settings";
+    GI_TYPELIB_PATH = "${pkgs.pango.out}/lib/girepository-1.0:${pkgs.harfbuzz}/lib/girepository-1.0";


This should be ideally done by wrapper (e.g. gobject-introspection + wrapGAppsNoGuiHook) in the package itself so that it works outside the context of the module: https://nixos.org/manual/nixpkgs/stable/#ssec-gnome-typelibs

jtojnar · 2022-09-01T19:47:52Z

nixos/modules/services/web-apps/weblate.nix

+    GI_TYPELIB_PATH = "${pkgs.pango.out}/lib/girepository-1.0:${pkgs.harfbuzz}/lib/girepository-1.0";
+  };
+  weblate-env = pkgs.writeShellScriptBin
+    "weblate-env"


Actually, is this used anywhere other then tests?

It would be nice to expose the django-admin stuff so that someone can do weblate-admin createsuperuser or whatever.

@jtojnar Yeah you would use it whenever you have to run admin tasks, as I see no other way to put the correct settings.py into the PYTHONPATH.

Maybe make settings.py in the package load /etc/weblate/settings.py or something?

What would be a nice thing in the future would be to have a "package with a setting", i.e. cfg.weblate.withSettings (...) and the settings would be baked into it.

But why would one want to run the package outside NixOS in the first place?
Upstreams intention seems to be, that you download the git repository on your server, run a few install scripts and then edit the settings.py by hand to your likings. Before you edit the file, their software isn't usable as well. I don't see why we should offer a way to use Weblate outside of a deployment framework, when upstream doesn't.

Precisely to be able to do admin tasks when needed, without having to use weird weblate-env runner. That is also a thing going out of way of upstream but a one much less convenient to use.

But why would one want to run the package outside NixOS in the first place? Upstreams intention seems to be, that you download the git repository on your server, run a few install scripts and then edit the settings.py by hand to your likings. Before you edit the file, their software isn't usable as well. I don't see why we should offer a way to use Weblate outside of a deployment framework, when upstream doesn't.

As said, no strong feeling on this. It helps a lot to manage multi-instances, diagnose existing setups, etc.
It can be done in a later PR if anyone has a need for it.

jtojnar · 2022-09-01T19:48:46Z

nixos/modules/services/web-apps/weblate.nix

+    ((pkgs.lib.concatStrings (pkgs.lib.mapAttrsToList (n: v: "export ${n}=${v}\n") environment)) + ''
+      eval -- "\$@"
+    '');
+  weblatePath = with pkgs; [


Again, this should be already on PATH, in the package.

See #169797 (comment)

jtojnar · 2022-09-01T19:56:35Z

nixos/modules/services/web-apps/weblate.nix

+        User = "postgres";
+        Group = "postgres";
+        ExecStart = ''
+          ${pkgs.postgresql}/bin/psql weblate -c "CREATE EXTENSION IF NOT EXISTS pg_trgm"


This looks pretty invasive but not sure what else to do. According to https://www.postgresql.org/docs/current/pgtrgm.html, it might not be necessary to run it as postgres user.

Yes, it is enough to add CREATE in ensurePermissions and run this under the weblate user.

(Do note that it may provide access to the postgres user anyway because if someone can do arbitrary CREATE EXTENSION, they may be able to load their own ext and do bad stuff, the postgres solution is not that bad.)

Didn't look further into it, but if the situation is like the two of you described I'd rather execute known code once with higher privileges, than to give the weblate user more privileges for potentially malicious code?

Yes, I think letting as-is, is better.

jtojnar · 2022-09-01T20:03:16Z

nixos/modules/services/web-apps/weblate.nix

+        "postgresql.service"
+      ];
+      environment = environment // {
+        CELERY_WORKER_RUNNING = "1";


pkgs/servers/web-apps/weblate/default.nix

jtojnar · 2022-09-01T20:08:34Z

pkgs/servers/web-apps/weblate/default.nix

+  pyproject = ./pyproject.toml;
+  poetrylock = ./poetry.lock;


Can we upstream these?

I never tried it, but I'd like to, yes. Could imagine upstream not to be very willing to change their build system / maintain a second one though.

Maybe open an issue explaining the needs for lock files first?

Co-authored-by: Jan Tojnar <jtojnar@gmail.com>

… socket anyway

This reverts commit ad4f421.

This reverts commit bace881.

…the package" This reverts commit 706438c.

erictapen · 2023-11-09T13:45:01Z

I just learned, that poetry2nix was dropped from Nixpkgs.
That would make it infeasible to add Weblate support to NixOS through this PR. I'd try to continue maintaining the Flake instead.

erictapen requested a review from jtojnar April 22, 2022 14:03

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Apr 22, 2022

ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Apr 22, 2022

wiredhikari approved these changes Jul 22, 2022

View reviewed changes

erictapen force-pushed the weblate branch from c9349c8 to 2f49fe2 Compare August 3, 2022 12:54

erictapen changed the title ~~weblate: init at 4.12, add module~~ weblate: init at 4.13.1, add module Aug 3, 2022

erictapen added the 8.has: module (new) This PR adds a module in `nixos/` label Aug 3, 2022

erictapen force-pushed the weblate branch from 2f49fe2 to 584fd8d Compare August 22, 2022 15:57

RaitoBezarius reviewed Sep 1, 2022

View reviewed changes

jtojnar reviewed Sep 1, 2022

View reviewed changes

erictapen and others added 13 commits September 1, 2022 23:14

weblate: init at 4.12

d4d0b30

nixos/weblate: init from https://github.com/ngi-nix/weblate

c1642f2

weblate: 4.12 -> 4.13

b9081db

weblate: 4.12 -> 4.13.1

df9a8c6

weblate: 4.13.1 -> 4.14

1c82142

nixos/weblate: use proper attribute to get site-packages path

811ccf7

Co-authored-by: Jan Tojnar <jtojnar@gmail.com>

nixos/weblate: remove unneeded code

f5fdb8b

weblate: formatting, better comment

a33c2e7

weblate: move setting of GI_TYPELIB_PATH into package

ad4f421

weblate: remove outdated poetry overrides

7b1a4e7

nixos/weblate: remove postgresql password and port as we use the unix…

0b20e7f

… socket anyway

weblate: use necessary packages in path from the module into the package

706438c

Revert "weblate: move setting of GI_TYPELIB_PATH into package"

c0d78a2

This reverts commit ad4f421.

erictapen added 3 commits September 2, 2022 01:51

weblate: some wip fixup for moving path into the package

bace881

Revert "weblate: some wip fixup for moving path into the package"

9974691

This reverts commit bace881.

Revert "weblate: use necessary packages in path from the module into …

58e74f1

…the package" This reverts commit 706438c.

erictapen force-pushed the weblate branch from 7a426a3 to 58e74f1 Compare September 2, 2022 11:49

weblate: 4.14 -> 4.14.1

26aa22f

erictapen closed this Nov 9, 2023

erictapen mentioned this pull request Jul 8, 2024

weblate: init at 5.6.2, add module #325541

Merged

13 tasks

weblate: init at 4.13.1, add module #169797

weblate: init at 4.13.1, add module #169797

Conversation

erictapen commented Apr 22, 2022

Description of changes

Things done

erictapen commented Jul 25, 2022

wiredhikari commented Jul 26, 2022

RaitoBezarius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictapen Sep 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictapen commented Sep 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictapen Sep 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictapen commented Nov 9, 2023

erictapen Sep 2, 2022 •

edited

Loading

erictapen Sep 2, 2022 •

edited

Loading