Netbox 3.6.8 broke something on saml ?

[CTV-2023]

Hi,

I'm trying to debug an issue after upgrading from 3.6.7 to 3.7
EDIT : the issue appeared with 3.6.8
My SAML configuration is broken, nginx returns 502 Bad Gateway (local auth is OK).
nginx error log tells me something is happening on upstream (gunicorn ?)

2024/01/01 13:14:02 [error] 361213#0: *20 upstream prematurely closed connection while reading response header from upstream, client: W.X.Y.Z, server: netbox-ing.acme.com, request: "POST /oauth/complete/saml/ HTTP/2.0", upstream: "http://127.0.0.1:8001/oauth/complete/saml/", host: "netbox-ing.acme.com", referrer: "https://login.microsoftonline.com/"

The authentication itself is OK on EntraID (Azure AD) side. Is there a way to diagnose this ? Django debug mode doesn't seem to give me more information
I'll try to restore the VM tomorrow and upgrade from 3.6.7 to 3.6.8 and 3.6.9, maybe the issue appeared before 3.7. I didn't see any changes related to SSO/auth backend in release notes (and breaking changes)

The issue seems random but frequent.
It seems I don't see the request with "tcpdump -i lo -nn -s0 -A tcp port 8001" when the issue occurs

[candlerb]

Please can you also try downgrading to 3.6.7. i.e. re-run ./upgrade.sh so it rebuilds the virtualenv from scratch.

It's possible that some dependency outside of Netbox has changed. Netbox doesn't pin its dependencies, so what you get when you install today may be different to what you got yesterday, even on the same version of Netbox.

EDIT: on the working system try /opt/netbox/venv/bin/pip freeze >pip-working.out and then compare that with what you get on a non-working system. This will show you if any dependencies have changed.

[jeremystretch]

Netbox doesn't pin its dependencies

This is incorrect. NetBox does pin its direct dependencies, and has for some time. We don't freeze the entire virtual environment because that has created problems in the past, and instead rely on each required package to accurately specify its own dependencies.

[CTV-2023]

EDIT : I restored the VM to 3.6.7 and upgraded manually to 3.6.8 (our custom custom script upgraded to "latest" <= note the custom still uses default upgrade.sh script after some changes) to notice the issue was still present. Then I upgraded from 3.6.8 to 3.7.0

pip freeze differences between integration (3.7) and production (3.6.7)

diff pip-working-3.6.7.out pip-not-working-3.7.out
5c5
< attrs==23.1.0
---
> attrs==23.2.0
28c28
< django-rq==2.9.0
---
> django-rq==2.10.1
30c30
< django-taggit==4.0.0
---
> django-taggit==5.0.1
44c44
< importlib-metadata==7.0.0
---
> importlib-metadata==7.0.1
50c50
< jsonschema-specifications==2023.11.2
---
> jsonschema-specifications==2023.12.1
52,53c52,53
< lxml==4.9.3
< Markdown==3.3.7
---
> lxml==5.0.0
> Markdown==3.5.1
60c60
< mkdocs-material==9.5.2
---
> mkdocs-material==9.5.3
69c69
< ntc_templates==4.0.1
---
> ntc_templates==4.1.0
80,81c80,81
< psycopg==3.1.15
< psycopg-binary==3.1.15
---
> psycopg==3.1.16
> psycopg-binary==3.1.16
87c87
< pymdown-extensions==10.4
---
> pymdown-extensions==10.7
100c100
< regex==2023.10.3
---
> regex==2023.12.25
104c104
< rpds-py==0.15.2
---
> rpds-py==0.16.2
107d106
< sentry-sdk==1.39.1

I modified my gunicorn config file to add debug logs

[2024-01-02 13:26:22 +0100] [3937] [DEBUG] GET /oauth/login/saml/
[2024-01-02 13:26:22 +0100] [3937] [DEBUG] Closing connection.
[2024-01-02 13:26:22 +0100] [3915] [DEBUG] POST /oauth/complete/saml/
[2024-01-02 13:26:23 +0100] [3382] [ERROR] Worker (pid:3915) was sent code 139!
[2024-01-02 13:26:23 +0100] [4070] [INFO] Booting worker with pid: 4070
[2024-01-02 13:26:23 +0100] [3927] [DEBUG] GET /favicon.ico
[2024-01-02 13:26:23 +0100] [3927] [DEBUG] Closing connection.
[2024-01-02 13:26:23 +0100] [3937] [DEBUG] GET /login/
[2024-01-02 13:26:23 +0100] [3937] [DEBUG] Closing connection.

EDIT 2 : code 139 = segmentation fault on gunicorn side ?

[candlerb]

If you could add the -u flag to your diff command, and also insert three backticks on a line of their own (```) before and after the text you paste, that would make it much easier to read.

[CTV-2023]

Sorry

root@xxx:~# diff -u pip-working-3.6.7.out pip-not-working-3.7.out
--- pip-working-3.6.7.out       2024-01-02 13:38:42.293479607 +0100
+++ pip-not-working-3.7.out     2024-01-02 13:29:10.305581075 +0100
@@ -2,7 +2,7 @@
 asgiref==3.7.2
 astunparse==1.6.3
 async-timeout==4.0.3
-attrs==23.1.0
+attrs==23.2.0
 Babel==2.14.0
 backports.zoneinfo==0.2.1
 bcrypt==4.1.2
@@ -25,9 +25,9 @@
 django-prometheus==2.3.1
 django-redis==5.4.0
 django-rich==1.8.0
-django-rq==2.9.0
+django-rq==2.10.1
 django-tables2==2.7.0
-django-taggit==4.0.0
+django-taggit==5.0.1
 django-timezone-field==6.1.0
 djangorestframework==3.14.0
 drf-spectacular==0.27.0
@@ -41,23 +41,23 @@
 graphql-relay==3.2.0
 gunicorn==21.2.0
 idna==3.6
-importlib-metadata==7.0.0
+importlib-metadata==7.0.1
 importlib-resources==6.1.1
 inflection==0.5.1
 isodate==0.6.1
 Jinja2==3.1.2
 jsonschema==4.20.0
-jsonschema-specifications==2023.11.2
+jsonschema-specifications==2023.12.1
 junos-eznc==2.6.8
-lxml==4.9.3
-Markdown==3.3.7
+lxml==5.0.0
+Markdown==3.5.1
 markdown-it-py==3.0.0
 MarkupSafe==2.1.3
 mdurl==0.1.2
 mergedeep==1.3.4
 mkdocs==1.5.3
 mkdocs-autorefs==0.5.0
-mkdocs-material==9.5.2
+mkdocs-material==9.5.3
 mkdocs-material-extensions==1.3.1
 mkdocstrings==0.24.0
 mkdocstrings-python-legacy==0.2.3
@@ -66,7 +66,7 @@
 netaddr==0.9.0
 netmiko==4.3.0
 netutils==1.6.0
-ntc_templates==4.0.1
+ntc_templates==4.1.0
 oauthlib==3.2.2
 packaging==23.2
 paginate==0.5.6
@@ -77,14 +77,14 @@
 platformdirs==4.1.0
 prometheus-client==0.19.0
 promise==2.3
-psycopg==3.1.15
-psycopg-binary==3.1.15
+psycopg==3.1.16
+psycopg-binary==3.1.16
 psycopg-pool==3.2.0
 pycparser==2.21
 pyeapi==1.0.2
 Pygments==2.17.2
 PyJWT==2.8.0
-pymdown-extensions==10.4
+pymdown-extensions==10.7
 PyNaCl==1.5.0
 pyparsing==3.1.1
 pyserial==3.5
@@ -97,14 +97,13 @@
 pyyaml_env_tag==0.1
 redis==5.0.1
 referencing==0.32.0
-regex==2023.10.3
+regex==2023.12.25
 requests==2.31.0
 requests-oauthlib==1.3.1
 rich==13.7.0
-rpds-py==0.15.2
+rpds-py==0.16.2
 rq==1.15.1
 scp==0.14.5
-sentry-sdk==1.39.1
 sgmllib3k==1.0.0
 six==1.16.0
 social-auth-app-django==5.4.0

[candlerb]

I don't see anything obvious there that would break SAML, although perhaps lxml changing from 4 to 5 is suspicious.

You could use the pip freeze output from the working Netbox instance to build the 3.6.8 instance (temporarily replace requirements.txt with the good pip freeze output) and see if it works. That would prove conclusively whether it was a change in Netbox itself, or a change in dependencies.

EDIT: for lxml 5 changes see https://pypi.org/project/lxml/

[CTV-2023]

I'll restore again the VM and don't upgrade it to 3.7, which makes harder to diagnose between versions. I think I needed libxml2-devel for python-saml. But as this one was not updated, maybe another package needed lxml updated to version 5 ?

[CTV-2023]

Just in case that helps

 dmesg | grep gunicorn
[ 5350.896759] gunicorn[2958]: segfault at 7fbf3321ccec ip 00007fb8c7640778 sp 00007fb8cf8fc030 error 4 in libxml2.so.2.9.7[7fb8c75e0000+15d000]
[ 5358.653725] gunicorn[2962]: segfault at 0 ip 00007fb8dde4b6dc sp 00007fb8cf8fa280 error 6 in libpython3.8.so.1.0[7fb8ddc69000+33a000]
[ 5365.635774] gunicorn[2986]: segfault at 288 ip 00007fb8c7640778 sp 00007fb8cf8fc030 error 4 in libxml2.so.2.9.7[7fb8c75e0000+15d000]
[ 5371.789691] traps: gunicorn[2984] trap stack segment ip:7fb8c794a2b0 sp:7fb8cf8fbf70 error:0 in libxml2.so.2.9.7[7fb8c784d000+15d000]
[ 5378.164000] traps: gunicorn[3007] general protection fault ip:7fb8c7640778 sp:7fb8cf8fbfe0 error:0 in libxml2.so.2.9.7[7fb8c75e0000+15d000]
[ 5444.708715] gunicorn[2964]: segfault at 29 ip 00007fb8c7336f46 sp 00007fb8cf8fccc0 error 4 in etree.cpython-38-x86_64-linux-gnu.so[7fb8c7120000+331000]
[ 5448.233907] traps: gunicorn[3017] general protection fault ip:7fb8c78ad778 sp:7fb8cf8fc030 error:0 in libxml2.so.2.9.7[7fb8c784d000+15d000]
[ 5579.459901] traps: gunicorn[2997] general protection fault ip:7fb8ddd782cd sp:7fb8cf8f73a0 error:0 in libpython3.8.so.1.0[7fb8ddc69000+33a000]
[ 6170.998775] traps: gunicorn[3218] trap stack segment ip:7fe4242832b0 sp:7fe4285fdf70 error:0 in libxml2.so.2.9.7[7fe424186000+15d000]
[ 6175.281977] traps: gunicorn[3220] general protection fault ip:7fe43687a2cd sp:7fe4285fc650 error:0 in libpython3.8.so.1.0[7fe43676b000+33a000]
[ 6182.080652] gunicorn[3241]: segfault at fa ip 00007fe41fd952b0 sp 00007fe4285fdf20 error 4 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6188.420128] traps: gunicorn[3217] general protection fault ip:7fe41fcf8778 sp:7fe4285fdfe0 error:0 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6194.623528] traps: gunicorn[3260] trap stack segment ip:7fe41fd952b0 sp:7fe4285fdf70 error:0 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6200.679531] gunicorn[3270]: segfault at 9db5cfce ip 00007fe41fcf8778 sp 00007fe4285fdfe0 error 4 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6328.665881] gunicorn[3212]: segfault at 8125c8cd ip 00007fe4271f29ac sp 00007fe4285fe028 error 4 in libxml2.so.2.9.7[7fe427146000+15d000]
[ 6335.142824] gunicorn[3294]: segfault at 8aaaffcd ip 00007fe41fd449ac sp 00007fe4285fe028 error 4 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6340.707355] traps: gunicorn[3304] trap stack segment ip:7fe41fd952b0 sp:7fe4285fdf20 error:0 in libxml2.so.2.9.7[7fe41fc98000+15d000]
[ 6679.557550] gunicorn[3391]: segfault at 288 ip 00007f1de5e5f778 sp 00007f1dee68c030 error 4 in libxml2.so.2.9.7[7f1de5dff000+15d000]
[ 6733.608873] traps: gunicorn[3423] trap stack segment ip:7f1de5efc2b0 sp:7f1dee68bf70 error:0 in libxml2.so.2.9.7[7f1de5dff000+15d000]
[ 6827.920702] gunicorn[3420]: segfault at 1 ip 00007f1de5efc2b0 sp 00007f1dee68bf20 error 4 in libxml2.so.2.9.7[7f1de5dff000+15d000]
[ 6929.147324] traps: gunicorn[3416] general protection fault ip:7f1dfc9082cd sp:7f1dee68c660 error:0 in libpython3.8.so.1.0[7f1dfc7f9000+33a000]
[ 7087.427678] traps: gunicorn[3490] general protection fault ip:7f1de621f778 sp:7f1dee68bfe0 error:0 in libxml2.so.2.9.7[7f1de61bf000+15d000]
[ 7143.635524] traps: gunicorn[3521] general protection fault ip:7f1de5e5f778 sp:7f1dee68c030 error:0 in libxml2.so.2.9.7[7f1de5dff000+15d000]
[ 7427.633986] gunicorn[3418]: segfault at 1 ip 00007f1de5ebc2b0 sp 00007f1dee68bf70 error 4 in libxml2.so.2.9.7[7f1de5dbf000+15d000]
[ 7456.031664] gunicorn[3613]: segfault at 31b1 ip 00007f1de5e6b9ac sp 00007f1dee68c028 error 4 in libxml2.so.2.9.7[7f1de5dbf000+15d000]
[ 8140.849211] traps: gunicorn[3618] general protection fault ip:7f1dfc9082cd sp:7f1dee689bf0 error:0 in libpython3.8.so.1.0[7f1dfc7f9000+33a000]
[ 8147.116511] gunicorn[3708]: segfault at 98 ip 00007f1de5e5f778 sp 00007f1dee68bfe0 error 4 in libxml2.so.2.9.7[7f1de5dff000+15d000]
[12764.036684] gunicorn[3508]: segfault at 7f1de83bd770 ip 00007f1de83bd770 sp 00007f1dee68c2b8 error 15
[12767.910535] gunicorn[3414]: segfault at 40 ip 00007f1ded2d12b0 sp 00007f1dee68bf20 error 4 in libxml2.so.2.9.7[7f1ded1d4000+15d000]
[12770.820755] traps: gunicorn[3897] general protection fault ip:7f1de625f778 sp:7f1dee68c030 error:0 in libxml2.so.2.9.7[7f1de61ff000+15d000]
[12776.398498] traps: gunicorn[3615] trap stack segment ip:7f1de62bc2b0 sp:7f1dee68bf20 error:0 in libxml2.so.2.9.7[7f1de61bf000+15d000]
[13384.874396] gunicorn[3945]: segfault at 7f1a19f5b7d4 ip 00007f1de5e5f778 sp 00007f1dee68bfe0 error 4 in libxml2.so.2.9.7[7f1de5dff000+15d000]

[candlerb]

Bleugh: python segfaulting is not good. But it's in libxml2 which strengthens my theory that lxml 5 is to blame.

[CTV-2023]

It seems 502 errors on nginx side are related to segfault messages from gunicorn yes :(

[CTV-2023]

I had napalm in my local requirements

Collecting lxml>=4.3.0 (from napalm->-r local_requirements.txt (line 1))
Downloading lxml-5.0.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (6.7 kB)

I removed it by python-saml package itself downloads lxml5

Collecting lxml!=4.7.0,>=4.6.5 (from python3-saml->-r local_requirements.txt (line 1))
Using cached lxml-5.0.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (6.7 kB)

Issue unrelated to Netbox
SAML-Toolkits/python3-saml#389

[candlerb]

Great: then you should be able to workaround the problem by pinning e.g. lxml==4.9.3 in (local_)requirements.txt

There are some pip freeze outputs from various versions of Netbox here. From those, it appears that lxml is not normally pulled in as a requirement. As you say, it gets pulled in via optional extensions (napalm, python3-saml).

It's a tricky problem, because pip historically doesn't handle dependency trees very well - although it's supposed to be better now. It's not surprising that third party libraries only specify a minimum version, not a maximum (they weren't to know that lxml 5.0 would be released in a broken state)

[CTV-2023]

Yes it's solved that way thanks 👍