Issues after encountering issues with netbox instance

[ijjoseph]

Hi All,

I need some assistance with recovering a database.
See below error message received. I'm not sure where to begin. Can someone help?

Server Error
A database programming error was detected while processing this request. Common causes include the following:

Database migrations missing - When upgrading to a new NetBox release, the upgrade script must be run to apply any new database migrations. You can run migrations manually by executing python3 manage.py migrate from the command line.

Unsupported PostgreSQL version - Ensure that PostgreSQL version 9.6 or higher is in use. You can check this by connecting to the database using NetBox's credentials and issuing a query for SELECT VERSION().

The complete exception is provided below:

<class 'django.db.utils.ProgrammingError'>

column extras_customfield.created does not exist
LINE 1: SELECT "extras_customfield"."created", "extras_customfield"....
^

Python version: 3.8.10
NetBox version: 3.1.10

[jeremystretch]

The database error indicates that your migrations have not been run. What action did you take prior to receiving this error? Was NetBox recently upgraded?

[ijjoseph]

This is the error message I got when I ran manage.py migrate

/opt/netbox/netbox$ python3 manage.py migrate
/opt/netbox/netbox/netbox/settings.py:57: UserWarning: The CACHE_TIMEOUT configuration parameter was removed in v3.0.0 and no longer has any effect.
warnings.warn(
/opt/netbox/netbox/netbox/settings.py:61: UserWarning: The RELEASE_CHECK_TIMEOUT configuration parameter was removed in v3.0.0 and no longer has any effect.
warnings.warn(
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 377, in execute
django.setup()
File "/usr/local/lib/python3.8/dist-packages/django/init.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.8/dist-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.8/dist-packages/django/apps/config.py", line 90, in create
module = import_module(entry)
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'graphiql_debug_toolbar'

[samvdl]

Hi ijjoseph,

I encountered exactly the same situation.
First I wanted to upgrade from v2.8.9 to v3.2,5 which horribly failed on this issue
Than I read somewhere that you can't do a major upgrade in one time and you need to at least go by a minor one.
So I reverted / rolled back my VM and tried a minor one (v2.9.10), and I encountered the same issue.
So somewhere deep in the discussions on the net, there was this tread suggesting to start fiddling with the postgre database tables itself, but i'd pass for that (already deep enough in 'wonderland' as it is). The other suggestion was that you can's skip any minor upgrade...
So... again I reverted / rolled back my VM and tried the smallest minor upgrade (all patches = last number), so from V2.8.9 to V2.9.11. And yet again same error.
My PostgreSQL version is checked by Select version() and showing me PostgreSQL 11.16
applying Python3 /opt/netbox/netbox/manage.py migrate (knowing that /opt/netbox is a symlink to /opt/netbox/netbox-2.9.11 --> which is the new netbox version)

You got any solutions found in the mean while?

Thanks,
Sam

[cpmills1975]

In prod/dev I have a daily migration happening between 2.10.3 and 3.2.5. The requirement as I understand it is that you can't leap between 2.10 and 3.anything, you need to go to 2.11.something first.

This works consistently.

[samvdl]

In prod/dev I have a daily migration happening between 2.10.3 and 3.2.5. The requirement as I understand it is that you can't leap between 2.10 and 3.anything, you need to go to 2.11.something first.

This works consistently.

Well that's nice and possibly a whole other unrelated story all together.
The issue at hand which 'ijjoseph' initiated, is about the fact that whenever upgrading netbox, you get a database error altough your python version, your PostgreSQL version, etc is high enough according to the netbox upgrade documentation...
And furthermore, trying to perform "manage.py migrate" is generating the errors like his post showed (and mind for that matter).

So question is not if it's possible to upgrade , yes or no, but why we get stuck on the database migration...

[candlerb]

/usr/local/lib/python3.8/dist-packages/django/apps/config.py

That shows you're running the migrations wrongly. You must run Netbox within its virtualenv - not with some random system-installed version of Django. That is:

cd /opt/netbox
. venv/bin/activate
python netbox/manage.py migrate

However, if instead you run the supplied upgrade.sh then it takes care of that for you.

is it possible to revert to the previous version of NetBox? We had an issue with a NetBox installation that was running the version stated in the screenshot below. I had neglected to save or backup the installation before I proceeded to update to the latest version via cloning from the git repo.
How can I salvage this.?

No, it's not possible to move back from a newer version of Netbox to an older one, if you've already applied or partially-applied the migrations. You have to restore from a backup. If you don't have backups then your data is gone, unless you are able to sort out pressing forward to the new version.

[samvdl]

(venv) [root@netboxVM bin]# /usr/bin/python3.9 /opt/netbox-2.9.11/netbox/manage.py migrate
Traceback (most recent call last):
  File "/opt/netbox-2.9.11/netbox/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

[candlerb]

You're running the wrong version of python. Given that (venv) is active, you should do

python netbox/manage.py ...

not

/usr/bin/python3.9 netbox/manage.py ...

(When the virtualenv is active, "python" is redirected to the correct version of python inside the virtualenv)

[samvdl]

(When the virtualenv is active, "python" is redirected to the correct version of python inside the virtualenv)

--> Well this might be the problem see:

(venv) [root@netboxVM netbox-2.9.11]# python --version
Python 2.7.5

I need to redirect to the correct version inside this new venv... How?

and so...

(venv) [root@netboxVM netbox-2.9.11]# python netbox/manage.py migrate
Traceback (most recent call last):
  File "netbox/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management

[samvdl]

I used python 3 (-> points to 3.6) to create the new virtual environment, so I am actually uncertain why 'python version' in that virtual enviroment gives me Python-2.7.5
[ansible@netboxVM ~]$ /usr/bin/pyhton3 -m venv /opt/netbox-2.9.11/venv

[markkuleinio]

It always puzzles me how people get into such a corner that they need to run various commands manually instead of letting the documented upgrade.sh (with or without specifying the PYTHON variable) command deal with the details.

[markkuleinio]

Moreover I wonder if .upgrade.sh of the new netbox version is vulnerable to the context (meaning /opt/nebox being a symlink to the installation dir and not the native one).

That's not a problem, as you can see in the upgrade instructions: https://docs.netbox.dev/en/stable/installation/upgrading/

[samvdl]

Moreover I wonder if .upgrade.sh of the new netbox version is vulnerable to the context (meaning /opt/nebox being a symlink to the installation dir and not the native one).

That's not a problem, as you can see in the upgrade instructions: https://docs.netbox.dev/en/stable/installation/upgrading/

Well, it is when your native installation is in /opt/netbox/ and so when opt/netbox is NOT a symlink.
Imagine for 1 moment this to be the truth, than...
Than the documentation doesn't make any sense:
with X.Y.Z. being the new version:
sudo ln -sfn /opt/netbox-X.Y.Z/ /opt/netbox

is actually saying that /opt/netbox needs to change from a native installation directory towards a symlink pointing towards the new version directory.
Ofcourse this is nonsense... I see now.
So the documentation on upgrading is correct (well I pressume) assuming that the primordial installation was done accordingly...
And that's for starters, not the case with me (!) (ouch) I inherited it otherwise.
so I already now rectified this context towards the "meant to be" and restarted the netbox service. And that's good (working).
So now I can restart the upgrade (with I minor version) from a clean situation.
I'll keep you posted.

[markkuleinio]

So now I can restart the upgrade (with I minor version) from a clean situation.

Yes: as long as /opt/netbox is the correct root path for your NetBox installation (whether a symlink or git clone or whatever), the install path is normal and the instructions apply as is.

[samvdl]

Moreover I wonder if .upgrade.sh of the new netbox version is vulnerable to the context (meaning /opt/nebox being a symlink to the installation dir and not the native one).

That's not a problem, as you can see in the upgrade instructions: https://docs.netbox.dev/en/stable/installation/upgrading/

Well, it is when your native installation is in /opt/netbox/ and so when opt/netbox is NOT a symlink. Imagine for 1 moment this to be the truth, than... Than the documentation doesn't make any sense: with X.Y.Z. being the new version: sudo ln -sfn /opt/netbox-X.Y.Z/ /opt/netbox

is actually saying that /opt/netbox needs to change from a native installation directory towards a symlink pointing towards the new version directory. Ofcourse this is nonsense... I see now. So the documentation on upgrading is correct (well I pressume) assuming that the primordial installation was done accordingly... And that's for starters, not the case with me (!) (ouch) I inherited it otherwise. so I already now rectified this context towards the "meant to be" and restarted the netbox service. And that's good (working). So now I can restart the upgrade (with I minor version) from a clean situation. I'll keep you posted.

AND....

In the upgrade documentation
...remains the fact that X.Y.Z means the new version when creating the ne symlink,
but when copying the current config files to the new version trough the symlink, X.Y.Z is actually meant 'the current version'.
So there I calls out that the documentation invokes ambiguity.
You agree?

[markkuleinio]

Well it says "latest version" and "current version". If you'd like another wording, you are free to check the contribution guidelines and propose a specific improvement in the documentation, that's how things go forward.

[samvdl]

so....
Okay, new virtual environment.... created with python 3.9
and so....

(venv) [ansible@netboxVM-2.9.11]$ python --version
Python 3.9.10
(venv) [ansible@netboxVM-2.9.11]$ pwd
/opt/netbox-2.9.11
(venv) [ansible@netboxVM-2.9.11]$ python netbox/manage.py migrate
Traceback (most recent call last):
  File "/opt/netbox-2.9.11/netbox/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'
(venv) [ansible@netboxVM-2.9.11]$

but yep....
and oh by the way.... @candlerb , I did

cd /opt/netbox-2.9.11
. venv/bin/activate

(I saw you post on https://groups.google.com/g/netbox-discuss/c/D30kPhx9lAw)

I'm in the right virtualenv... yes?

[markkuleinio]

ModuleNotFoundError: No module named 'django'

Quite obviously, if you haven't ever run upgrade.sh (or manually done all the same operations it does)

[candlerb]

This single command:

PYTHON=/usr/bin/python3.9 /opt/netbox/upgrade.sh

will create a fresh virtualenv, populate all the dependencies, apply any outstanding migrations, collect static files - everything.

[samvdl]

So.... basically I succeeded now in the minor upgrade.
Thing is that it is imperative that /opt/netbox is a symlink to you new version folder and that you go from there.
My situation (inherited) was that the original installed version was in /opt/netbox/ itself...
And so than starting in documentation with the upgrade is conflicting.

But I must also say that in the documentation of the upgrade, X.Y.Z are used interchangeably between first the new version (for symlink creation) and than secondly the current version (for copy conf files trough symlink), which is not very consistent and was very confusing . even worse to grasp when assuming that my initial installation setup was (unknowingly) not according to initial netbox installation documentation).

...at the end I still got a 502, cause the service was not running because I needed to restart netbox and netbox-rq in the virtual environment (venv) (dumbass me!)

It was not necessary (since I was already on 2.8.9) but I thought I had to copy the services themselves from the netbox/contrib/ folder to the /etc/systemd/system/ folder as well (reload, restart and enable them) afterwards. So I did that as well ;-)

So yes, indeed now it works. I upgraded to the latest minor version 2.11.12 and it's working

[markkuleinio]

Great!

Thing is that it is imperative that /opt/netbox is a symlink to you new version folder

Yes, if you are following the installation option A. In general, no, /opt/netbox is not required to be a symlink, it is just expected that /opt/netbox can be used to find the NetBox installation root.

But anyway, good that you got through it.

[samvdl]

so there you have it ;-)....
"..in general no, except if....." and away we are ;-)