[Enhancement] Migration from official securecompliance/gvm image

[landonstewart]

Describe the solution you'd like
Many folks, including myself, are using the official but outdated image securecompliance/gvm. It would be nice to be able to switch from that image to this more actively developed one.

When using deineagenturug/gvm:latest-data-full as a drop in replacement for securecompliance/gvm:debian-master-data there are problems with PostgreSQL. Likely because of the version differences.

A check on first start and migration would be really excellent. Failing that - a method to migrate existing data would be great.

ALSO - I'm capable and willing to clone and send pull requests if you can give me a basic overview of how the check/migration should work. At this point its not clear and I'd need to get up to speed before attempting to contribute changes.

[Dexus]

I started on this see the mention in #14

I would check the version in the database storage path, and them it need to update major version to the next major version, excepted of this behavior is all pre 10 major version.

I‘m not sure currently how much versions need to migrated so my initial Dockerfiles includes more then I think that are needed.

11to12 and 12 to 13 should work easy only from 13 to 14 needs additional steps.

If you like to contribute, you are welcome. But I need to know this, else I will do it myself hopefully by end of the next week.

regards Josef

[Dexus]

Btw. It will not work within the default container and should be done in an extra image that can reused with other installations (#14)

[landonstewart]

If you like to contribute, you are welcome. But I need to know this, else I will do it myself hopefully by end of the next week.

Hi Josef,

I tried to get going on this but won't have time before next week with the holidays and all.

[Dexus]

I started with a initial setup here: https://github.com/DeineAgenturUG/greenbone-gvm-openvas-for-docker/tree/db_upgrade

It will need for sure more work, maybe a other way. I try how I can fix the issues that I see.

[Dexus]

Here a working development build.

⚠️ Please make sure you have shutdown/stopped the old GVM container from securecompliance or whatever your old image might be!!!!

Once you have done, please make sure you have a backup created before you running this following steps.
We will not touch the old data, but you'll know it is better to have a backup, then sorry. We create a temp copy of the Database before we starting the upgrade process to postgresql version 13.

Also this step create the right permissions for the new database. So you don't need to have a eye on this. Right after all steps are done, you can use our image deineagenturug/gvm:latest with the new database of postgresql 13.

docker pull deineagenturug/pgdb-upgrade-develop:latest

docker run -ti --rm -v "/path/to/old/securecompliance/database:/mnt/pg_old/data" -v "/path/to/new/deineagenturug/database:/mnt/pg_new/data" deineagenturug/pgdb-upgrade-develop:latest bash

Then you have to run in the container shell: /opt/setup/scripts/db_upgrade.sh

ℹ️ I hope you can enjoy the "DB Upgrade" tool and let me know how it works for you, any feedback is welcome.

/cc @markdesilva @landonstewart

[markdesilva]

Hi @Dexus,

Thank you! Will try it out as soon as I'm done coaching my son for his mid year exams.

Best Regards,
Mark

Here a working development build.

⚠️ Please make sure you have shutdown/stopped the old GVM container from securecompliance or whatever your old image might be!!!!

Once you have done, please make sure you have a backup created before you running this following steps. We will not touch the old data, but you'll know it is better to have a backup, then sorry. We create a temp copy of the Database before we starting the upgrade process to postgresql version 13.

Also this step create the right permissions for the new database. So you don't need to have a eye on this. Right after all steps are done, you can use our image deineagenturug/gvm:latest with the new database of postgresql 13.

docker pull deineagenturug/pgdb-upgrade-develop:latest

docker run -ti --rm -v "/path/to/old/securecompliance/database:/mnt/pg_old/data" -v "/path/to/new/deineagenturug/database:/mnt/pg_new/data" deineagenturug/pgdb-upgrade-develop:latest bash

Then you have to run in the container shell: /opt/setup/scripts/db_upgrade.sh

ℹ️ I hope you can enjoy the "DB Upgrade" tool and let me know how it works for you, any feedback is welcome.

/cc @markdesilva @landonstewart

[Dexus]

Once it's tested a bit more and I get some feedback on it, it will be added to the gvm image as well. So there is no need for an extra image.

[landonstewart]

So here's what I did to test it with your instructions in mind:

1. docker-compose down
2. mv /data/gvm/database /data/gvm/database.OLD
3. $ docker run -ti --rm -v "/data/gvm/database.OLD:/mnt/pg_old/data" -v "/data/gvm/database:/mnt/pg_new/data" deineagenturug/pgdb-upgrade-develop:latest bash
4. Within the container shell executed:
   # /opt/setup/scripts/db_upgrade.sh (answered 'y' when asked)
   # exit
5. Modified my docker-compose.yml to use image: deineagenturug/gvm:latest-data-full
6. Executed: docker-compose up -d

It worked without a hitch. Thank you!

I've attached the output of the docker container during the upgrade:
database_upgrade_output.txt

[Dexus]

@landonstewart
Thank you for sharing the path in more detailed steps.

Your log shows an identical output to mine. Perfect!

[netbix]

The conversion is successful but when the docker starts this error is thrown:

gvmdebian | redis: started
gvmdebian | Wait for redis socket to be created...
gvmdebian | Testing redis status...
gvmdebian | Redis ready.
gvmdebian | Creating Database folder and rights...
gvmdebian | Starting PostgreSQL...
gvmdebian | 2022-04-23 12:47:48,993 INFO spawned: 'postgresql' with pid 590
gvmdebian | 2022-04-23 12:47:59,063 INFO success: postgresql entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
gvmdebian | postgresql: started
gvmdebian | Generate SSH-HOST Keys
gvmdebian | Creating Greenbone Vulnerability Manager database
gvmdebian | createuser: error: creation of new role failed: ERROR: role "gvm" already exists
gvmdebian | 2022-04-23 12:47:59,277 INFO exited: init (exit status 1; not expected)

[netbix]

I think i found the problem.
in the new database folder these two files are missing, present in the old database folder:

-rw-r--r-- 1 syslog input 0 Apr 23 14:13 .firstrun

and

-rw-r--r-- 1 syslog input 0 Apr 23 14:13 .upgrade_to_21.4.0

[markdesilva]

Hi @landonstewart, would it be alright to send me your docker-compose.yml please so I can take a look? I think I'm doing something wrong with the volumes.

Thank you.

So here's what I did to test it with your instructions in mind:

1. `docker-compose down`

2. `mv /data/gvm/database /data/gvm/database.OLD`

3. `$ docker run -ti --rm -v "/data/gvm/database.OLD:/mnt/pg_old/data" -v "/data/gvm/database:/mnt/pg_new/data" deineagenturug/pgdb-upgrade-develop:latest bash`

4. Within the container shell executed:
   `# /opt/setup/scripts/db_upgrade.sh` (answered 'y' when asked)
   `# exit`

5. Modified my docker-compose.yml to use `image: deineagenturug/gvm:latest-data-full`

6. Executed: `docker-compose up -d`

It worked without a hitch. Thank you!

I've attached the output of the docker container during the upgrade: database_upgrade_output.txt

[Dexus]

@netbix you previous image/tag was which one?
Will look into it and fix it when I get the same result.

Okay don't need to have this info, it's right that the file is missing because it will not copied. Will fix it in the evening.

[netbix]

i migrated from:
securecompliance/gvm:debian-master-data TO deineagenturug/gvm:latest-data-full

I have change my config with:

USERNAME=admin
PASSWORD=admin
DB_PASSWORD=xxxxxxxxxx
HTTPS=true
SSHD=true
AUTO_SYNC=YES
AUTO_SYNC_ON_START=YES
TZ=Europe/Rome
RELAYHOST=x.x.x.x
SMTPPORT=25
OPT_PDF=1

and compose:

version: '3.4'

services:

gvmdebian:
image: deineagenturug/gvm:latest-data-full
hostname: gvmdebian
container_name: gvmdebian
restart: always
ports:
- "8082:9392"
- "5432:5432"
- "2222:22"
env_file:
- ./config/local.env
volumes:
- ./storage/postgres-db:/opt/database
- ./storage/openvas-plugins:/var/lib/openvas/plugins
- ./storage/gvm:/var/lib/gvm
- ./storage/ssh:/etc/ssh

with those two files everything works

[Dexus]

Yes I had not thought about the lock files. Will fix it in the evening, and release a new upgrade image.

[netbix]

what is this variable for?

export NO_AUTO_UPDATE=${NO_AUTO_UPDATE:-NO}

[netbix]

thank you !

[Dexus]

what is this variable for?

export NO_AUTO_UPDATE=${NO_AUTO_UPDATE:-NO}

I hope I did not use it, had used it in a local test. It's the opposite of the AUTO_SYNC_ON_START=YES when I'm get it right from remembers.

have to say you all thank you for the feedback.

[netbix]

further problem:

scanning does not work.
OpenVAS_scanner is not found

[netbix]

root@gvmdebian:/# gvmd --get-scanners
root@gvmdebian:/# gvmd --get-users
root@gvmdebian:/# gvmd --version
Greenbone Vulnerability Manager 21.4.5
Manager DB revision 242
Copyright (C) 2009-2021 Greenbone Networks GmbH
License: AGPL-3.0-or-later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[netbix]

the process seems to work:

[netbix]

I was wrong:

but why is it not detected and the scans no longer work?

[netbix]

[markdesilva]

Hi @Dexus, I ran the latest deineagenturug/pgdb-upgrade-develop:latest on my old db and it fails with:

Start checking of Postgresql pg_upgrade from 12 to 13
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt

Failure, exiting
root@dc2dd6914f3a:/work# more loadable_libraries.txt
could not load library "/usr/local/lib/libgvm-pg-server": ERROR:  could not access file "/usr/local/lib/libgvm-pg-server": No such file or directory
In database: gvmd

Have confirmed I'm running the latest deineagenturug/pgdb-upgrade-develop:

REPOSITORY                            TAG           IMAGE ID       CREATED       SIZE
deineagenturug/pgdb-upgrade-develop   latest        02275e667d97   6 hours ago   676MB

@markdesilva @netbix Thanks for your support here and testing.

New Version is ready.

docker pull deineagenturug/pgdb-upgrade-develop:latest

Fixed:

* correct old `libgvm-pg-server` path

* fix rm to remove also DOT files from the new DB directory

[Dexus]

@markdesilva can you please send me a copy from your old DB as a .tar.gz/xz ? https://nc.josef-froehle.de/s/GmMW3txfAW8mCpY
I can so test it local and I would be a bit faster to fix problems.
And don't worry, I will work with it under data protection aspects.As soon as I have done my tests, I delete everything again.

[markdesilva]

@markdesilva can you please send me a copy from your old DB as a .tar.gz/xz ? https://nc.josef-froehle.de/s/GmMW3txfAW8mCpY I can so test it local and I would be a bit faster to fix problems. And don't worry, I will work with it under data protection aspects.As soon as I have done my tests, I delete everything again.

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

[Dexus]

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

Should not be a problem for my Server, if you have an alternate storage where I can download it, I can do it. Then you can send me a link to github@josef-froehle.de

[markdesilva]

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

Should not be a problem for my Server, if you have an alternate storage where I can download it, I can do it. Then you can send me a link to github@josef-froehle.de

After compression its a few hundred MB. Uploading now.

[markdesilva]

After compression its a few hundred MB. Uploading now.

Upload is complete. Thanks!

[Dexus]

After compression its a few hundred MB. Uploading now.

Upload is complete. Thanks!

Hmm, nothing there. But got an error in the logs. Forgot to set the Upload limit in one config file which now is fixed. Can you re-try?

Edit: @markdesilva thank you upload is there now.

[markdesilva]

Hmm, nothing there. But got an error in the logs. Forgot to set the Upload limit in one config file which now is fixed. Can you re-try?

Done! Thanks.

[Dexus]

New Version is ready.

docker pull deineagenturug/pgdb-upgrade-develop:latest

Fixed:

• correct path of the /usr/lib/libgvm-pg-server.so symlink to /usr/local/lib/libgvm-pg-server.so now with the extension