Updating working docker shlink to 2.9.1 and fails to start.

[cwldev]

How Shlink is set-up

• Shlink Version: 2.9.1
• PHP Version: n/a
• How do you serve Shlink: Docker Image on Ubuntu
• Database engine used: MariaDB

Summary

Run update of the docker container, reloaded it, and the server fails to start..

Current behavior

The server does not start.

Expected behavior

Expected the server to start. It ran well on version 2.9.0

How to reproduce

Update a working docker container from 2.9.0 to 2.9.1.

Additional information from logs (seems to imply database access issues):

Shlink.ERROR - Shlinkio\Shlink\Rest\Exception\MercureException: This Shlink instance is not integrated with a mercure hub. in /etc/shlink/module/Rest/src/Exception/MercureException.php:21
Shlink.ERROR - PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.forward_query' in 'field list' in /etc/shlink/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:137
#0 /etc/shlink/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php(137): PDO->query('SELECT DISTINCT...', NULL)
#1 /etc/shlink/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOQueryImplementation.php(24): Doctrine\DBAL\Driver\PDOConnection->doQuery('SELECT DISTINCT...', NULL)

Next Doctrine\DBAL\Exception\InvalidFieldNameException: An exception occurred while executing 'SELECT DISTINCT s0_.id AS id_0, s0_.original_url AS original_url_1, s0_.short_code AS short_code_2, s0_.date_created AS date_created_3, s0_.valid_since AS valid_since_4, s0_.valid_until AS valid_until_5, s0_.max_visits AS max_visits_6, s0_.import_source AS import_source_7, s0_.import_original_short_code AS import_original_short_code_8, s0_.title AS title_9, s0_.title_was_auto_resolved AS title_was_auto_resolved_10, s0_.crawlable AS crawlable_11, s0_.forward_query AS forward_query_12, s0_.domain_id AS domain_id_13, s0_.author_api_key_id AS author_api_key_id_14 FROM short_urls s0_ WHERE 1 = 1 ORDER BY s0_.date_created DESC LIMIT 5':

Also, shlink-web-client works after a docker image update (but cannot connect to the server).

Also, reverting to the previous shlink:290 image works fine and the server runs.

[acelaya]

The forward_query column was introduced in v2.9.0, so it's not possible that version works and 2.9.1 doesn't.

If I had to guess, the latest migration did not run for some reason, when you updated from an older version to either 2.9.0 or 2.9.1. Can you check in your database, if the migrations table has the Version20211002072605 one?

[cwldev]

Thanks @acelaya , it looks like the database does not. Huh.

[acelaya]

You are missing the 2 most recent migrations, Version20210720143824 and Version20211002072605.

The container should run the migrations on startup, as it's part of the entry point. How do you run the docker container?

[cwldev]

I run the container initially as a docker run statement:

docker run
--name shlink
-p 8078:8080
-e SHORT_DOMAIN_HOST=my.domain.com
-e SHORT_DOMAIN_SCHEMA=http
-e GEOLITE_LICENSE_KEY=xxxx
-e DB_DRIVER=maria
-e DB_USER=dbuser
-e DB_PASSWORD=mypassword
-e DB_HOST=10.0.0.2
--network=mynetwork
--ip 10.0.0.15
shlinkio/shlink:stable

Then do docker pull and restart in portainer most often. Things are working normally now in the older version 2.9.0.

[acelaya]

It's not possible that v2.9.0 works without latest migration, so there's something odd here.

[cwldev]

My mistake here. Sorry about this.. I'm realizing I missed a few updates here.. there must have been a flurry of updates lately. I notice my working server runs the older version v2.8.1. Is there a sequence to get to 2.9.1 you think? Is 2.8.1 to 2.9.1 not supported?

Adding a screenshot;

[acelaya]

There should be no problem to go from v2.8.1 to v2.9.1, however, something is preventing your migrations to be run, but has been for months, as your last migration is one created in May.

Is your database up, running and available when the Shlink container starts?

[cwldev]

Is your database up, running and available when the Shlink container starts?

Yes, indeed. It all works. Shlink-web-client, the short links, the domain. Everything is operational on 2.8.1.

In fact, here's a link to try on my server: http://g.ecwl.cc/P2VHs

[acelaya]

Yes, it will work as long as none of the paths that require new fields is touched, but as I said, the problem is on the image start-up. The migrations command is not running or it's failing for some reason, leaving the database outdated.

[cwldev]

No doubt. I'll swing back and try again. Perhaps shlink had trouble adding the table to the DB.. maybe the DB didn't respond fast enough. New short links are added to the database in 2.8.1 (see http://g.ecwl.cc/Oe2nP ) so that's possible. It could have been a one-time thing.

[acelaya]

The database is clearly available after shlink has started-up. The key here is to know if it is when it starts-up.

Do you stop the database when updating the Shlink container?

[cwldev]

Do you stop the database when updating the Shlink container?

No, the database is never stopped during or after the updates.

However, this time - to troubleshoot - I restarted the database container before using the shlinkio/shlink:stable image. Once the database started, I tested it and all seemed fine. I then updated the image to shlinkio/shlink:stable and started. The server once again failed and a similar log occurred.

[cwldev]

I just wonder if, somehow, mariaDB is refusing to create the table. Maybe if I manually create Version20210720143824 and Version20211002072605 tables that might get things moving. I don't see any errors in the maridb log.. still looking for info.. all other services using mariadb are working currently.

[acelaya]

Yeah. I think the best option for now is too run the migrations manually.

Try opening a shell in the container, once up and running, and execute shlink db:migrate -vvv.

If everything goes right, you should get rid of the error. If it fails, we'll get more detailed info why.

[cwldev]

Yeah. I think the best option for now is too run the migrations manually.

Try opening a shell in the container, once up and running, and execute shlink db:migrate -vvv.

If everything goes right, you should get rid of the error. If it fails, we'll get more detailed info why.

That's great. I'll try that when I can and let you know how it goes. Thanks for the assistance with this, I appreciate it.

[imagisk]

I'm having exactly the same problem here.

[acelaya]

I'm having exactly the same problem here.

Then, can you try the solution proposed here?

[cwldev]

I'm going to try and get into this today.. I'll report back on how things go.

[cwldev]

I noticed that 2.9.2 was out, so I pulled the newest stable image: docker pull shlinkio/shlink:stable

Restarted 2.9.2 and went into the container's shell

# shlink db:migrate -vvv
Migrating database...
  RUN  '/usr/local/bin/php' 'vendor/doctrine/migrations/bin/doctrine-migrations.php' 'migrations:migrate' '--no-interaction'
  OUT  
  OUT  
  RES  Command ran successfully

                                                                                                                        
 [OK] Database properly migrated!            

Restart the container, still get the following error when loading the client:

Next Doctrine\DBAL\Exception\InvalidFieldNameException: An exception occurred while executing a query: SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.forward_query' in 'field list' in /etc/shlink/vendor/doctrine/dbal/src/Driver/API/MySQL/ExceptionConverter.php:65

[cwldev]

Working to enable more logging on the MariaDB side, but rebooted the dbms and tried again from shlink and saw more here:

And, on the mariaDB log side, I've isolated the migration process in my MySQL.log file (there's a lot in there) but I don't see any errors that jump out at me.

It's almost as if it's looking for REFERENCED_COLUMN_NAME` /*!50116 whereas I haven't had a working migration in a while (see below).. and maybe the previous migration was something like "ShlinkMigrations\Versionxxxxxxxxxx50116? Since Shlink doesn't see it, the whole thing fails?

[acelaya]

Mmmm, I think those warnings are not really the root cause.

How many Shlink instance are you running in parallel? I'm wondering if the locking that prevents multiple migrations in parallel could be related.

Can you try running php vendor/doctrine/migrations/bin/doctrine-migrations.php migrations:migrate inside the container?

The db:migrate command should have run that as a side effect, but with some extra logic that could be getting on the way. Let's try that and see if it isolates the problem.

[cwldev]

Dang, and I shut everything down and reverted back to 2.9.0.. will go back and try those things. I'm running only ONE instance of shlink.

Back on 2.9.2 inside the container shell:

/etc/shlink # shlink -V
Shlink 2.9.2

Running php vendor/doctrine/migrations/bin/doctrine-migrations.php migrations:migrate:

/etc/shlink # php vendor/doctrine/migrations/bin/doctrine-migrations.php migrations:migrate
 WARNING! You are about to execute a migration in database "shlink" that could result in schema changes and data loss. Are you sure you wish to continue? (yes/no) [yes]:
 > yes

No messages after execution. Checked on the database and ShlinkMigrations\Version20210522124633 is still the last record in migrations, not much changed that I can see.

[acelaya]

Thanks for the help debugging this @cwldev.

I really don't see why it is reporting no pending migrations. Maybe the warning above is indeed causing doctrine migrations to silently fail.

If I remember correctly, there should be some output after hitting "yes". The last check would be to run the same with -vvv and see if it prints something else.

I will try to dig in the library repo to see if there's any known issue.

[cwldev]

No worries.. I appreciate you taking the time with this.. I'm hoping to find a way to get there. This is making me wonder if I can save all my link data (obviously not the database), build a new instance of 2.9.2 from scratch and restore them to operation.

Also, I should mention I did run shlink db:migrate -vvv immediately after that PHP command and the output was exactly the same (see below).

[cwldev]

So, I've built a new instance in parallel to the old one and I'm trying to work out the importing (from the old) into the new. I'm working from this document. Naturally, the import isn't working, but more than that I need clarity on this.

Running bin/cli short-url:import shlink asks a few questions. But first and foremost, WHERE do I run this? It's not clear in the doc. My presumption is the shell inside the NEW/Destination container.

1. What is your Shlink instance base URL?:
   This is also less clear. I presume this is how the import process sees the other instance, but ALL of this is behind a reverse proxy, so to directly address the old container I would do something like http://olddomain.com:8065 - no variation of that works. If this is my NEW domain, how then does the importer find the old server?

2. What is your Shlink instance API key?:
   Knowing which server I'm supposed to reference would dictate the API key. I'm using the same one used in the working web client.

And, just for clarity, I'm importing from the OLD domain into a server running a different domain. Is the importer going to expect to see the domains exactly the same? Does it just pull the slugs and links leaving the domain alone? Knowing this would help me troubleshoot why this process might be failing.

[cwldev]

I actually did a list of URLs (on the old) and built a CSV of all the static links and imported them into the new. That's good enough, for now, to keep things running and on the newest version. Time to decommission the old server and move on.

[imagisk]

After a while I firgued it out. Updating the image does not mean that it will update the code inside the volume itself: Mirgation*.php
I have to create a new instance of shlink to a new volume, go to that new volume and copy all the missing Mirgation*.php to the old volume which is missing some new Mirgation*.php files. Start the old instance and it worked.

[acelaya]

After a while I firgued it out. Updating the image does not mean that it will update the code inside the volume itself Mirgation*.php I have to create a new instance of shlink to a new volume, go to that new volume and copy all the missing Mirgation*.php to the old volume which is missing Mirgation*.php files. Start the old instance and it worked.

Why do you need to mount a volume for the migrations in the first place?

The volume is overwriting the content of the container itself.