Bug: Manually curated server list bound to /gluetun/servers.json gets overwritten

[jberkh]

Is this urgent?

None

Host OS

Ubuntu 22.04

CPU arch

x86_64

VPN service provider

ProtonVPN

What are you using to run the container

docker-compose

What is the version of Gluetun

Running version latest built on 2024-07-12T19:57:02.146Z (commit 9d50c23)

What's the problem 🤔

ProtonVPN server list is outdated even after update following https://github.com/qdm12/gluetun-wiki/blob/main/setup/servers.md#update-the-vpn-servers-list. (As was also noted in #2332).

Attempted fix: manually create a servers.json and bind to /gluetun/servers.json
Expected behaviour: Gluetun runs with the manually created servers.json.
Observed behaviour: Gluetun overwrites the manual servers.json and then can't find specified hostname, because servers.json was overwritten.

Attempted fix 2: manually create a servers.json and bind to /gluetun/servers.json, read-only
Expected behaviour: Gluetun runs with the manually created servers.json, doesn't overwrite file
Observed behaviour: Gluetun cannot overwrite file and quits.

Share your logs (at least 10 lines)

2024-07-13T17:32:21+02:00 INFO [routing] default route found: interface eth0, gateway 192.168.53.1, assigned IP 192.168.53.2 and family v4
2024-07-13T17:32:21+02:00 INFO [routing] local ethernet link found: eth0
2024-07-13T17:32:21+02:00 INFO [routing] local ipnet found: 192.168.53.0/24
2024-07-13T17:32:21+02:00 INFO [firewall] enabling...
2024-07-13T17:32:21+02:00 INFO [firewall] enabled successfully
2024-07-13T17:32:22+02:00 INFO [storage] merging by most recent 19425 hardcoded servers and 474 servers read from /gluetun/servers.json
2024-07-13T17:32:22+02:00 ERROR VPN settings: provider settings: server selection: for VPN service provider protonvpn: the hostname specified is not valid: [...]
2024-07-13T17:32:22+02:00 INFO Shutdown successful

----

2024-07-13T17:46:20+02:00 INFO [routing] default route found: interface eth0, gateway 192.168.53.1, assigned IP 192.168.53.2 and family v4
2024-07-13T17:46:20+02:00 INFO [routing] local ethernet link found: eth0
2024-07-13T17:46:20+02:00 INFO [routing] local ipnet found: 192.168.53.0/24
2024-07-13T17:46:20+02:00 INFO [firewall] enabling...
2024-07-13T17:46:20+02:00 INFO [firewall] enabled successfully
2024-07-13T17:46:20+02:00 INFO [storage] merging by most recent 19425 hardcoded servers and 474 servers read from /gluetun/servers.json
2024-07-13T17:46:20+02:00 ERROR writing servers to file: open /gluetun/servers.json: read-only file system
2024-07-13T17:46:20+02:00 INFO Shutdown successful

Share your configuration

No response

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[qdm12]

ProtonVPN server list is outdated even after update following https://github.com/qdm12/gluetun-wiki/blob/main/setup/servers.md#update-the-vpn-servers-list. (As was also noted in #2332).

What servers are you missing? See my response #2332 (comment) as well.

Attempted fix: manually create a servers.json and bind to /gluetun/servers.json
Expected behaviour: Gluetun runs with the manually created servers.json.

This is by design, the servers.json you have bind mounted is a merge of the the most recent data between your local servers.json and the built-in servers.json. You probably didn't change the timestamp for protonvpn, hence the override. On the other hand, the next servers update will then throw away your local servers data, so you should avoid changing it manually. Let's figure this out together and I can add it in the updating code so it's bundled now and in the future. Are the servers you are missing present in https://api.protonmail.ch/vpn/logicals ?

[qdm12]

Let's please continue the conversation on (now re-opened) issue #2332 to figure out what servers are missing and how we can automatically add them to the list. Closing this since the behavior or servers.json is as expected.

[github-actions]

Closed issues are NOT monitored, so commenting here is likely to be not seen.
If you think this is still unresolved and have more information to bring, please create another issue.

This is an automated comment setup because @qdm12 is the sole maintainer of this project
which became too popular to monitor issues closed.

[justinhorner]

Just wanted to share my findings, as I was having this same issue.

The solution that worked for me was to ensure the provider 'version' in my hardcoded servers.json file was the same as the original servers.json file, then update (in my hardcoded server.json file) the provider's timestamp to be greater than the provider's timestamp value in the original servers.json file.

You can see here [1] in the code that it only reads hardcoded values with the same server version and here in the code [2] that it does the timestamp comparison.

1. gluetun/internal/storage/read.go

   Lines 15 to 17 in ddd9f4d

   	// readFromFile reads the servers from server.json.
   	// It only reads servers that have the same version as the hardcoded servers version
   	// to avoid JSON decoding errors.

2. https://github.com/qdm12/gluetun/blob/master/internal/storage/merge.go#L32

I was originally incrementing, in my custom servers.json, both the provider version and timestamp but after updating (incrementing) only the provider timestamp, my custom server.json loaded up just fine and my 'SERVER_HOSTNAMES' env var was recognized and all worked as expected.