Skip to content

Add deprecation warnings to "docker-entrypoint.sh" #424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 23, 2021

Conversation

tianon
Copy link
Member

@tianon tianon commented Jul 21, 2020

This is a strawman for discussing the changes discussed in #422 of finally deprecating our entrypoint script.

To start the conversation, I've just added some TODO comments in all the places I think we ought to add a deprecation warning, but would love feedback on whether I've gone too far (and some of this behavior is OK and should stay) and/or I haven't gone far enough and there's more places we should explicitly add a warning. 😄

@@ -181,6 +181,8 @@ done
# if long and short hostnames are not the same, use long hostnames
if [ "$(hostname)" != "$(hostname -s)" ]; then
: "${RABBITMQ_USE_LONGNAME:=true}"

# TODO show warning about implied RABBITMQ_USE_LONGNAME variable setting
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you thinking of removing this? I'm not sure what you meant by "implied variable setting".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah -- right now this block will set RABBITMQ_USE_LONGNAME to true if hostname and hostname -s have different values and RABBITMQ_USE_LONGNAME is not already set. Do you think we should keep that behavior in the future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Friendly ping -- do you think it makes sense for us to continue setting RABBITMQ_USE_LONGNAME=true for users when the result of hostname and hostname -s differ, or do you think we should deprecate that implied behavior and have users who need/want it be explicit about their environment variable instead?

@@ -195,6 +197,8 @@ if [ "${RABBITMQ_ERLANG_COOKIE:-}" ]; then
echo "$RABBITMQ_ERLANG_COOKIE" > "$cookieFile"
fi
chmod 600 "$cookieFile"

# TODO show warning about use of RABBITMQ_ERLANG_COOKIE variable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This env var is actually one of the bad necessities, as captured here: rabbitmq/rabbitmq-prometheus@83cff5a

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting -- in that case, do you think it makes sense to move the behavior/support for it into RabbitMQ itself, or would you rather it stay as a Docker-specific feature?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess what I'm explicitly proposing here is that if this environment variable is deemed necessary, perhaps RabbitMQ 3.9 should check for RABBITMQ_ERLANG_COOKIE to be non-empty (and use it if so) before it looks for/creates the cookie file?

@@ -401,6 +407,8 @@ if [ "$haveSslConfig" ] && [[ "$1" == rabbitmq* ]] && [ ! -f "$combinedSsl" ]; t
cat "$RABBITMQ_SSL_KEYFILE"
} > "$combinedSsl"
chmod 0400 "$combinedSsl"

# TODO show a warning that we created "combined" SSL file?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed.

@@ -410,6 +418,8 @@ if [ "$haveSslConfig" ] && [ -f "$combinedSsl" ]; then
sslErlArgs="-pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_opt server_certfile $combinedSsl -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"
export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="${RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS:-} $sslErlArgs"
export RABBITMQ_CTL_ERL_ARGS="${RABBITMQ_CTL_ERL_ARGS:-} $sslErlArgs"

# TODO show a warning that we set ERL_SSL_PATH, RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS, and RABBITMQ_CTL_ERL_ARGS (with their set values)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS & RABBITMQ_CTL_ERL_ARGS environment variables should be allowed. The docker-entrypoint shouldn't need to do anything about them though.

This contains all environment variables that get used before the Erlang VM starts, most of them used to configure the Erlang VM: https://github.com/rabbitmq/rabbitmq-server/blob/v3.8.5/scripts/rabbitmq-env

Here is a comprehensive list of all environment variables that RabbitMQ reads during the boot phase, after the Erlang VM starts: https://github.com/rabbitmq/rabbitmq-common/blob/v3.8.5/src/rabbit_env.erl#L39-L74

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this and the "combined" file creation are really two parts of the same whole -- we only set these if we created a combined file, and these are set so that said combined file gets used everywhere, so I think we're fully in agreement there. 👍

@gerhard
Copy link
Contributor

gerhard commented Jul 24, 2020

Comments on lines not modified yet

# Allowed env vars that will be read from mounted files (i.e. Docker Secrets):
fileEnvKeys=(
default_user
default_pass
)

Could these secrets be mounted in /etc/rabbitmq/conf.d/default_user.conf instead?


default_vhost
vm_memory_high_watermark

Could these be mounted as separate conf file instead?


# set defaults for missing values (but only after we're done with all our checking so we don't throw any of that off)
for conf in "${!configDefaults[@]}"; do
default="${configDefaults[$conf]}"
var="RABBITMQ_${conf^^}"
[ -z "${!var:-}" ] || continue
eval "export $var=\"\$default\""
done

Why configure any defaults?


rabbit_env_config() {

How is this is related to: https://github.com/rabbitmq/rabbitmq-server/blob/9b5b78d316e36c653736fa328517dc142192a1f6/scripts/rabbitmq-defaults#L25-L27


# determine whether to set "vm_memory_high_watermark" (based on cgroups)
memTotalKb=
if [ -r /proc/meminfo ]; then
memTotalKb="$(awk -F ':? +' '$1 == "MemTotal" { print $2; exit }' /proc/meminfo)"
fi
memLimitB=
if [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
# "18446744073709551615" is a valid value for "memory.limit_in_bytes", which is too big for Bash math to handle
# "$(( 18446744073709551615 / 1024 ))" = 0; "$(( 18446744073709551615 * 40 / 100 ))" = 0
memLimitB="$(awk -v totKb="$memTotalKb" '{
limB = $0;
limKb = limB / 1024;
if (!totKb || limKb < totKb) {
printf "%.0f\n", limB;
}
}' /sys/fs/cgroup/memory/memory.limit_in_bytes)"
fi
if [ -n "$memLimitB" ]; then
# if we have a cgroup memory limit, let's inform RabbitMQ of what it is (so it can calculate vm_memory_high_watermark properly)
# https://github.com/rabbitmq/rabbitmq-server/pull/1234
rabbit_set_config 'total_memory_available_override_value' "$memLimitB"
fi
# https://www.rabbitmq.com/memory.html#memsup-usage
if [ "${RABBITMQ_VM_MEMORY_HIGH_WATERMARK:-}" ]; then
# https://github.com/docker-library/rabbitmq/pull/105#issuecomment-242165822
vmMemoryHighWatermark="$(
echo "$RABBITMQ_VM_MEMORY_HIGH_WATERMARK" | awk '
/^[0-9]*[.][0-9]+$|^[0-9]+([.][0-9]+)?%$/ {
perc = $0;
if (perc ~ /%$/) {
gsub(/%$/, "", perc);
perc = perc / 100;
}
if (perc > 1.0 || perc < 0.0) {
printf "error: invalid percentage for vm_memory_high_watermark: %s (must be >= 0%%, <= 100%%)\n", $0 > "/dev/stderr";
exit 1;
}
printf "vm_memory_high_watermark.relative %0.03f\n", perc;
next;
}
/^[0-9]+$/ {
printf "vm_memory_high_watermark.absolute %s\n", $0;
next;
}
/^[0-9]+([.][0-9]+)?[a-zA-Z]+$/ {
printf "vm_memory_high_watermark.absolute %s\n", $0;
next;
}
{
printf "error: unexpected input for vm_memory_high_watermark: %s\n", $0;
exit 1;
}
'
)"
if [ "$vmMemoryHighWatermark" ]; then
vmMemoryHighWatermarkKey="${vmMemoryHighWatermark%% *}"
vmMemoryHighWatermarkVal="${vmMemoryHighWatermark#$vmMemoryHighWatermarkKey }"
rabbit_set_config "$vmMemoryHighWatermarkKey" "$vmMemoryHighWatermarkVal"
case "$vmMemoryHighWatermarkKey" in
# make sure we only set one or the other
'vm_memory_high_watermark.absolute') rabbit_comment_config 'vm_memory_high_watermark.relative' ;;
'vm_memory_high_watermark.relative') rabbit_comment_config 'vm_memory_high_watermark.absolute' ;;
esac
fi
fi

All of this looks terribly complicated. We can either ask users to configure this if limiting the container's memory or, my preference would be to handle this in RabbitMQ itself. If memory is limited via cgroups, RabbitMQ should use that value to compute the relative value. WDYT @michaelklishin?

@tianon
Copy link
Member Author

tianon commented Jul 24, 2020

Could these secrets be mounted in /etc/rabbitmq/conf.d/default_user.conf instead?
...
Could these be mounted as separate conf file instead?

Yes, absolutely -- that's exactly the intent of the warning that'll be generated by https://github.com/infosiftr/rabbitmq/blob/17680a790251d50090aeff7179a76e252d262b19/docker-entrypoint.sh#L398

My plan there is to include any setting we're currently generating a configuration value for, and to print out the full config file contents (with all the final generated settings in it), so it's very obvious that literally every thing we configure is something they could easily provide via configuration file.

What I do think this probably means we need to include is specific advice WRT Docker/Kubernetes "secret" objects, config objects, etc in the warning text itself, making it clear to users that they actually gain more flexibility from this than just environment variables can provide.

Why configure any defaults?

The end goal is not to, hence the warnings. We only do now for historical reasons. My approach with this PR was to go through the script and try and figure out the minimal set of places I could add warnings to cover any explicit or implied behavior our script is providing to warn users it'll be going away (so if they are relying on any settings/values we're putting in place, they can adjust their usage accordingly to provide those themselves).

Once we've had a suitable deprecation period with warnings being printed, it might make sense to have a further period where we check for our variables, but error+exit instead of warn to force users to handle them before we start silently ignoring them? Maybe we even keep that detection/erroring until the next major release? (That code would be significantly simpler than our current script -- just looping over a list of environment variables and throwing an error for any of them being set. 😅)

All of this looks terribly complicated. We can either ask users to configure this if limiting the container's memory or, my preference would be to handle this in RabbitMQ itself. If memory is limited via cgroups, RabbitMQ should use that value to compute the relative value. WDYT @michaelklishin?

Yeahhhh -- the gist of this code is "if the user gave us $RABBITMQ_VM_MEMORY_HIGH_WATERMARK set to some reasonable absolute or relative value, set vm_memory_high_watermark to that value; otherwise, set it to a reasonable value based on cgroup memory limits similar to what RabbitMQ itself would do based on total memory".

So, I agree it would be significantly less complicated if it were implemented in RabbitMQ itself, because I think the only change necessary is to consider both of total memory and the cgroup memory limit, and use whichever one is lower (where it currently only considers total memory).

@tianon
Copy link
Member Author

tianon commented Sep 18, 2020

Also, friendly ping for thoughts on having RabbitMQ 3.9's built-in handling of vm_memory_high_watermark take cgroup limits into account automatically? 😇

Edit: I could try my hand at a PR, but I must warn you that my Erlang is very rusty. 😄

@tianon
Copy link
Member Author

tianon commented Dec 29, 2020

Updated most of the TODO comments into explicit proposed WARNING outputs in https://github.com/docker-library/rabbitmq/compare/17680a790251d50090aeff7179a76e252d262b19..1efad3e6cca64a758e4c86c18045899810448047, which should hopefully make it more clear what I mean? The biggest one is a catch-all warning any time we generate any configuration into rabbitmq.conf, which then dumps the entire (generated/modified) file contents to the logs so that they can copy it into their configuration directly and stop setting the variables that led them to this behavior.

@tianon
Copy link
Member Author

tianon commented Dec 29, 2020

Regarding vm_memory_high_watermark, I guess that's related to rabbitmq/rabbitmq-server#1224 and thus rabbitmq/rabbitmq-server#1234, so the simple solution for users would be to set total_memory_available_override_value to the same value as the memory limit they provide to the container (assuming you don't want to update the default for that detection to take cgroup limits into account).

@tianon
Copy link
Member Author

tianon commented Dec 29, 2020

If you do want to check cgroups in RabbitMQ server itself, I'd suggest reading:

If either is a number, the "total memory" for the system should be the smallest value of the three (memory.limit_in_bytes vs memory.max vs MemTotal).

Edit: ... basically https://github.com/kovyl2404/cg_mon, but actively maintained (and likely doing more robust calculations/detection like the code being changed in erlang/otp#2922, but hard-coding /sys/fs/cgroup/... would definitely catch the majority of users)

@tianon
Copy link
Member Author

tianon commented Apr 9, 2021

Copying the relevant bits of #440 (comment) so we can try to address the final outstanding bits for this: 🙏

  1. we currently set RABBITMQ_USE_LONGNAME=true if hostname and hostname -s return different values -- should we keep that behavior, or deprecate it? Add deprecation warnings to "docker-entrypoint.sh" #424 (comment) (current PR code adds a deprecation warning in this situation; I just want confirmation that makes sense)

  2. it's been determined that RABBITMQ_ERLANG_COOKIE is generally useful -- should that functionality move into RabbitMQ itself? Add deprecation warnings to "docker-entrypoint.sh" #424 (comment) (current PR code has a TODO comment for this, no warning)

  3. should vm_memory_high_watermark in RabbitMQ itself be updated to take cgroups into account natively? Add deprecation warnings to "docker-entrypoint.sh" #424 (comment) / Add deprecation warnings to "docker-entrypoint.sh" #424 (comment) (current code will warn, but users don't have a migration path for "percentage of my cgroup limits" natively which is what our implementation was doing and why it was more complex than simply "set this value in the config")

@gerhard @michaelklishin any thoughts/opinions? 😅 😇 ❤️

@michaelklishin
Copy link
Collaborator

I personally don't have an opinion on 1. I can't think of anything better, so let's keep it unless you have evidence that it confuses enough users?

RABBITMQ_ERLANG_COOKIE is not the most secure option. Using a secret (on Kubernetes) and generating a file or mounting a file is what our team would recommend. RabbitMQ CLI tools support that or a very similar env variable.
I don't think there is any interest in making RabbitMQ nodes override the cookie. Secrets generally do not belong to env variables IMO.

If you can explain how we can compute absolute amount of memory using cgroups without any native code, I'd say it should be realistic. @gerhard @mkuratczyk do you think it's something realistic or necessary to have in RabbitMQ itself?

@tianon
Copy link
Member Author

tianon commented May 27, 2021

I personally don't have an opinion on 1. I can't think of anything better, so let's keep it unless you have evidence that it confuses enough users?

Sounds good, thanks for giving it some thought! 👍

RABBITMQ_ERLANG_COOKIE is not the most secure option. Using a secret (on Kubernetes) and generating a file or mounting a file is what our team would recommend. RabbitMQ CLI tools support that or a very similar env variable.
I don't think there is any interest in making RabbitMQ nodes override the cookie. Secrets generally do not belong to env variables IMO.

Totally fair (and is incidentally why Docker Swarm's "secrets" functionality doesn't even support environment variables 😄) - so you think we should deprecate our code around it?

If you can explain how we can compute absolute amount of memory using cgroups without any native code, I'd say it should be realistic. @gerhard @mkuratczyk do you think it's something realistic or necessary to have in RabbitMQ itself?

There's more details in #424 (comment), but the short version is that I'd suggest that where you're currently looking at MemTotal, you should instead use min(MemTotal, /sys/fs/cgroup/memory.max, /sys/fs/cgroup/memory/memory.limit_in_bytes) (if either of those files exists and contains an integer).

@michaelklishin
Copy link
Collaborator

Totally fair (and is incidentally why Docker Swarm's "secrets" functionality doesn't even support environment variables 😄) > - so you think we should deprecate our code around it?

Yes, definitenly

There's more details in #424 (comment), but the short version is that I'd suggest that where you're currently looking at
MemTotal, you should instead use min(MemTotal, /sys/fs/cgroup/memory.max,
/sys/fs/cgroup/memory/memory.limit_in_bytes) (if either of those files exists and contains an integer).

We use the proc filesystem even though I don't remember the specific location. So this suggestion sounds doable.
Is there any documentation on when /sys/fs/cgroups would or would not be available? Can I assume any kernel released in the last 5-6 years would have it?

@tianon
Copy link
Member Author

tianon commented May 27, 2021

We use the proc filesystem even though I don't remember the specific location. So this suggestion sounds doable.
Is there any documentation on when /sys/fs/cgroups would or would not be available? Can I assume any kernel released in the last 5-6 years would have it?

Yes, mostly -- the difference is which version of the cgroups filesystem is in use:

The main situation I'm aware of where they wouldn't be available is containers on older versions of Docker (before it mounted the appropriate filesystem inside the containers), and potentially if RabbitMQ isn't in an explicit memory cgroup (I think), but falling back to total memory when they aren't available (or are set to the special value max) is totally appropriate.

The /proc path you're likely reading is /proc/meminfo.

@tianon
Copy link
Member Author

tianon commented May 27, 2021

You can also set up cgroup limits with systemd, so this would have value outside containers too (although more limited because I think that's going to be a lot less common).

@tianon tianon force-pushed the entrypoint-deprecation branch 2 times, most recently from 516bf71 to 9d5e671 Compare May 27, 2021 22:10
@tianon tianon marked this pull request as ready for review May 27, 2021 22:10
@HoloRin
Copy link
Contributor

HoloRin commented May 31, 2021

With regard to RABBITMQ_ERLANG_COOKIE, it's OTP itself that reads .erlang.cookie. If it were possible to provide the value through an env var, I would expect that to be handled by OTP as well. If RabbitMQ were to do so, it would a special case that I think would be better to avoid.
If we later stop writing the file, but the user really wants to use an env var, they should be able to use RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS with -setcookie ... if I'm not mistaken.
So in short, same behavior with the new warning, as is the current state of the PR, looks good to me.

Copy link
Collaborator

@michaelklishin michaelklishin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally think this is ready to go except for the wording in one warning message.
Requesting changes (sorry) instead of leaving a comment to make this more visible. This is almost there :)

@tianon tianon force-pushed the entrypoint-deprecation branch from 9d5e671 to 5b8b491 Compare June 14, 2021 17:05
@tianon tianon force-pushed the entrypoint-deprecation branch from 1b1a4a5 to 2fab209 Compare June 14, 2021 17:06
@tianon
Copy link
Member Author

tianon commented Jun 14, 2021

@tianon tianon force-pushed the entrypoint-deprecation branch from 5b8b491 to 1b1a4a5 Compare June 14, 2021 17:07
@tianon
Copy link
Member Author

tianon commented Jun 22, 2021

Is this good now, @michaelklishin? I'd love to merge this before we get 3.9 up so that all the 3.9 builds can remove the deprecated bits, if you agree with that aggressive of a timeline 😅

@tianon tianon mentioned this pull request Jun 22, 2021
@yosifkit yosifkit merged commit 59d7871 into docker-library:master Jun 23, 2021
@yosifkit yosifkit deleted the entrypoint-deprecation branch June 23, 2021 18:22
docker-library-bot added a commit to docker-library-bot/official-images that referenced this pull request Jun 23, 2021
Changes:

- docker-library/rabbitmq@59d7871: Merge pull request docker-library/rabbitmq#424 from infosiftr/entrypoint-deprecation
- docker-library/rabbitmq@bc88db1: Merge pull request docker-library/rabbitmq#492 from mkuratczyk/master
- docker-library/rabbitmq@051164d: Switch from SKS to Ubuntu keyserver
- docker-library/rabbitmq@973c614: Add missing `\`
- docker-library/rabbitmq@38ead37: Enable JIT, install g++
- docker-library/rabbitmq@8942a4b: Merge pull request docker-library/rabbitmq#490 from J0WI/alpine-3.14
- docker-library/rabbitmq@59c5913: Alpine 3.14
- docker-library/rabbitmq@2fab209: Add deprecation warnings to "docker-entrypoint.sh"
docker-library-bot added a commit to docker-library-bot/official-images that referenced this pull request Jun 26, 2021
Changes:

- docker-library/rabbitmq@226c6de: Update 3.8-rc to 3.8.18-rc.1
- docker-library/rabbitmq@4792573: Update 3.8 to 3.8.18
- docker-library/rabbitmq@59d7871: Merge pull request docker-library/rabbitmq#424 from infosiftr/entrypoint-deprecation
- docker-library/rabbitmq@bc88db1: Merge pull request docker-library/rabbitmq#492 from mkuratczyk/master
- docker-library/rabbitmq@051164d: Switch from SKS to Ubuntu keyserver
- docker-library/rabbitmq@973c614: Add missing `\`
- docker-library/rabbitmq@38ead37: Enable JIT, install g++
- docker-library/rabbitmq@8942a4b: Merge pull request docker-library/rabbitmq#490 from J0WI/alpine-3.14
- docker-library/rabbitmq@59c5913: Alpine 3.14
- docker-library/rabbitmq@2fab209: Add deprecation warnings to "docker-entrypoint.sh"
docker-library-bot added a commit to docker-library-bot/official-images that referenced this pull request Jun 28, 2021
Changes:

- docker-library/rabbitmq@ad1824a: Update 3.8-rc to otp 24.0.3
- docker-library/rabbitmq@5414666: Update 3.8 to otp 24.0.3
- docker-library/rabbitmq@226c6de: Update 3.8-rc to 3.8.18-rc.1
- docker-library/rabbitmq@4792573: Update 3.8 to 3.8.18
- docker-library/rabbitmq@59d7871: Merge pull request docker-library/rabbitmq#424 from infosiftr/entrypoint-deprecation
- docker-library/rabbitmq@bc88db1: Merge pull request docker-library/rabbitmq#492 from mkuratczyk/master
- docker-library/rabbitmq@051164d: Switch from SKS to Ubuntu keyserver
- docker-library/rabbitmq@973c614: Add missing `\`
- docker-library/rabbitmq@38ead37: Enable JIT, install g++
- docker-library/rabbitmq@8942a4b: Merge pull request docker-library/rabbitmq#490 from J0WI/alpine-3.14
- docker-library/rabbitmq@59c5913: Alpine 3.14
- docker-library/rabbitmq@2fab209: Add deprecation warnings to "docker-entrypoint.sh"
@tianon tianon mentioned this pull request Dec 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants