Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WFCORE-6935] Improve systemd service units #6179

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
= How to configure WildFly as a systemd service

The following steps describe the process to configure WildFly as a systemd service.
You can use the provided wildfly-[standalone,domain].service file as a template for your own systemd unit file.
It can be adjusted to your needs, for example, you can manually change the user that runs the service,
the location of the WildFly installation, logs, etc.
Alternatively, you can use the generate_systemd_unit.sh script to automatically generate a new systemd unit file
using this server installation as the WildFly home.

If you want to install WildFly as a systemd service from scratch as a systemd service, follow the steps below.

== Install WildFly and initialize the WILDFLY_HOME variable

# unzip -d /opt wildfly-34.0.0.Final.zip
# ln -s /opt/wildfly-34.0.0.Final /opt/wildfly
Comment on lines +14 to +15
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we might want to do here is use ${project.version} and allow processing of this resource to replace the properties. We could probably do something for wildfly too so it's generic.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to check but not sure if that would be fine taking into account that this file contains some other linux commands, I don't want to make them be replaced by maven applying environment properties across this file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. I don't see any others with the pattern ${property.name}, but it's definitely something to consider since that is the linux way.

I was mostly trying to think of a way where we don't have to keep updating the version, but maybe it doesn't matter.

# WILDFLY_HOME=/opt/wildfly

== Standalone or Domain mode ?

# MODE=standalone

== Create a wildfly user and group

# groupadd -r wildfly
# useradd -r -g wildfly -d "${WILDFLY_HOME}" -s /sbin/nologin wildfly

== Configure systemd

# cd "${WILDFLY_HOME}/bin/systemd"
# cp "wildfly-${MODE}.conf" /etc/sysconfig/
# cp "wildfly-${MODE}.service" $(pkg-config systemd --variable=systemdsystemunitdir)
# chown -R wildfly:wildfly "${WILDFLY_HOME}"/
# chmod +x "${WILDFLY_HOME}/bin/systemd/launch.sh"
# systemctl daemon-reload

=== If SELinux is enabled, you need to set the appropriate context for the launch.sh script:

# sudo chcon -R -t bin_t "${WILDFLY_HOME}/"
# sudo semanage fcontext -a -t systemd_unit_file_t '${WILDFLY_HOME}(/.*)?'

== Start and enable WildFly

# systemctl start "wildfly-${MODE}.service"
# systemctl enable "wildfly-${MODE}.service"

== Check the status of WildFly

# systemctl status "wildfly-${MODE}.service"



= How to remove WildFly as a systemd service

== Standalone or Domain mode ?

# MODE=standalone

== Stop and disable WildFly

# systemctl stop "wildfly-${MODE}.service"
# systemctl disable "wildfly-${MODE}.service"

== If you are using SELinux, remove the custom context added

sudo semanage fcontext -d '${WILDFLY_HOME}(/.*)?'

== Remove WildFly systemd service

# rm -f "$(pkg-config systemd --variable=systemdsystemunitdir)/wildfly-${MODE}.service"
# rm -f "/etc/sysconfig/wildfly-${MODE}.conf"
# systemctl daemon-reload
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not add systemctl status wildfy and systemctl start wildfly with an example of the output?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point, this is the section that describes how to remove the service.

The start / enable / status commands are described in the previous section. There is no sample of status output, though


== Remove WildFly installation

# rm -rf $(readlink "${WILDFLY_HOME}")
# rm -rf "${WILDFLY_HOME}"

== Remove WildFly user and group

# userdel wildfly
# groupdel wildfly
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/bin/sh

# This script is just a helper to generate the systemd unit files for WildFly
# assuming that the server home will be the one that hold this script file.
# It also allows to specify the user and group that will run the server.

function print_usage() {
echo "INFO: Usage: "
echo "${SCRIPT_NAME} -m,--mode standalone|domain [-u,--user user] [-g,--group group] [-c,--confirm y|n]"

exit 1
}

SCRIPT_NAME=$(basename "$0")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCRIPT_NAME=$(basename "$0") to SCRIPT_NAME=$(basename "${0}")

SCRIPT_DIR="$(dirname "$(realpath "$0")")"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCRIPT_DIR="$(dirname "$(realpath "$0")")" to SCRIPT_DIR="$(dirname "$(realpath "${0}")")"

JBOSS_HOME="$(realpath "${SCRIPT_DIR}/../..")"
JBOSS_SYSTEMD_MODE="${1}"

JBOSS_SYSTEMD_MODE=""
JBOSS_SYSTEMD_USER="wildfly"
JBOSS_SYSTEMD_GROUP="wildfly"
JBOSS_CONFIRM="n"

while [ "$#" -gt 0 ]
do
# For each argument option, checks whether the argument length is zero or starts with hyphen(s)
case "$1" in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can use caseopts to simplify a bit this part of the script.

OPTSTRING=":ab"

while getopts ${OPTSTRING} opt; do
  case ${opt} in
    a)
      echo "Option -a was triggered."
      ;;
    b)
      echo "Option -b was triggered."
      ;;
    ?)
      echo "Invalid option: -${OPTARG}."
      exit 1
      ;;
  esac
done

-m|--mode)
if [ -z "$2" ] || [ ! "$2" = `echo "$2" | sed 's/^-*//'` ]; then
echo "Error: Missing argument for $1" >&2
print_usage
fi
JBOSS_SYSTEMD_MODE="$2"
shift 2
;;
-u|--user)
if [ -z "$2" ] || [ ! "$2" = `echo "$2" | sed 's/^-*//'` ]; then
echo "Error: Missing argument for $1" >&2
print_usage
fi
JBOSS_SYSTEMD_USER="$2"
shift 2
;;
-g|--group)
if [ -z "$2" ] || [ ! "$2" = `echo "$2" | sed 's/^-*//'` ]; then
echo "Error: Missing argument for $1" >&2
print_usage
fi
JBOSS_SYSTEMD_GROUP="$2"
shift 2
;;
-c|--confirm)
if [ -z "$2" ] || [ ! "$2" = `echo "$2" | sed 's/^-*//'` ]; then
echo "Error: Missing argument for $1" >&2
print_usage
fi
JBOSS_CONFIRM="$2"
shift 2
;;
*)
echo "Error: Unknown argument $1" >&2
print_usage
;;
esac
done

if [ "${JBOSS_SYSTEMD_MODE}" != "standalone" ] && [ "${JBOSS_SYSTEMD_MODE}" != "domain" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting into super nickpicking zone :)

Technically by using "" you are telling the script interpreter to look for cmd substitution or variable to be replaced by their content into the string. So it would be better to have 'standalone' and 'domain' instead of "standalone" and "domain".

I did say it was nickpicking, right? ;)

print_usage
fi

SYSTEMD_FILE="${SCRIPT_DIR}/wildfly-${JBOSS_SYSTEMD_MODE}.service"

echo "INFO: systemd unit file to generate: ${SYSTEMD_FILE}"
echo "INFO: Using JBOSS_HOME: ${JBOSS_HOME}"
echo "INFO: User: ${JBOSS_SYSTEMD_USER}"
echo "INFO: Group: ${JBOSS_SYSTEMD_GROUP}"
echo "INFO: Confirm: ${JBOSS_CONFIRM}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not echo this variable, as it will only be used for interactive mode. Having this value in logfile of automated install will be ... strange, and might confused ops ("Wait, why my JBOSS is not confirmed? What does it means???".
If you intend to echo the value for debugging purpose, remember that simply running the script with the -x option will echo all the variable values.


if [ 'y' != "${JBOSS_CONFIRM}" ] && [ 'Y' != "${JBOSS_CONFIRM}" ];then
while true; do
read -p "Do you want to generate ${SYSTEMD_FILE}? (y/n): " choice
case "$choice" in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

case "$choice" in to case "${choice}" in

Also, as I suspect choice will not change value after the script starts, I would suggest to go uppercase to cleary display that (so rename it to CHOICE).

Ideally, I would prefix any vars declaration by readonly, but I'm not sure if this supported across all interpreter or just a bash feature. It's a good practise to use readonly, because it prevent some already defined variable in the shell to override a value in the script.

y|Y ) break;;
n|N ) echo "Operation cancelled."; exit 1;;
* ) ;;
esac
done
fi

sed -e "s|@@@JBOSS_SYSTEMD_SERVER_HOME@@@|${JBOSS_HOME}|g" \
-e "s|@@@JBOSS_SYSTEMD_USER@@@|${JBOSS_SYSTEMD_USER}|g" \
-e "s|@@@JBOSS_SYSTEMD_GROUP@@@|${JBOSS_SYSTEMD_GROUP}|g" \
"${SYSTEMD_FILE}.template" > "${SYSTEMD_FILE}"

systemd-analyze verify --recursive-errors=no "${SYSTEMD_FILE}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh cool! I'm so going to steal that for the Ansible collection for Wildfly! :)


echo ""
echo "INFO: systemd unit file generated."
echo "INFO: The ${JBOSS_SYSTEMD_USER}:${JBOSS_SYSTEMD_GROUP} are the user:group configured to launch the server. You have to ensure this user and group exist and have the necessary permissions to read and launch the server."
echo "INFO: Use ${JBOSS_HOME}/bin/systemd/wildfly-${JBOSS_SYSTEMD_MODE}.conf to configure the server environment."
echo "INFO: After configuring your environment, to install the server as a systemd service do the following:"
echo ""
echo "sudo cp ${SYSTEMD_FILE} $(pkg-config systemd --variable=systemdsystemunitdir)"
echo "sudo cp ${JBOSS_HOME}/bin/systemd/wildfly-${JBOSS_SYSTEMD_MODE}.conf /etc/sysconfig"
echo "sudo systemctl daemon-reload"
echo "sudo systemctl start $(basename "${SYSTEMD_FILE}")"
echo ""
echo "INFO: In case of issues, you can check the service logs with:"
echo "sudo journalctl -u $(basename "${SYSTEMD_FILE}")"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure most ops are going to be thrilled by this output. You may want to consider displaying it only if JBOSS_CONFIRM is set to true? (the more I think about it, the more I think a less confusing name would be better for this variable).

Also, you might want to move this to a separate file and a do a sort of template as you did for the configuratiion file. It might be easier to maintain on the long run.

Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest /bin/bash so that we can use bash specific mechanism. It also better because with /bin/sh we should test the script with all sort of shell interpreter.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use /sh to maximize the portability now and in the future, it is the same shell used by the launch scripts. Using bash opens the door to not be compatible in the future.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but do you ensure that CI test the script with different flavor of shell? Unlikely... By using /sh you are opening the door to people filling issue because it does not work with "not Bash". It's generally consider best practises to use explicit /bin/bash for this very reason. As this is a new script, I would take the chance to reduce the scope of incoming issue. That being said, it's not a big deal.

if [ "$WILDFLY_SYSTEMD_DEBUG" == "true" ]; then
set -x
fi

echo "INFO: Systemd Unit File server launch script"

# Disable color output for the standard output
export JAVA_OPTS="${JAVA_OPTS} -Dorg.jboss.logmanager.nocolor=true"
export PATH="${JAVA_HOME}/bin:${PATH}"

logDir=$(dirname "${WILDFLY_CONSOLE_LOG}")
if [ ! -d "${logDir}" ]; then
mkdir -p "${logDir}"
fi

wildflyOpts="${WILDFLY_OPTS}"
if [ -n "${WILDFLY_SUSPEND_TIMEOUT}" ]; then
wildflyOpts="-Dorg.wildfly.sigterm.suspend.timeout=${WILDFLY_SUSPEND_TIMEOUT} ${wildflyOpts}"
fi

if [ -z "${WILDFLY_HOST_CONFIG}" ]; then
"${WILDFLY_SH}" -c "${WILDFLY_SERVER_CONFIG}" -b "${WILDFLY_BIND}" ${wildflyOpts} > "${WILDFLY_CONSOLE_LOG}" 2>&1
else
"${WILDFLY_SH}" --host-config="${WILDFLY_HOST_CONFIG}" -c "${WILDFLY_SERVER_CONFIG}" -b "${WILDFLY_BIND}" ${wildflyOpts} > "${WILDFLY_CONSOLE_LOG}" 2>&1
fi
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# The configuration you want to run

# Location of java in the JRE (the default)
#JAVA_HOME=/usr/lib/jvm/jre
# Use the following for location of java in the SDK
#JAVA_HOME=/usr/lib/jvm/java


# Configure the suspend timeout. By default, the suspend timeout is disabled, meaning that the server will not wait
# in the SUSPENDING state for any in-flight requests. This property configures the maximum number of seconds to wait for
# in-flight requests to complete. A value of -1 means the server will wait indefinitely.
#
# See https://docs.wildfly.org/32/Admin_Guide.html#graceful-shutdown-from-an-os-signal
#
# Note that increasing this value will extend the time the service unit takes to shut down the server.
# If WILDFLY_SUSPEND_TIMEOUT is configured, ensure that the TimeoutStopSec directive in the systemd service unit
# is set to a value that allows the server to wait until the suspend timeout expires and the server is stopped.
#WILDFLY_SUSPEND_TIMEOUT=300

# Location of the server standard out, e.g /opt/wildfly/standalone/log/service-console.log
#WILDFLY_CONSOLE_LOG="/opt/wildfly/standalone/log/service-console.log"

# Define the script to use to start wildfly
#WILDFLY_SH="/opt/wildfly/bin/domain.sh"

# Define server configuration to start, eg. domain.xml
#WILDFLY_SERVER_CONFIG=domain.xml

# Define the host controller configuration, eg. host.xml
#WILDFLY_HOST_CONFIG=host.xml

# The address to bind to
#WILDFLY_BIND=0.0.0.0

# Additional server args to include on startup
# For example, to add two properties to the server:
#WILDFLY_OPTS="-Dproperty1=value1 -Dproperty2=value2"

# Enables debug traces for the server launch script used by the systemd service unit
#WILDFLY_SYSTEMD_DEBUG=true
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[Unit]
Description=WildFly (domain mode)
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
Environment="JAVA_HOME=/usr/lib/jvm/jre"

Environment="WILDFLY_SH=/opt/wildfly/bin/domain.sh"
Environment="WILDFLY_SERVER_CONFIG=domain.xml"
Environment="WILDFLY_CONSOLE_LOG=/opt/wildfly/domain/log/systemd-console.log"
Environment="WILDFLY_HOST_CONFIG="
Environment="WILDFLY_BIND=0.0.0.0"
EnvironmentFile=-/etc/sysconfig/wildfly-domain.conf
jamezp marked this conversation as resolved.
Show resolved Hide resolved

User=wildfly
Group=wildfly
LimitNOFILE=102642
ExecStart=/bin/sh -c "/opt/wildfly/bin/systemd/launch.sh"

[Install]
WantedBy=multi-user.target
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[Unit]
Description=WildFly (domain mode)
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
Environment="JAVA_HOME=/usr/lib/jvm/jre"

Environment="WILDFLY_SH=@@@JBOSS_SYSTEMD_SERVER_HOME@@@/bin/domain.sh"
Environment="WILDFLY_SERVER_CONFIG=domain.xml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICS the ony difference between domain and standlone is this variable. I would suggest to avoid duplication, and use some bash savyness to avoid duplicating the files (and this have only one template and one file):

Environment="WILDFLY_SERVER_CONFIG=${CONFIG_FILE:-'standalone.xml'}" # Then we just need to ensure that CONFIG_FILE defaults to domain.xml in the launch script.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, looking at the next file I reallized, we might simply be able to WILDFLY_SERVER_CONFIG which defaults to 'standalone.xml'.

Environment="WILDFLY_CONSOLE_LOG=@@@JBOSS_SYSTEMD_SERVER_HOME@@@/domain/log/systemd-console.log"
Environment="WILDFLY_HOST_CONFIG="
Environment="WILDFLY_BIND=0.0.0.0"
EnvironmentFile=-/etc/sysconfig/wildfly-domain.conf

User=@@@JBOSS_SYSTEMD_USER@@@
Group=@@@JBOSS_SYSTEMD_GROUP@@@
LimitNOFILE=102642
ExecStart=/bin/sh -c "@@@JBOSS_SYSTEMD_SERVER_HOME@@@/bin/systemd/launch.sh"

[Install]
WantedBy=multi-user.target
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# The configuration you want to run

# Location of java in the JRE (the default)
#JAVA_HOME=/usr/lib/jvm/jre
# Use the following for location of java in the SDK
#JAVA_HOME=/usr/lib/jvm/java


# Configure the suspend timeout. By default, the suspend timeout is disabled, meaning that the server will not wait
# in the SUSPENDING state for any in-flight requests. This property configures the maximum number of seconds to wait for
# in-flight requests to complete. A value of -1 means the server will wait indefinitely.
#
# See https://docs.wildfly.org/32/Admin_Guide.html#graceful-shutdown-from-an-os-signal
#
# Note that increasing this value will extend the time the service unit takes to shut down the server.
# If WILDFLY_SUSPEND_TIMEOUT is configured, ensure that the TimeoutStopSec directive in the systemd service unit
# is set to a value that allows the server to wait until the suspend timeout expires and the server is stopped.
#WILDFLY_SUSPEND_TIMEOUT=300

# Location of the server standard out, e.g /opt/wildfly/standalone/log/service-console.log
#WILDFLY_CONSOLE_LOG="/opt/wildfly/standalone/log/service-console.log"

# Define the script to use to start wildfly
#WILDFLY_SH="/opt/wildfly/bin/standalone.sh"

# Define server configuration to start, eg. standalone.xml
#WILDFLY_SERVER_CONFIG=standalone.xml

# The address to bind to
#WILDFLY_BIND=0.0.0.0

# Additional server args to include on startup
# For example, to add two properties to the server:
#WILDFLY_OPTS="-Dproperty1=value1 -Dproperty2=value2"

# Enables debug traces for the server launch script used by the systemd service unit
#WILDFLY_SYSTEMD_DEBUG=true
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[Unit]
Description=WildFly (standalone mode)
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
Environment="JAVA_HOME=/usr/lib/jvm/jre"

Environment="WILDFLY_SH=/opt/wildfly/bin/standalone.sh"
Environment="WILDFLY_SERVER_CONFIG=standalone.xml"
Environment="WILDFLY_CONSOLE_LOG=/opt/wildfly/standalone/log/systemd-console.log"
Environment="WILDFLY_BIND=0.0.0.0"
EnvironmentFile=-/etc/sysconfig/wildfly-standalone.conf

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparing to the service configuration we use for the Ansible collection for Wildfly, there is a few options you might to consider adding:

  1. Creating a PIDFile and giving it to systemd will sensibly enhance its ability to manage the server:
PIDFile=/opt/wildfly/standalone/wildfly.pid
  1. Adding the sucessful exit statuses will prevent Systemd to misinterpret when the server is successfully stopping:
SuccessExitStatus = 0 143
  1. As Java services tends to start significantly slower than native ones, configuring a timeout will prevent a scenario where apps deployment on the service boot causes Wildfly to be tagged as "fail to start":
TimeoutSec=600
  1. That more of a nice to have, but you could configure to automatically restart the service in case of failure:
RestartSec = 120
Restart = on-failure

User=wildfly
Group=wildfly
LimitNOFILE=102642
ExecStart=/bin/sh -c "/opt/wildfly/bin/systemd/launch.sh"


[Install]
WantedBy=multi-user.target
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[Unit]
Description=WildFly (standalone mode)
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
Environment="JAVA_HOME=/usr/lib/jvm/jre"

Environment="WILDFLY_SH=@@@JBOSS_SYSTEMD_SERVER_HOME@@@/bin/standalone.sh"
Environment="WILDFLY_SERVER_CONFIG=standalone.xml"
Environment="WILDFLY_CONSOLE_LOG=@@@JBOSS_SYSTEMD_SERVER_HOME@@@/standalone/log/systemd-console.log"
Environment="WILDFLY_BIND=0.0.0.0"
EnvironmentFile=-/etc/sysconfig/wildfly-standalone.conf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you the configuration suggested above to the service conf, you should probably update the template accordingly.


User=@@@JBOSS_SYSTEMD_USER@@@
Group=@@@JBOSS_SYSTEMD_GROUP@@@
LimitNOFILE=102642
ExecStart=/bin/sh -c "@@@JBOSS_SYSTEMD_SERVER_HOME@@@/bin/systemd/launch.sh"

[Install]
WantedBy=multi-user.target
Loading
Loading