scylla-jmx.service: Failed at step CHDIR spawning /usr/lib/scylla/jmx/scylla-jmx: No such file or dir

[jefrati13]

formatting nvme drive to ext4, then setting workdir to that drive and removing old data and commitlog directories, then running scylla_setup causes jmx server to crash

systemd[8722]: scylla-jmx.service: Changing to the requested working directory failed: No such file or directory
systemd[8722]: scylla-jmx.service: Failed at step CHDIR spawning /usr/lib/scylla/jmx/scylla-jmx: No such file or dir
systemd[1]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
scylla-jmx.service: Failed with result 'exit-code'.

for some reason
/usr/lib/systemd/system/scylla-jmx.service isnt regenerated
and WorkingDirectory is left at old path.
setting new path doesn't help and sends back the same error.

only workaround so far is leaving empty path like in this 3 year old commit
#6311525

hopefully this saves people formatting and mounting new drives some time

[mykaul]

It's unclear to me if scylla was properly reinstalled. A log of the installation would be helpful.

[jefrati13]

scylla has been running for 350 hrs uptime no issues before, same installation as i have did in past. cql, rpc, api, evreything worked.
when replacing a broken nvme mounted as / with a new nvme mounted at /mnt/db-example/scylla the config was also edited to change all directory paths and working dir path to the new mounted nvme. scylla server starts properly, connects to all nodes as expected. only issue is nodetool that is unable to connect to api at port 10000 with the chdir error.
leaving a blank entry for workjing dir in
/usr/lib/systemd/system/scylla-jmx.service
and reloading daemon the restrating scylla-jmx.service solves issue for some reason

root@scylla3:~# systemctl status scylla-jmx.service
● scylla-jmx.service - Scylla JMX
     Loaded: loaded (/lib/systemd/system/scylla-jmx.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/scylla-jmx.service.d
             └─sysconfdir.conf
     Active: failed (Result: exit-code) since Sun 2024-02-11 08:16:12 WET; 2s ago
    Process: 325523 ExecStart=/opt/scylladb/jmx/scylla-jmx $SCYLLA_JMX_PORT $SCYLLA_API_PORT $SCYLLA_API_ADDR $SCYLLA_JMX_ADDR $SCYLLA_JMX_FILE $SCYLLA_JMX>
   Main PID: 325523 (code=exited, status=200/CHDIR)
        CPU: 2ms

Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: Started Scylla JMX.
Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: scylla-jmx.service: Failed with result 'exit-code'.

root@scylla3:~# systemctl status scylla-jmx.service
● scylla-jmx.service - Scylla JMX
     Loaded: loaded (/lib/systemd/system/scylla-jmx.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/scylla-jmx.service.d
             └─sysconfdir.conf
     Active: failed (Result: exit-code) since Sun 2024-02-11 08:16:12 WET; 8s ago
    Process: 325523 ExecStart=/opt/scylladb/jmx/scylla-jmx $SCYLLA_JMX_PORT $SCYLLA_API_PORT $SCYLLA_API_ADDR $SCYLLA_JMX_ADDR $SCYLLA_JMX_FILE $SCYLLA_JMX>
   Main PID: 325523 (code=exited, status=200/CHDIR)
        CPU: 2ms

Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: Started Scylla JMX.
Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
Feb 11 08:16:12 scylla3.datacon-il.com systemd[1]: scylla-jmx.service: Failed with result 'exit-code'.

root@scylla3:~# nano /usr/lib/systemd/system/scylla-jmx.service
root@scylla3:~# systemctl daemon-reload
root@scylla3:~# systemctl restart scylla-jmx.service
root@scylla3:~# systemctl status scylla-jmx.service
● scylla-jmx.service - Scylla JMX
     Loaded: loaded (/lib/systemd/system/scylla-jmx.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/scylla-jmx.service.d
             └─sysconfdir.conf
     Active: active (running) since Sun 2024-02-11 08:17:04 WET; 2s ago
   Main PID: 325584 (java)
      Tasks: 30 (limit: 309060)
     Memory: 156.4M
        CPU: 2.162s
     CGroup: /scylla.slice/scylla-helper.slice/scylla-jmx.service
             └─325584 /usr/lib/jvm/default-java/bin/java -Xmx256m -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.authentic>

Feb 11 08:17:04 scylla3.datacon-il.com systemd[1]: Started Scylla JMX.
Feb 11 08:17:04 scylla3.datacon-il.com scylla-jmx[325584]: Picked up JAVA_TOOL_OPTIONS:
Feb 11 08:17:04 scylla3.datacon-il.com scylla-jmx[325584]: Using config file: /etc/scylla/scylla.yaml
Feb 11 08:17:04 scylla3.datacon-il.com scylla-jmx[325584]: Connecting to http://192.168.44.3:10000
Feb 11 08:17:04 scylla3.datacon-il.com scylla-jmx[325584]: Starting the JMX server
Feb 11 08:17:04 scylla3.datacon-il.com scylla-jmx[325584]: JMX is enabled to receive remote connections on port: 7199

as you can see my log is broken since im not near my pc and using windows for some reason cmd doesnt allow me to copy evreything, ill edit this with full logs tommorow once im near my pc.