Linux 4.14.4: Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`

[thelounge-zz]

Fatal: ink_sys_control.cc:59: failed assertion setrlimit(MAGIC_CAST(which), &rl) >= 0

rebootet and restarted trafficserver mutiple times - no chance
boot back to 4.13.16-200.fc26.x86_64 and everything is fine

ATS: 7.1.1

Dec 10 02:35:15 proxy traffic_server[887]: NOTE: --- traffic_server Starting ---
Dec 10 02:35:15 proxy traffic_server[887]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.1 - (build # 112114 on Nov 21 2017 at 14:12:18)
Dec 10 02:35:15 proxy traffic_server[887]: NOTE: RLIMIT_NOFILE(7):cur(675000),max(675000)
Dec 10 02:35:15 proxy traffic_server[887]: Fatal: ink_sys_control.cc:59: failed assertion setrlimit(MAGIC_CAST(which), &rl) >= 0
Dec 10 02:35:15 proxy systemd[1]: Started Process Core Dump (PID 889/UID 0).
Dec 10 02:35:15 proxy systemd-coredump[890]: Process 887 (traffic_server) of user 0 dumped core.#12#012Stack trace of thread 887:#12#0 0x00007f5d6c71e9fb raise (libc.so.6)#12#1 0x00007f5d6c72077d abort (libc.so.6)#12#2 0x00007f5d6e9533a0 _Z9ink_abortPKcz (libtsutil.so.7)#12#3 0x00007f5d6e943045 _ink_assert (libtsutil.so.7)#12#4 0x00007f5d6e954398 _Z18ink_max_out_rlimitibb (libtsutil.so.7)#12#5 0x000055812812ad0f main (traffic_server)#12#6 0x00007f5d6c70888a __libc_start_main (libc.so.6)#12#7 0x0000558128130efa _start (traffic_server)#12#012Stack trace of thread 888:#12#0 0x00007f5d6c7eef63 __select (libc.so.6)#12#1 0x00005581281ef374 _Z11mgmt_selectiP6fd_setS0_S0_P7timeval (traffic_server)#12#2 0x00005581281ef408 _Z17mgmt_read_timeoutiii (traffic_server)#12#3 0x00005581281fb5cd _ZN14ProcessManager16pollLMConnectionEv (traffic_server)#12#4 0x00005581281fb85e _Z19startProcessManagerPv (traffic_server)#12#5 0x00007f5d6cac436d start_thread (libpthread.so.0)#12#6 0x00007f5d6c7f8e1f __clone (libc.so.6)
Dec 10 02:35:18 proxy traffic_server[893]: NOTE: --- traffic_server Starting ---
Dec 10 02:35:18 proxy traffic_server[893]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.1 - (build # 112114 on Nov 21 2017 at 14:12:18)
Dec 10 02:35:18 proxy traffic_server[893]: NOTE: RLIMIT_NOFILE(7):cur(675000),max(675000)
Dec 10 02:35:18 proxy traffic_server[893]: Fatal: ink_sys_control.cc:59: failed assertion setrlimit(MAGIC_CAST(which), &rl) >= 0
Dec 10 02:35:18 proxy systemd[1]: Started Process Core Dump (PID 896/UID 0).
Dec 10 02:35:18 proxy systemd-coredump[897]: Process 893 (traffic_server) of user 0 dumped core.#12#012Stack trace of thread 893:#12#0 0x00007fd7455539fb raise (libc.so.6)#12#1 0x00007fd74555577d abort (libc.so.6)#12#2 0x00007fd7477883a0 _Z9ink_abortPKcz (libtsutil.so.7)#12#3 0x00007fd747778045 _ink_assert (libtsutil.so.7)#12#4 0x00007fd747789398 _Z18ink_max_out_rlimitibb (libtsutil.so.7)#12#5 0x000055e60d7fcd0f main (traffic_server)#12#6 0x00007fd74553d88a __libc_start_main (libc.so.6)#12#7 0x000055e60d802efa _start (traffic_server)#12#012Stack trace of thread 894:#12#0 0x00007fd745623f63 __select (libc.so.6)#12#1 0x000055e60d8c1374 _Z11mgmt_selectiP6fd_setS0_S0_P7timeval (traffic_server)#12#2 0x000055e60d8c1408 _Z17mgmt_read_timeoutiii (traffic_server)#12#3 0x000055e60d8cd5cd _ZN14ProcessManager16pollLMConnectionEv (traffic_server)#12#4 0x000055e60d8cd85e _Z19startProcessManagerPv (traffic_server)#12#5 0x00007fd7458f936d start_thread (libpthread.so.0)#12#6 0x00007fd74562de1f __clone (libc.so.6)

[thelounge-zz]

maybe that stuff from /var/log/trafficserver/traffic.out is better readable

i also temporary removed all systemd-namespaces and seccomp filters
no change, a few seconds after start it crashs on 4.1.4 and from my expierience the assert is a bug

Fatal: ink_sys_control.cc:59: failed assertion setrlimit(MAGIC_CAST(which), &rl) >= 0
traffic_server: using root directory '/usr'
traffic_server - STACK TRACE:
/usr/lib64/libpthread.so.0(+0x123b0)[0x7feef76393b0]
/usr/lib64/libc.so.6(gsignal+0xcb)[0x7feef72889fb]
/usr/lib64/libc.so.6(abort+0x1b0)[0x7feef728a800]
/usr/lib64/trafficserver/libtsutil.so.7(_Z25signal_start_check_threadPFviP9siginfo_tPvE+0x0)[0x7feef94bd3a0]
/usr/lib64/trafficserver/libtsutil.so.7(+0x1f045)[0x7feef94ad045]
/usr/lib64/trafficserver/libtsutil.so.7(_Z18ink_max_out_rlimitibb+0x118)[0x7feef94be398]
/usr/bin/traffic_server(main+0x6ff)[0x560417b4fd0f]
/usr/lib64/libc.so.6(__libc_start_main+0xea)[0x7feef727288a]
/usr/bin/traffic_server(_start+0x2a)[0x560417b55efa]
[TrafficManager] ==> Cleaning up and reissuing signal #15

[thelounge-zz]

https://bugzilla.redhat.com/show_bug.cgi?id=1524083

and i agree here - set os-limits within the software is broken by design

--- Comment #1 from Laura Abbott labbott@redhat.com ---
The change is caused by commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=04e35f4495dd560db30c25efca4eecae8ec8c375
. Based on discussions with the patch author (Kees Cook), what's happening is
the stack limit once got lowered so future attempts to set it to ulimited are
breaking the call to setrlimit.

The userspace checking here is pretty buggy though. Relying on setulimit to
always succeed is asking for trouble. Kees also noted the check is incorrect
(should be checking r.rlim_cur != r.rlim_max). The behavior of no kernel
regressions doesn't apply to buggy programs so I don't think there is anything
for the kernel to fix here. I moved this to trafficserver for it to get fixed
there.

[zwoop]

Nice catch. We should fix this, patches welcome too :-). But this should likely go into both master, and 7.x releases.

[thelounge-zz]

this must be fixed in 7.x - Kernel 4.13 is EOL, you won't see any kernel security update on Fedora which is not 4.14 or 4.15 in a few months but when you run ATS currently you must hold back kernel updates - sorry, ican't provide a patch, wrong programming language

[root@proxy:~]$ cat /etc/yum.repos.d/lounge-buildserver.repo | grep exclude
excludepkgs=kernel-core-4.14.4-200.fc26.x86_64

[zwoop]

Does #2929 seem reasonable? It merely changes the assertion, and does not try to address the actual issue of calling the setrlimits. I'm slightly concerned that this is just a bandaid solution, but if you are ok with it, this would be easy and fine to get into 7.x.

[thelounge-zz]

either i am to dumb for github or there is no way to get a "unified patch file" which can be included within a rpmbuild, please give me such a file and i will rebuild 7.1.1, test if everything works ok, deploy it to production and upgrade the kernel

[thelounge-zz]

ok - googled it add .patch to the URL - patch works!

BTW - what's that?
20171214.20h11m12s [header_rewrite] Unknown condition: INBOUND
20171214.20h11m12s [header_rewrite] failed to parse configuration file /etc/trafficserver/header_rewrite.config

[root@buildserver:/var/log/trafficserver]$ cat /etc/trafficserver/header_rewrite.config
cond %{SEND_REQUEST_HDR_HOOK}
cond %{INBOUND:TLS} NOT ""
set-header X-TL-TLS-Offload 1
set-header X-TL-TLS-Offload-Port %{INCOMING-PORT}

[root@buildserver:/var/log/trafficserver]$ cat *
[Dec 14 20:11:12.305] Server {0x7f48144bbe80} STATUS: opened /var/log/trafficserver/diags.log
[Dec 14 20:11:12.305] Server {0x7f48144bbe80} NOTE: updated diags config
[Dec 14 20:11:12.309] Server {0x7f48144bbe80} NOTE: cache clustering disabled
[Dec 14 20:11:12.311] Server {0x7f48144bbe80} NOTE: ip_allow.config updated, reloading
[Dec 14 20:11:12.314] Server {0x7f48144bbe80} NOTE: cache clustering disabled
[Dec 14 20:11:12.315] Server {0x7f48144bbe80} NOTE: logging initialized[3], logging_mode = 1
[Dec 14 20:11:12.316] Server {0x7f48144bbe80} NOTE: loading plugin '/usr/lib64/trafficserver/plugins/header_rewrite.so'
[Dec 14 20:11:12.319] Server {0x7f48144bbe80} NOTE: loading SSL certificate configuration from /etc/trafficserver/ssl_multicert.config
[Dec 14 20:11:12.327] Server {0x7f48144bbe80} WARNING: previously indexed '*.thelounge.net' with SSL_CTX (nil), cannot index it with SSL_CTX #2 now
[Dec 14 20:11:12.327] Server {0x7f48144bbe80} WARNING: previously indexed 'thelounge.net' with SSL_CTX 0x1, cannot index it with SSL_CTX #2 now
[Dec 14 20:11:12.335] Server {0x7f48144bbe80} WARNING: previously indexed 'contentlounge.buildserver.thelounge.net' with SSL_CTX 0x2, cannot index it with SSL_CTX #3 now
[Dec 14 20:11:12.344] Server {0x7f48144bbe80} WARNING: previously indexed 'default.buildserver.thelounge.net' with SSL_CTX 0x3, cannot index it with SSL_CTX #4 now
[Dec 14 20:11:12.358] Server {0x7f48144bbe80} WARNING: Unrecognized configuration value 'proxy.config.http.normalize_ae'
[Dec 14 20:11:12.359] Server {0x7f48144bbe80} NOTE: traffic server running
[Dec 14 20:11:12.384] Server {0x7f480ef0c700} NOTE: cache enabled
20171214.20h11m12s [header_rewrite] Unknown condition: INBOUND
20171214.20h11m12s [header_rewrite] failed to parse configuration file /etc/trafficserver/header_rewrite.config
20171214.20h11m12s [header_rewrite] failed to parse any configuration file
[Dec 14 20:11:09.263] Manager {0x7f9308266cc0} STATUS: opened /var/log/trafficserver/manager.log
[Dec 14 20:11:09.263] Manager {0x7f9308266cc0} NOTE: updated diags config
[Dec 14 20:11:09.289] Manager {0x7f9308266cc0} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '4.14.4-200.fc26.x86_64'
[Dec 14 20:11:09.289] Manager {0x7f9308266cc0} NOTE: [LocalManager::listenForProxy] Listening on port: 8080 (ipv4)
[Dec 14 20:11:09.289] Manager {0x7f9308266cc0} NOTE: [LocalManager::listenForProxy] Listening on port: 8443 (ipv4)
[Dec 14 20:11:09.289] Manager {0x7f9308266cc0} NOTE: [TrafficManager] Setup complete
[Dec 14 20:11:10.291] Manager {0x7f9308266cc0} NOTE: [ProxyStateSet] Traffic Server Args: '--bind_stdout /var/log/trafficserver/traffic.out --bind_stderr /var/log/trafficserver/traffic.out -M'
[Dec 14 20:11:10.291] Manager {0x7f9308266cc0} NOTE: [LocalManager::listenForProxy] Listening on port: 8080 (ipv4)
[Dec 14 20:11:10.291] Manager {0x7f9308266cc0} NOTE: [LocalManager::listenForProxy] Listening on port: 8443 (ipv4)
[Dec 14 20:11:10.291] Manager {0x7f9308266cc0} NOTE: [LocalManager::startProxy] Launching ts process
[Dec 14 20:11:10.297] Manager {0x7f9308266cc0} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '15'
[Dec 14 20:11:10.297] Manager {0x7f9308266cc0} NOTE: [Alarms::signalAlarm] Server Process born

[duke8253]

Yeah, we discussed this issue here at yahoo and this patch is just an temporary fix to get ats working on 4.14+ kernels. I'm still working on an actual fix. And just to note, this issue is caused by not having permission to set the limit correctly, so we're thinking we might need to elevate to root and set it, or just set it externally then run ats.

[thelounge-zz]

don't set kernel limits at all - it's not the job of userland

if Fedora would pull systemd/systemd#6952 on F26 you would not have the permissions even as root on a proper setup because "SystemCallFilter" would contain @resources too and you could only read the limits

LimitNOFILE=1000000
LimitMEMLOCK=infinity
TasksMax=30000
PrivateTmp=yes
MemoryDenyWriteExecute=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_LOCAL AF_UNIX AF_NETLINK
CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE CAP_IPC_LOCK CAP_KILL CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @reboot @swap acct modify_ldt add_key adjtimex clock_adjtime delete_module fanotify_init finit_module get_mempolicy init_module io_destroy io_getevents iopl ioperm io_setup io_submit io_cancel kcmp kexec_load keyctl lookup_dcookie mbind migrate_pages mount move_pages open_by_handle_at perf_event_open pivot_root process_vm_readv process_vm_writev ptrace remap_file_pages request_key set_mempolicy swapoff swapon umount2 uselib vmsplice

[duke8253]

Yeah, we just had another discussion about that, and we're thinking just remove the code for changing the hard limit and only keep the code for setting the soft limit.

[zwoop]

Lets resolve (close) this issue, and open another issue on the removal of the setrlimit for the hard limits. @thelounge-zz can you open such an issue? And perhaps link to this discussion as well.