Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaked zqd process on ungraceful brim shutdown #1018

Closed
mattnibs opened this issue Aug 24, 2020 · 1 comment · Fixed by #1031
Closed

Leaked zqd process on ungraceful brim shutdown #1018

mattnibs opened this issue Aug 24, 2020 · 1 comment · Fixed by #1031
Assignees

Comments

@mattnibs
Copy link
Contributor

mattnibs commented Aug 24, 2020

If brim is ungracefully shutdown the zqd process launched by brim continues on. There should be some functionality in zqd (or mode for running zqd) where zqd will terminate if the parent Brim process ceases to exist.

Steps to reproduce:

  1. Start brim
  2. Find brim pid
  3. Run kill -9 $brimpid
  4. Observe that the brim launched zqd process is still running ps aux | grep zqd

Note: This does not appear to be a problem on the Windows version of Brim.

@mattnibs mattnibs self-assigned this Aug 24, 2020
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit to brimdata/super that referenced this issue Aug 27, 2020
Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF brimdata/zui#1018
mattnibs added a commit that referenced this issue Aug 27, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
brim-bot pushed a commit that referenced this issue Aug 27, 2020
…y mattnibs

This is an auto-generated commit with a zq dependency update. The zq PR
brimdata/super#1184, authored by @mattnibs,
has been merged.

zqd: prevent being orphaned by Brim (unix only)

Add hidden -brimfd flag to zqd listen. Brim will use this option
to prevent zqd from being orphaned in the event that Brim is terminated
with a SIGKILL signal.

If set zqd with listen to the file descriptor and gracefully should
it be closed.

PART OF #1018
mattnibs added a commit that referenced this issue Aug 28, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
mattnibs added a commit that referenced this issue Aug 28, 2020
Prevent orphaned zqd when Brim exits with SIGKILL.

If platform is not windows, create posix pipe and pass read fd
to zqd via the -brimfd argument. Should brim be closed with SIGKILL
the fd will be closed and zqd, seeing the fd has closed will exit.

Closes #1018
@philrz
Copy link
Contributor

philrz commented Sep 7, 2020

Verified on Brim commit 76f506a talking to zqd commit 42103ef.

I circled back first and became more acquainted with the repro as of Brim GA tagged v0.16.0 talking to zqd GA tagged v0.20.0.

On macOS, when the app is launched, here I confirm the zqd was left behind:

$ ps auxww | grep -i brim
phil             43377   0.6  0.0  4268324    716 s000  S+    1:58PM   0:00.00 grep -i brim
phil             43371   0.0  0.7  4982824 121288   ??  S     1:58PM   0:01.88 /Applications/Brim.app/Contents/Frameworks/Brim Helper (Renderer).app/Contents/MacOS/Brim Helper (Renderer) --type=renderer --field-trial-handle=1718379636,2445424761253024992,10173580791034911523,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --app-path=/Applications/Brim.app/Contents/Resources/app --enable-experimental-web-platform-features --node-integration --no-sandbox --no-zygote --background-color=#fff --enable-websql --num-raster-threads=4 --enable-zero-copy --enable-gpu-memory-buffer-compositor-resources --enable-main-frame-before-activation --service-request-channel-token=5199212984589039801 --renderer-client-id=5 --no-v8-untrusted-code-mitigations
phil             43370   0.0  0.1  4598660  20000   ??  S     1:58PM   0:00.07 /Applications/Brim.app/Contents/Frameworks/Brim Helper.app/Contents/MacOS/Brim Helper --type=utility --field-trial-handle=1718379636,2445424761253024992,10173580791034911523,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --service-sandbox-type=network --service-request-channel-token=1302575996252582287
phil             43369   0.0  0.1  5016680  18224   ??  S     1:58PM   0:00.02 /Applications/Brim.app/Contents/Resources/app/zdeps/zqd listen -l localhost:9867 -data /Users/phil/Library/Application Support/Brim/data/spaces -config /Users/phil/Library/Application Support/Brim/zqd-config.yaml -zeekrunner /Applications/Brim.app/Contents/Resources/app/zdeps/zeek/zeekrunner
phil             43367   0.0  0.3  4697740  49972   ??  S     1:58PM   0:00.67 /Applications/Brim.app/Contents/Frameworks/Brim Helper (GPU).app/Contents/MacOS/Brim Helper (GPU) --type=gpu-process --field-trial-handle=1718379636,2445424761253024992,10173580791034911523,131072 --disable-features=SpareRendererForSitePerProcess --gpu-preferences=KAAAAAAAAAAgAAAAAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAOgAAAAcAAAA4AAAAAAAAADoAAAAAAAAAPAAAAAAAAAA+AAAAAAAAAAAAQAAAAAAAAgBAAAAAAAAEAEAAAAAAAAYAQAAAAAAACABAAAAAAAAKAEAAAAAAAAwAQAAAAAAADgBAAAAAAAAQAEAAAAAAABIAQAAAAAAAFABAAAAAAAAWAEAAAAAAABgAQAAAAAAAGgBAAAAAAAAcAEAAAAAAAB4AQAAAAAAAIABAAAAAAAAiAEAAAAAAACQAQAAAAAAAJgBAAAAAAAAoAEAAAAAAACoAQAAAAAAALABAAAAAAAAuAEAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAABgAAABAAAAAAAAAAAAAAAAcAAAAQAAAAAAAAAAAAAAAIAAAAEAAAAAAAAAAAAAAACgAAABAAAAAAAAAAAAAAAAsAAAAQAAAAAAAAAAAAAAANAAAAEAAAAAAAAAABAAAAAAAAABAAAAAAAAAAAQAAAAYAAAAQAAAAAAAAAAEAAAAHAAAAEAAAAAAAAAABAAAACAAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAAEAAAAAAAAAABAAAADQAAABAAAAAAAAAABAAAAAAAAAAQAAAAAAAAAAQAAAAGAAAAEAAAAAAAAAAEAAAABwAAABAAAAAAAAAABAAAAAgAAAAQAAAAAAAAAAQAAAAKAAAAEAAAAAAAAAAEAAAACwAAABAAAAAAAAAABAAAAA0AAAAQAAAAAAAAAAYAAAAAAAAAEAAAAAAAAAAGAAAABgAAABAAAAAAAAAABgAAAAcAAAAQAAAAAAAAAAYAAAAIAAAAEAAAAAAAAAAGAAAACgAAABAAAAAAAAAABgAAAAsAAAAQAAAAAAAAAAYAAAANAAAA --service-request-channel-token=8403775284667094587
phil             43366   0.0  0.5  5492940  82524   ??  S     1:58PM   0:01.31 /Applications/Brim.app/Contents/MacOS/Brim

$ kill -9 43366

$ ps auxww | grep -i brim
phil             43379   0.0  0.0  4268324    716 s000  S+    1:59PM   0:00.00 grep -i brim
phil             43369   0.0  0.1  5016680  18228   ??  S     1:58PM   0:00.02 /Applications/Brim.app/Contents/Resources/app/zdeps/zqd listen -l localhost:9867 -data /Users/phil/Library/Application Support/Brim/data/spaces -config /Users/phil/Library/Application Support/Brim/zqd-config.yaml -zeekrunner /Applications/Brim.app/Contents/Resources/app/zdeps/zeek/zeekrunner

The same is true on Linux (Ubuntu 18.04 LTS):

$ ps auxww | grep -i brim
phil      1794  0.0  0.0  14424  1124 pts/0    S+   14:00   0:00 grep --color=auto -i brim
phil@phil-ubuntu:~$ ps auxww | grep -i brim
phil      1804  5.1  3.4 856852 140688 tty2    Sl+  14:00   0:00 /usr/lib/brim/Brim
phil      1808  0.1  0.9 370660 39280 tty2     S+   14:00   0:00 /usr/lib/brim/Brim --type=zygote
phil      1810  0.0  0.2 370660  8736 tty2     S+   14:00   0:00 /usr/lib/brim/Brim --type=zygote
phil      1837  3.3  2.4 645760 98160 tty2     Sl+  14:00   0:00 /usr/lib/brim/Brim --type=gpu-process --field-trial-handle=5992394519748964762,5694922008230354064,131072 --disable-features=SpareRendererForSitePerProcess --gpu-preferences=KAAAAAAAAAAgAAAgAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=5048164259241637999
phil      1841  0.3  0.6 1256588 26088 tty2    SLl+ 14:00   0:00 /usr/lib/brim/resources/app/zdeps/zqd listen -l localhost:9867 -data /home/phil/.config/Brim/data/spaces -config /home/phil/.config/Brim/zqd-config.yaml -zeekrunner /usr/lib/brim/resources/app/zdeps/zeek/zeekrunner
phil      1855  0.2  1.2 421008 50592 tty2     Sl+  14:00   0:00 /usr/lib/brim/Brim --type=utility --field-trial-handle=5992394519748964762,5694922008230354064,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --service-sandbox-type=network --service-request-channel-token=15536150142862248315 --shared-files=v8_context_snapshot_data:100,v8_natives_data:101
phil      1862  9.3  3.3 704796 133316 tty2    Sl+  14:00   0:01 /usr/lib/brim/Brim --type=renderer --field-trial-handle=5992394519748964762,5694922008230354064,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --app-path=/usr/lib/brim/resources/app --enable-experimental-web-platform-features --node-integration --no-sandbox --no-zygote --background-color=#fff --enable-websql --num-raster-threads=2 --enable-main-frame-before-activation --service-request-channel-token=16574448475296558375 --renderer-client-id=5 --no-v8-untrusted-code-mitigations --shared-files=v8_context_snapshot_data:100,v8_natives_data:101
phil      1887  0.0  0.0  14424  1040 pts/0    R+   14:00   0:00 grep --color=auto -i brim

$ kill -9 1804

$ ps auxww | grep -i brim
phil      1808  0.0  0.9 370660 39280 tty2     S+   14:00   0:00 /usr/lib/brim/Brim --type=zygote
phil      1810  0.0  0.2 370660  8736 tty2     S+   14:00   0:00 /usr/lib/brim/Brim --type=zygote
phil      1837  1.9  2.4 641664 98160 tty2     Sl+  14:00   0:00 /usr/lib/brim/Brim --type=gpu-process --field-trial-handle=5992394519748964762,5694922008230354064,131072 --disable-features=SpareRendererForSitePerProcess --gpu-preferences=KAAAAAAAAAAgAAAgAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=5048164259241637999
phil      1841  0.1  0.6 1256588 26088 tty2    SLl+ 14:00   0:00 /usr/lib/brim/resources/app/zdeps/zqd listen -l localhost:9867 -data /home/phil/.config/Brim/data/spaces -config /home/phil/.config/Brim/zqd-config.yaml -zeekrunner /usr/lib/brim/resources/app/zdeps/zeek/zeekrunner
phil      1894  0.0  0.0  14424  1056 pts/0    S+   14:01   0:00 grep --color=auto -i brim

Notice there's other "droppings" seemingly left behind here other than just the zqd process.

On Windows, the kill equivalent I'm aware of is hitting "End Task" in the Task Manager. When Brim is running, it looks like this:

Task Manager

What I observed is that if I shot any of those top three tasks labeled "Brim" it behaved effectively like a no-op, i.e. the Brim window either appeared undisturbed or refreshed briefly as if a process went away and was immediately replaced. When I shot the bottom Brim task, the whole set of tasks including the zqd.exe one disappeared.

Now on macOS with Brim commit 76f506a talking to zqd commit 42103ef, we see the zqd process taken out as we'd hope.

$ ps auxww | grep -i brim
phil             43750   0.0  0.0  4277540    740 s001  S+    2:16PM   0:00.00 grep -i brim
phil             43731   0.0  0.8  4900208 129640   ??  S     2:15PM   0:04.22 /Applications/Brim.app/Contents/Frameworks/Brim Helper (Renderer).app/Contents/MacOS/Brim Helper (Renderer) --type=renderer --field-trial-handle=1718379636,13697804025987751387,18118803573008897309,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --app-path=/Applications/Brim.app/Contents/Resources/app --enable-experimental-web-platform-features --node-integration --no-sandbox --no-zygote --background-color=#fff --enable-websql --num-raster-threads=4 --enable-zero-copy --enable-gpu-memory-buffer-compositor-resources --enable-main-frame-before-activation --service-request-channel-token=7420086962641409962 --renderer-client-id=5 --no-v8-untrusted-code-mitigations
phil             43730   0.0  0.1  4605240  20108   ??  S     2:15PM   0:00.13 /Applications/Brim.app/Contents/Frameworks/Brim Helper.app/Contents/MacOS/Brim Helper --type=utility --field-trial-handle=1718379636,13697804025987751387,18118803573008897309,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --service-sandbox-type=network --service-request-channel-token=975529729247464869
phil             43729   0.0  0.1  5060680  19652   ??  S     2:15PM   0:00.06 /Applications/Brim.app/Contents/Resources/app/zdeps/zqd listen -l localhost:9867 -data /Users/phil/Library/Application Support/Brim/data/spaces -config /Users/phil/Library/Application Support/Brim/zqd-config.yaml -zeekrunner /Applications/Brim.app/Contents/Resources/app/zdeps/zeek/zeekrunner -brimfd=3
phil             43727   0.0  0.3  4700364  51816   ??  S     2:15PM   0:01.57 /Applications/Brim.app/Contents/Frameworks/Brim Helper (GPU).app/Contents/MacOS/Brim Helper (GPU) --type=gpu-process --field-trial-handle=1718379636,13697804025987751387,18118803573008897309,131072 --disable-features=SpareRendererForSitePerProcess --gpu-preferences=KAAAAAAAAAAgAAAAAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAOgAAAAcAAAA4AAAAAAAAADoAAAAAAAAAPAAAAAAAAAA+AAAAAAAAAAAAQAAAAAAAAgBAAAAAAAAEAEAAAAAAAAYAQAAAAAAACABAAAAAAAAKAEAAAAAAAAwAQAAAAAAADgBAAAAAAAAQAEAAAAAAABIAQAAAAAAAFABAAAAAAAAWAEAAAAAAABgAQAAAAAAAGgBAAAAAAAAcAEAAAAAAAB4AQAAAAAAAIABAAAAAAAAiAEAAAAAAACQAQAAAAAAAJgBAAAAAAAAoAEAAAAAAACoAQAAAAAAALABAAAAAAAAuAEAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAABgAAABAAAAAAAAAAAAAAAAcAAAAQAAAAAAAAAAAAAAAIAAAAEAAAAAAAAAAAAAAACgAAABAAAAAAAAAAAAAAAAsAAAAQAAAAAAAAAAAAAAANAAAAEAAAAAAAAAABAAAAAAAAABAAAAAAAAAAAQAAAAYAAAAQAAAAAAAAAAEAAAAHAAAAEAAAAAAAAAABAAAACAAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAAEAAAAAAAAAABAAAADQAAABAAAAAAAAAABAAAAAAAAAAQAAAAAAAAAAQAAAAGAAAAEAAAAAAAAAAEAAAABwAAABAAAAAAAAAABAAAAAgAAAAQAAAAAAAAAAQAAAAKAAAAEAAAAAAAAAAEAAAACwAAABAAAAAAAAAABAAAAA0AAAAQAAAAAAAAAAYAAAAAAAAAEAAAAAAAAAAGAAAABgAAABAAAAAAAAAABgAAAAcAAAAQAAAAAAAAAAYAAAAIAAAAEAAAAAAAAAAGAAAACgAAABAAAAAAAAAABgAAAAsAAAAQAAAAAAAAAAYAAAANAAAA --service-request-channel-token=7431305296922332173
phil             43724   0.0  0.5  5467184  86116   ??  S     2:15PM   0:02.55 /Applications/Brim.app/Contents/MacOS/Brim

$ kill -9 43724

$ ps auxww | grep -i brim
phil             43754   0.0  0.0  4268324    716 s001  S+    2:16PM   0:00.00 grep -i brim

On Linux with Brim commit 76f506a talking to zqd commit 42103ef, we're glad to see that killing Brim not only gets rid of zqd but also the "droppings" we saw earlier.

$ ps auxww | grep -i brim
phil      1794 35.0  4.2 1415428 171376 ?      SLl  14:23   0:22 gnome-software --local-filename=/home/phil/brim/dist/installers/brim_amd64.deb
phil      2026  6.5  3.3 835212 136364 tty2    Sl+  14:24   0:00 /usr/lib/brim/Brim
phil      2030  0.0  0.9 370660 39860 tty2     S+   14:24   0:00 /usr/lib/brim/Brim --type=zygote
phil      2032  0.0  0.2 370660  8824 tty2     S+   14:24   0:00 /usr/lib/brim/Brim --type=zygote
phil      2061  3.7  2.4 643972 98740 tty2     Sl+  14:24   0:00 /usr/lib/brim/Brim --type=gpu-process --field-trial-handle=11237330488576187713,9353552982362025423,131072 --disable-features=SpareRendererForSitePerProcess --gpu-preferences=KAAAAAAAAAAgAAAgAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=7011466320088090641
phil      2063  0.0  0.6 1329732 26392 tty2    Sl+  14:24   0:00 /usr/lib/brim/resources/app/zdeps/zqd listen -l localhost:9867 -data /home/phil/.config/Brim/data/spaces -config /home/phil/.config/Brim/zqd-config.yaml -zeekrunner /usr/lib/brim/resources/app/zdeps/zeek/zeekrunner -brimfd=3
phil      2077  0.2  1.2 429204 51188 tty2     Sl+  14:24   0:00 /usr/lib/brim/Brim --type=utility --field-trial-handle=11237330488576187713,9353552982362025423,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --service-sandbox-type=network --service-request-channel-token=8107782965694897881 --shared-files=v8_context_snapshot_data:100,v8_natives_data:101
phil      2082 10.3  3.9 707936 160160 tty2    Sl+  14:24   0:01 /usr/lib/brim/Brim --type=renderer --field-trial-handle=11237330488576187713,9353552982362025423,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --app-path=/usr/lib/brim/resources/app --enable-experimental-web-platform-features --node-integration --no-sandbox --no-zygote --background-color=#fff --enable-websql --num-raster-threads=2 --enable-main-frame-before-activation --service-request-channel-token=9585659228422678141 --renderer-client-id=5 --no-v8-untrusted-code-mitigations --shared-files=v8_context_snapshot_data:100,v8_natives_data:101
phil      2126  0.0  0.0  14424  1088 pts/0    R+   14:24   0:00 grep --color=auto -i brim

$ kill -9 2026

$ ps auxww | grep -i brim
phil      1794 27.7  4.2 1407232 171360 ?      SLl  14:23   0:22 gnome-software --local-filename=/home/phil/brim/dist/installers/brim_amd64.deb
phil      2128  0.0  0.0  14424  1052 pts/0    S+   14:24   0:00 grep --color=auto -i brim

Thanks @mattnibs!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants