Add short-term analytical logging

[raxod502]

Riju has been brought down temporarily a few times over the last week or two, which seems most likely to be due to people running pathological code. To understand and address the failure conditions, we need some visibility into what operations and usage patterns exactly are causing resource exhaustion. As such, we should add some analytical logging that will record what API operations are being called by which clients at which times, with a retention of perhaps a few days. This data will allow me to investigate the root cause of abnormal behavior that shows up in monitoring, e.g.:

[raxod502]

Excerpt from logs for above incident

Jul 26 07:35:23 [de0e9f24842d4ccfb7156b1cf794d141] Creating session, language python
Jul 26 07:35:24 [dc8cceb4b653408f81ee85e2a5c0c22c] Tearing down session
Jul 26 07:35:25 [8bb60347d08b4575b4a2f94c139ec4e8] Creating session, language python
Jul 26 07:35:36 [f82cd1502790405cb26f145e6d1d3cc3] Tearing down session
Jul 26 07:35:38 [34cc278ac06e48288fc0f95804efacfb] Creating session, language csharp
Jul 26 07:35:51 [8d68dea8021543ee925deecdad929c01] Tearing down session
Jul 26 07:35:52 [f166086aca8b44e7a6d7b2fbcded9e18] Creating session, language javascript
Jul 26 07:36:07 [51a32612c26943eea3804fb88a433337] Tearing down session
Jul 26 07:36:08 [9dd5e2cc6d434729bf9078a048392a4c] Creating session, language ook
Jul 26 07:36:08 [4a6f5129ba664ffd8273c74f49356889] Tearing down session
Jul 26 07:36:09 [c8862dd558ac4ab1bdda64a703a6bf4f] Creating session, language mysql
Jul 26 07:36:50 [80a8ded92c024539a5969378e0874211] Tearing down session
Jul 26 07:36:51 [791fa137edc8416fa1eda1e83056abae] Creating session, language python
Jul 26 07:37:04 [de0e9f24842d4ccfb7156b1cf794d141] Tearing down session
Jul 26 07:37:05 [10e29d57ab264242b56810a7b9db4b99] Creating session, language python
Jul 26 07:37:07 [8bb60347d08b4575b4a2f94c139ec4e8] Tearing down session
Jul 26 07:37:08 [b4f03cd28c4842278488f06160d9609c] Creating session, language python
Jul 26 07:37:20 [34cc278ac06e48288fc0f95804efacfb] Tearing down session
Jul 26 07:37:22 [b3bb84d5b2de46e18104bab22b4c0716] Creating session, language csharp
Jul 26 07:37:34 [f166086aca8b44e7a6d7b2fbcded9e18] Tearing down session
Jul 26 07:37:35 [6631b5636f224f26ab6d34f3a8e30045] Creating session, language javascript
Jul 26 07:37:49 [9dd5e2cc6d434729bf9078a048392a4c] Tearing down session
Jul 26 07:37:50 [2f231a146e584a519475d8c00f081037] Creating session, language ook
Jul 26 07:37:59 [c8862dd558ac4ab1bdda64a703a6bf4f] Tearing down session
Jul 26 07:38:00 [1a4632b23c284dbd9ea857e271bd8e33] Creating session, language mysql
Jul 26 07:38:33 [791fa137edc8416fa1eda1e83056abae] Tearing down session
Jul 26 07:38:33 [5ab04d0d862946c495a0bfe953304764] Creating session, language python
Jul 26 07:38:47 [10e29d57ab264242b56810a7b9db4b99] Tearing down session
Jul 26 07:38:48 [9c2d3fac32014976b65a86192c0dce97] Creating session, language python
Jul 26 07:38:49 [b4f03cd28c4842278488f06160d9609c] Tearing down session
Jul 26 07:38:50 [dd0f11f9e33e49f4903227ec27e54c08] Creating session, language python
Jul 26 07:39:03 [b3bb84d5b2de46e18104bab22b4c0716] Tearing down session
Jul 26 07:39:08 [0121b193cda54f6db7ea9a61150d303f] Creating session, language csharp
Jul 26 07:39:17 [6631b5636f224f26ab6d34f3a8e30045] Tearing down session
Jul 26 07:39:18 [162e6bd7de954e4ab7e9cc8a3dcc1cb2] Creating session, language javascript
Jul 26 07:39:32 [2f231a146e584a519475d8c00f081037] Tearing down session
Jul 26 07:39:33 [349da532cd0b4375b9fd0da4d4c2d04b] Creating session, language ook
Jul 26 07:39:35 [7ee0f249ef684b27a4d4615b0de4ffcd] Creating session, language r
Jul 26 07:39:50 [1a4632b23c284dbd9ea857e271bd8e33] Tearing down session
Jul 26 07:39:51 [7ee0f249ef684b27a4d4615b0de4ffcd] Tearing down session
Jul 26 07:39:51 [c2123ae3a5e74a64ae6d4e9b1f26e758] Creating session, language mysql
Jul 26 07:40:06 [8c4b29613135466aac654127da0b429a] Creating session, language fishlang
Jul 26 07:40:08 [8c4b29613135466aac654127da0b429a] Tearing down session
Jul 26 07:40:15 [5ab04d0d862946c495a0bfe953304764] Tearing down session
Jul 26 07:40:16 [5e708cfa98e2468dbea2d419869a3f6d] Creating session, language python
Jul 26 07:40:19 [9864dc3c7fdd4f4388fb4b8d4252d33a] Creating session, language julia
Jul 26 07:40:22 [9864dc3c7fdd4f4388fb4b8d4252d33a] Tearing down session
Jul 26 07:47:06 [dd0f11f9e33e49f4903227ec27e54c08] Tearing down session
Jul 26 07:47:08 [9c2d3fac32014976b65a86192c0dce97] Tearing down session
Jul 26 07:47:08 [c2123ae3a5e74a64ae6d4e9b1f26e758] Tearing down session
Jul 26 07:47:08 [349da532cd0b4375b9fd0da4d4c2d04b] Tearing down session
Jul 26 07:47:08 [162e6bd7de954e4ab7e9cc8a3dcc1cb2] Tearing down session
Jul 26 07:47:08 [0121b193cda54f6db7ea9a61150d303f] Tearing down session
Jul 26 07:47:08 [5e708cfa98e2468dbea2d419869a3f6d] Tearing down session
Jul 26 07:47:08 [8cc2820b185d499ebb86e5e1f2e5fba4] Creating session, language cpp
Jul 26 07:47:09 [181fa1af7bde4df6915b2f3ad995180a] Creating session, language csharp
Jul 26 07:47:09 [e9f17bb84e06400f97c15378011883df] Creating session, language csharp
Jul 26 07:47:09 [4b765b928a1045d19df3d83981b7651e] Creating session, language javascript
Jul 26 07:47:09 [19ab53bb70954be9be950bdfcada2d28] Creating session, language ook
Jul 26 07:47:10 [d705dec67874457f9cb6a26f001ba0fa] Creating session, language cpp
Jul 26 07:47:10 [dcb4f85404a44a02b23e0c740f3edf48] Creating session, language csharp
Jul 26 07:47:11 [8c5fa29edcdc4c44b0a6319395ef9bcf] Creating session, language cpp
Jul 26 07:47:11 [51912a0733ee4249875436a1bdffb901] Creating session, language csharp
Jul 26 07:47:11 [c66a590ab6e843cf8c6de8211ee14ba6] Creating session, language python
Jul 26 07:47:12 [04f0b4054732481ab4b23f23820c4c83] Creating session, language cpp
Jul 26 07:47:12 [83b18df460004e1293e95a494376f022] Creating session, language python
Jul 26 07:47:12 [776c84ee03614ca98d5158b1c00f38a2] Creating session, language python
Jul 26 07:47:12 [48baea90c54641268b57c0e40ea62514] Creating session, language csharp
Jul 26 07:47:13 [66174e780315464cbabebe48fbaf4e10] Creating session, language python
Jul 26 07:47:13 [b2c630e9f6604dfc89c9cfb054ee1339] Creating session, language csharp
Jul 26 07:47:13 [59c67925508a407e80224a906be2fa00] Creating session, language cpp
Jul 26 07:47:14 [92da8e75fc2048acaf4a34ca2c69d309] Creating session, language python
Jul 26 07:47:14 [9ef2a0d30978493d8fab44a9f99e49fb] Creating session, language python
Jul 26 07:47:15 [4f72d5e223a5494d9b2142f6d8e10406] Creating session, language javascript
Jul 26 07:47:15 [a70b1c93ab384dde81c0bf91afe81030] Creating session, language ook
Jul 26 07:47:15 [2c21f7bd986640e08788b7682d09af60] Creating session, language cpp
Jul 26 07:47:16 [daeaf234ca3d48109ca70ad61fb4c337] Creating session, language python
Jul 26 07:47:16 [ea152132c537498097b13e999406e1a4] Creating session, language python
Jul 26 07:47:17 [eca1b783f63147a6a284d53fb6da1367] Creating session, language csharp
Jul 26 07:47:17 [5fc89cb43ff64742ab6c88745db31bba] Creating session, language javascript
Jul 26 07:47:17 [bfea8bedd1e9495d85945878c253b8e8] Creating session, language cpp
Jul 26 07:47:18 [41c8100cdd5b4d5aad1a93039d0820c5] Creating session, language ook
Jul 26 07:47:22 make: *** [Makefile:172: server] Killed
*** Not all processes have exited in time. Forcing them to exit.
Jul 26 07:47:58 node backend/server.js
Jul 26 07:47:59 Listening on http://0.0.0.0:6119
Jul 26 07:48:00 Loaded 224 language configuration(s) from disk
Jul 26 07:48:00 [88431ce0641d4c16800bdcf18581c65b] Creating session, language python
Jul 26 07:48:00 [025c3ddb95db4eacaba6cd1de3511fe2] Creating session, language python
Jul 26 07:48:01 [bef1da3bf5354f7f9d83fc1066204115] Creating session, language javascript
Jul 26 07:48:07 [53330fd07a1242ef821f69d3b3c70de4] Creating session, language cpp
Jul 26 07:48:07 [31f2baf9df11484d801d96000241eb5c] Creating session, language ook
Jul 26 07:48:13 [78ed11eb88b14dffb4b9ea293566a414] Creating session, language csharp
Jul 26 07:49:41 [88431ce0641d4c16800bdcf18581c65b] Tearing down session
Jul 26 07:49:41 [025c3ddb95db4eacaba6cd1de3511fe2] Tearing down session
Jul 26 07:49:42 [975b1a1378b349959fe0b64a35d54504] Creating session, language python
Jul 26 07:49:42 [ae0fef9c49724cb8bf4fef2a003adf71] Creating session, language python
Jul 26 07:49:43 [bef1da3bf5354f7f9d83fc1066204115] Tearing down session
Jul 26 07:49:44 [3b2e525de4054855ac09987139f4ac65] Creating session, language javascript
Jul 26 07:49:50 [31f2baf9df11484d801d96000241eb5c] Tearing down session
Jul 26 07:49:50 [53330fd07a1242ef821f69d3b3c70de4] Tearing down session
Jul 26 07:49:51 [967beec6131543c4b726b07a6a5e27ee] Creating session, language cpp
Jul 26 07:49:51 [3ef40ead1bea4c18bac9f25cb5898d9b] Creating session, language ook
Jul 26 07:49:54 [78ed11eb88b14dffb4b9ea293566a414] Tearing down session
Jul 26 07:49:56 [03a69d5ccadf40d994908445a836dfe9] Creating session, language csharp
Jul 26 07:50:27 [df63f2418d0240b084dbf0eddee301c6] Creating session, language python
Jul 26 07:50:34 [df63f2418d0240b084dbf0eddee301c6] Tearing down session
Jul 26 07:51:23 [975b1a1378b349959fe0b64a35d54504] Tearing down session
Jul 26 07:51:23 [ae0fef9c49724cb8bf4fef2a003adf71] Tearing down session
Jul 26 07:51:25 [3b2e525de4054855ac09987139f4ac65] Tearing down session
Jul 26 07:51:25 [798103f80de040858357d8acd6866e2d] Creating session, language python
Jul 26 07:51:25 [1b0ea04a199c43b09d942ec7aef48ef8] Creating session, language python
Jul 26 07:51:25 [c66cb8c79e61445ba6590b4e997edf77] Creating session, language javascript
Jul 26 07:51:30 [23443f71470f41d297f3bde43845ec1b] Creating session, language typescript
Jul 26 07:51:33 [3ef40ead1bea4c18bac9f25cb5898d9b] Tearing down session
Jul 26 07:51:33 [967beec6131543c4b726b07a6a5e27ee] Tearing down session
Jul 26 07:51:33 [68806dee4d2948bb8672a256d187d328] Creating session, language ook
Jul 26 07:51:35 [282b64380448491788d5111c53376653] Creating session, language cpp
Jul 26 07:51:37 [03a69d5ccadf40d994908445a836dfe9] Tearing down session
Jul 26 07:51:39 [b18ffe6e9638442eaa27ffe2d6995e85] Creating session, language csharp
Jul 26 07:51:40 [23443f71470f41d297f3bde43845ec1b] Tearing down session
Jul 26 07:51:44 [d06f88954fdb45de9b741d5104e11493] Creating session, language aheui
Jul 26 07:51:51 [d06f88954fdb45de9b741d5104e11493] Tearing down session
Jul 26 07:53:07 [798103f80de040858357d8acd6866e2d] Tearing down session
Jul 26 07:53:07 [1b0ea04a199c43b09d942ec7aef48ef8] Tearing down session
Jul 26 07:53:07 [c66cb8c79e61445ba6590b4e997edf77] Tearing down session
Jul 26 07:53:08 [85f343b0f2564947bdee0ffdde269539] Creating session, language python
Jul 26 07:53:08 [3f21dfb8390644348e4d531ce28431db] Creating session, language python
Jul 26 07:53:08 [c7d1cadc753243678a55d3a73a82a89d] Creating session, language javascript
Jul 26 07:53:15 [68806dee4d2948bb8672a256d187d328] Tearing down session
Jul 26 07:53:15 [3b5d1c3ff5cb4706ae7de8b1d2a1a29d] Creating session, language ook
Jul 26 07:53:16 [282b64380448491788d5111c53376653] Tearing down session
Jul 26 07:53:18 [316f72eecdd24ec582a1c6e3e195e4f0] Creating session, language cpp
Jul 26 07:53:20 [b18ffe6e9638442eaa27ffe2d6995e85] Tearing down session
Jul 26 07:53:22 [4878e4b80e7c49e7927b6c3bf27dbbd5] Creating session, language csharp
Jul 26 07:54:50 [85f343b0f2564947bdee0ffdde269539] Tearing down session
Jul 26 07:54:50 [c7d1cadc753243678a55d3a73a82a89d] Tearing down session
Jul 26 07:54:50 [3f21dfb8390644348e4d531ce28431db] Tearing down session
Jul 26 07:54:51 [422b4ad4c89b4d1daea65d473365dc26] Creating session, language python
Jul 26 07:54:51 [4b218f7860794f72b4048bfb0095048f] Creating session, language python
Jul 26 07:54:51 [391ca90fc62040df99511e29cf348596] Creating session, language javascript

Impact was about 3 minutes of downtime according to monitoring.