async_hooks: add currentResource #21313

mcollina · 2018-06-13T13:50:09Z

Remove the need for the destroy hook in the basic APM case.
APM and continuation local storage users can use only the init hook.

This lead to a 5% improvement in throughput. in the attached benchmark:

async_hooks/current-resource-vs-destroy.js n=1000000 method="callbacks" type="current-resource" benchmarker="wrk": 7,428.04
async_hooks/current-resource-vs-destroy.js n=1000000 method="async" type="current-resource" benchmarker="wrk": 5,938.27
async_hooks/current-resource-vs-destroy.js n=1000000 method="callbacks" type="destroy" benchmarker="wrk": 7,142.18
async_hooks/current-resource-vs-destroy.js n=1000000 method="async" type="destroy" benchmarker="wrk": 5,297.4

I've also measured this against a simple Hapi v17 server and it leads to a 15% improvement.

cc @nodejs/diagnostics @bmeurer @MayaLekova

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2018-06-13T13:50:10Z

@mcollina build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/51/pipeline

Remove the need for the destroy hook in the basic APM case.

mcollina · 2018-06-13T13:51:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15436/

addaleax

I think what you want is a per-Environment stack (possibly implemented as a JS Array) where you can push/pop the resources and their async IDs?

In particular, it seems like this would replace the async ID stack in that case, and follow its current semantics wrt unhandled exceptions

addaleax · 2018-06-13T14:11:41Z

lib/internal/async_hooks.js

@@ -357,6 +360,8 @@ function emitAfterScript(asyncId) {
  if (async_hook_fields[kAfter] > 0)
    emitAfterNative(asyncId);

+  setCurrentResource(null);


If we have nested emitBefore(…, res1) – emitBefore(…, res2) – emitAfter(…) – emitAfter(…), then after the first emitAfter we want the current resource to be res1

yeah, shouldn't setCurrentResource be pushCurrentResource/popCurrentResource?

If we have nested emitBefore(…, res1) – emitBefore(…, res2) – emitAfter(…) – emitAfter(…), then after the first emitAfter we want the current resource to be res1

I thought the same. However I struggled to execute user code after emitAfter. Can you add an example?

I think this should work:

const async_hooks = require("async_hooks"); const fs = require("fs"); let indent = 0; function init(asyncId, type, triggerAsyncId, resource) { const eid = async_hooks.executionAsyncId(); const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}${type}(${asyncId}): trigger: ${triggerAsyncId} execution: ${eid}\n`); } function before(asyncId) { const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}before: ${asyncId}\n`); indent += 2; } function after(asyncId) { indent -= 2; const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}after: ${asyncId}\n`); } function destroy(asyncId) { const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}destroy: ${asyncId}\n`); } async_hooks.createHook({ init, before, after, destroy }).enable(); class MyResource extends async_hooks.AsyncResource { constructor() { super("XXX"); } } const res = new MyResource(); setImmediate(() => { const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}between before\n`); res.runInAsyncScope(() => { const indentStr = ' '.repeat(indent); fs.writeSync(1, `${indentStr}inner \n`); }); fs.writeSync(1, `${indentStr}between after\n`); });

Is this only happening with runInAsyncScope? That would be easy to fix without adding a full stack where we push/pop.

Adding a full stack is also a serious refactoring of the async_hooks machinery, something that this PR currently avoids.

@mcollina We do have a full stack in place currently, though?

Is this only happening with runInAsyncScope?

It can happen with all mechanisms that trigger async hooks.

addaleax · 2018-06-13T14:12:30Z

src/async_wrap.cc

+void AsyncWrap::EmitBefore(Environment* env, double async_id,
+    v8::Local<v8::Object> resource) {
+  v8::Local<v8::Context> context = env->isolate()->GetCurrentContext();
+  context->SetEmbedderData(CURRENT_RESOURCE_FIELD, resource);


async_hooks are currently implemented as an per-Environment, not a per-Isolate or per-Context thing.

That's one thing that we want to change (soonish). cc @hashseed

I’d assume we want it to be a per-Isolate thing then, not per-Context like this currently is?

In any case, I’d prefer not to have this PR change it only halfway. If we do switch towards per-Isolate state (which I agree is a good idea), we should move all of async_hooks over to that

No, we'll switch towards per-Context for the PromiseHook as well. The fact that this is Isolate-wide was a design mistake on our end.

@mcollina Using embedder fields doesn’t seem simpler than storing it on the Environment … but I guess that depends on Benedikt’s reasoning here

As @bmeurer already mentioned, we eventually want to reach a state where we have the hooks per context and not per isolate. Storing on the isolate also opens up potential optimization in the future, e.g. if we expose a JS function that can access embedder slots.

Storing on the context opens up optimization potential 😁

erm. yeah. :)

That seems like an odd choice on V8’s side, since the µtask queue is also per-Isolate … unless you also want to migrate that?

But even so, I think for Node.js per-Environment would still make more sense?

addaleax · 2018-06-13T14:16:39Z

src/async_wrap.cc

  // If the user's callback failed then the after() hooks will be called at the
  // end of _fatalException().
  Emit(env, async_id, AsyncHooks::kAfter,
       env->async_hooks_after_function());
+
+  context->SetEmbedderData(CURRENT_RESOURCE_FIELD, v8::Null(isolate));


If there was an uncaught exception that was handled by an uncaughtException handles, this isn’t reached. For the async ID stack, we currently handle that case by using clear_async_id_stack() in the C++ code/clearAsyncIdStack() on the JS side

Can you make an example?

@mcollina I think this only becomes observable after we clean up the push/pop thing, I’ll do it then

bmeurer

Generally LGTM. Not so happy with the inconsistent naming.

bmeurer · 2018-06-13T14:13:35Z

benchmark/async_hooks/current-resource-vs-destroy.js

+const common = require('../common.js');
+const {
+  createHook,
+  currentResource,


Shouldn't this be named executionAsyncResource for consistency?

bmeurer · 2018-06-13T14:24:36Z

lib/async_hooks.js

@@ -25,6 +25,7 @@ const {
  emitBefore,
  emitAfter,
  emitDestroy,
+  currentResource


As mentioned above, how about using executionAsyncResource?

apapirovski

Looks great! A few nits (in addition to the other feedback by others).

apapirovski · 2018-06-13T14:26:54Z

benchmark/async_hooks/current-resource-vs-destroy.js

+    .listen(common.PORT)
+    .on('listening', function() {
+
+      bench.http({


I think I would prefer a benchmark that doesn't use http. It'll make it hard to notice incremental improvements to this code because the spread and overhead of http is so high.

apapirovski · 2018-06-13T14:30:50Z

src/async_wrap.cc

@@ -27,6 +27,10 @@
 #include "v8.h"
 #include "v8-profiler.h"

+// TODO(mcollina): should this be moved in a more public space?
+// embedders will not be able to use this field


A general nit for comments: start with a capital and end with ..

Flarna

I think the calls to emitBefore() and emitAfter() in class AsyncResource needs to be also adapted to pass the current resource.

Flarna · 2018-06-13T15:58:09Z

It's not directly related to the changes of this PR but I think it should be noted.

To my understanding the current implementation of AsyncResource in NAN is not compatible with this change as the API there doesn't hardwire the resource used in init() with the target used in runInAsyncScope().

mike-kaufman · 2018-06-13T15:59:32Z

benchmark/async_hooks/current-resource-vs-destroy.js

@@ -0,0 +1,165 @@
+'use strict';


IMO some comments explaining what this benchmark is testing will help readers going forward.

mike-kaufman · 2018-06-13T16:06:10Z

benchmark/async_hooks/current-resource-vs-destroy.js

+
+    var cr = currentResource();
+    if (cr) {
+      resource[cls] = cr[cls];


I don't think this necessarily pertinent to the intent of the benchmark here, but I'll call it out nonetheless: If I'm reading correctly, this results in every resource sharing the same object instance for cls, and every setCLS call will just overwrite the state field. For correctness here, each resource needs a copy (e.g., resource[cls] = {...cr[cls]}), or you need to maintain a "list to the root" (e.g., resource[cls] = {props:{}, parent: cr[cls]}.

Per conversation, here's an example of where values would be over-written:

function f0() { setCLS("a"); setTimeout(function f1() { setCLS("b"); setTimeout(function f2() { getCLS(); // expect "b", will get "c" }, 100); }, 10); setTimeout(function f3() => { getCLS(); // expect "a", will get "b" setCLS("c"); setTimeout(function f4() { getCLS(); // get "c" }, 25); }, 25); }; f();

Order of execution is f0, f1, f3, f4, f2.

But f0->f1->f2 is one path of the tree, and another indepdent path is f0->f3->f4. Values visible on one path should not be visible on the path.

I don't think branch containment really matters for this test. We just need to verify that an object can be passed through the tree via attachment to the current resource in some form.

I don't think branch containment really matters for this test.

Agreed, but it came up on the diag call & I offered to post example code. :)

vsemozhetbyt · 2018-06-13T20:29:17Z

doc/api/async_hooks.md

+
+const server = createServer(function(req, res) {
+  currentResource()[sym] = { state: req.url };
+  setTimeout(function() {


function() { -> () => {?

mike-kaufman · 2018-06-13T20:46:00Z

lib/internal/async_hooks.js

  // Validate the ids. An id of -1 means it was never set and is visible on the
  // call graph. An id < -1 should never happen in any circumstance. Throw
  // on user calls because async state should still be recoverable.
  validateAsyncId(asyncId, 'asyncId');
  validateAsyncId(triggerAsyncId, 'triggerAsyncId');

+  setCurrentResource(resource);


assert here is that currentResource === undefined || currentResource === null.

addaleax

Sorry to red-X this, just want to make sure that we do iron out the issues that are being pointed out here (since this is already getting approvals)

AndreasMadsen

Well, I would certainly advise against this, as there are some fundamental issues.

We make no guarantees that the resource object is unique for every async operation. Our documentation even says you shouldn't depend on this:

In some cases the resource object is reused for performance reasons, it is thus not safe to use it as a key in a WeakMap or add properties to it.

CLS should properly follow the triggerAsyncId graph and not the executeAsyncId graph. Following the executeAsyncId is almost guaranteed to make you lose context at some point. Certainly, all the APM implementations I have seen, follow the triggerAsyncId at some point.

There is also the option for following the resolve graph in case of promises, which again this doesn't really consider.

Not a fundamental issue: The resource objects MUST be pushed to a stack in emitBefore for this to work. Doing this could lead to some unexpected garbage collection or at least inefficient escape analysis. I don't really think it is the case, as emitAfter is guaranteed to empty the stack in each tick, but I think it is worth thinking about.

All In all, I think this is a pretty bad idea. The first point makes it logically invalid. The second point makes me doubt if it will be useful.

As a side note, this is how you benchmark this:

./node benchmark/scatter.js benchmark/async_hooks/current-resource-vs-destroy.js | tee scatter.csv
cat scatter.csv | Rscript benchmark/scatter.R --xaxis method --category type --plot scatter-plot.png

    method             type     rate confidence.interval
     async current-resource 5227.464            41.83307
     async          destroy 4934.202            36.62712
 callbacks current-resource 6469.581            59.48585
 callbacks          destroy 6251.789            65.31779

Fishrock123 · 2018-06-20T16:29:56Z

See also my async_hooks.getActiveResources() proposal in #21102 (review), which I think might solve this better?

mcollina · 2018-08-21T11:12:56Z

We make no guarantees that the resource object is unique for every async operation.

I've looked for this type of object reuse in Node.js but I didn't see any with different asyncId. Could you point me to it?
I thought that sentence left some room for a potential optimization (object reuse). This PR changes exactly that (I removed that sentence in the docs).

CLS should properly follow the triggerAsyncId graph and not the executeAsyncId graph.

I both agree and disagree with this sentence. Consider the adapted example from our docs:

const async_hooks = require('async_hooks');
async_hooks.createHook({
  init(asyncId, type, triggerAsyncId) {
    const eid = async_hooks.executionAsyncId();
    const r = async_hooks.executionAsyncResource();
    fs.writeSync(
      1, `${type}(${asyncId}): trigger: ${triggerAsyncId} execution: ${eid} resource: ${r}\n`);
  }
}).enable();

require('net').createServer((conn) => {}).listen(8080);

// $ nc localhost 8080
// 
// Output
// TCPSERVERWRAP(5): trigger: 1 execution: 1 resource: null
// TickObject(6): trigger: 5 execution: 1 resource: null
// TCPWRAP(7): trigger: 5 execution: 0 resource: null
// TickObject(8): trigger: 7 execution: 7 resource: [object TCP]
// TickObject(9): trigger: 8 execution: 8 resource: [object Object]
// TickObject(10): trigger: 8 execution: 8 resource: [object Object]

When the executionAsyncId() is 0, it means that this originated from C++, e.g. we have just received a new connection. IMHO receiving a new connection defines the start of a new transaction for most (if not all) Node.js servers. In this case, executionAsyncResource() returns null when the executionAsyncId() is 0, which would be correct.

While avoiding triggerAsyncId does not provide a complete solution, I think this would be useful for the vast majority of the cls and apm usecases, if we can confirm the performance benefits.

Are there any other cases that I'm missing?

Certainly, all the APM implementations I have seen, follow the triggerAsyncId at some point.

Look at https://github.com/elastic/apm-agent-nodejs/blob/a3131d526639c5434628f28b49f5ef2c083cc4e9/lib/instrumentation/async-hooks.js (cc @watson). It's important to note that https://github.com/GoogleCloudPlatform/cloud-trace-nodejs/blob/ebd33d802e5ab758e4317b547bdc846707f9dfd5/src/cls/async-hooks.ts follows the triggerId (cc @ofrobots). @ofrobots will executionAsyncResource() be enough?

AndreasMadsen · 2018-08-21T12:08:21Z

I've looked for this type of object reuse in Node.js but I didn't see any with different asyncId. Could you point me to it?

I'm pretty sure the HTTPParser is reused. Making the context of that wrong will affect all contexts in a HTTP server and also requests. I don't think it is possible to avoid without making new HTTPParser instances every time. And for sure doing that will be a much greater performance penalty.

jasnell · 2018-08-21T12:12:27Z

I'm pretty sure the HTTPParser is reused

Yep. Reused if there are any available, created if there aren't.

mcollina · 2018-08-21T13:37:23Z

I'm pretty sure the HTTPParser is reused. Making the context of that wrong will affect all contexts in a HTTP server and also requests. I don't think it is possible to avoid without making new HTTPParser instances every time. And for sure doing that will be a much greater performance penalty.

Here it is:

node/src/node_http_parser.cc

Lines 458 to 473 in 1744205

    
           static void Reinitialize(const FunctionCallbackInfo<Value>& args) { 
        
             Environment* env = Environment::GetCurrent(args); 
        
             http_parser_type type = 
        
                 static_cast<http_parser_type>(args[0]->Int32Value()); 
        
             CHECK(type == HTTP_REQUEST || type == HTTP_RESPONSE); 
        
             Parser* parser; 
        
             ASSIGN_OR_RETURN_UNWRAP(&parser, args.Holder()); 
        
             // Should always be called from the same context. 
        
             CHECK_EQ(env, parser->env()); 
        
             // The parser is being reused. Reset the async id and call init() callbacks. 
        
             parser->AsyncReset(); 
        
             parser->Init(type); 
        
           }

. In my examples and tests, this works because we are overriding/reinitializing the state at the beginning of each transaction. However this creates a problem for the GC, because those objects will survive long enough to go into old space.

I think this was an interesting experiment. However, I don't think it's possible to provide a generic enough API for this to be correct and useful. It might be possible to use the concept to implement a crude form of CLS directly in core. This will solve both problems, as we could erase the state after use and pass along the "cls" state together with the triggerId.

AndreasMadsen · 2018-08-21T13:55:24Z

@mcollina There are so many variations of CLS one can implement that it makes my head spin. I think it would be more valuable to look into optimizing the destroy part.

Are there cases where we can emit destroy manually (not via GC), perhaps even directly from JS. For example on .close() calls.
If promises is a particularly big issue, can we add a filter API to async_hooks such it doesn't emit destroy for some resource types, and then make a special optimization case for promises.
We keep telling the V8 team that we would like a dedicated destroy event in PromiseHooks. If we have good data support, maybe they will be more willing to help.

jasnell · 2018-08-21T15:25:56Z

Maybe we could emit destroy immediately after after, and not wait for gc.

that would likely be problematic if the destroy is relying on the assumption that gc has happened and would cause it to act differently than other types of objects. That doesn't mean we shouldn't do this, it just means we'll have to account for the differences.

bmeurer · 2018-08-21T15:44:39Z

@mcollina You mean collect the ones for which the handlers ran during runMicrotasks()? Probably expensive, plus you need to mark them as destroyed so that the GC doesn't trigger this again.

AndreasMadsen · 2018-08-22T07:05:49Z

that would likely be problematic if the destroy is relying on the assumption that gc has happened and would cause it to act differently than other types of objects. That doesn't mean we shouldn't do this, it just means we'll have to account for the differences.

I don't think that is an issue. destroy is not a GC event it is a "no more events" events. However, the current PromiseHooks semantics emits multiple before and after when multiple .then(fn) are called. So we can't simply emit destroy after the after event.

Flarna · 2018-09-20T08:25:15Z

Regarding reuse of resources: it may happen that one more reuse will be added in #22951 as the callback passed by user is used as resource and this is allowed several times.

As discussed in nodejs/diagnostics#248 and nodejs#21313, reusing HTTPParser resource is a blocker for landing `require('async_hooks').currentAsyncResource()`, since reusing an async resource interanlly would make it infeasible to use them reliably in WeakMaps. Two suggestions came up: have a wrapper around the HTTPParser which we would use as the async resource, or stop reusing HTTPParser in our HTTP server/client code. This commit implements the latter, since we have a better GC now and reusing HTTPParser might make sense anymore. This also will avoid one extra JS->C++ call in some cases, which should improve performance for these cases.

Change resource being used, previously HTTParser was being reused. We are now using IncomingMessage and ClientRequest objects. The goal here is to make the async resource unique for each async operatio Refs: nodejs#24330 Refs: nodejs/diagnostics#248 Refs: nodejs#21313 Co-authored-by: Matheus Marchini <mat@mmarchini.me>

Change resource being used, previously HTTParser was being reused. We are now using IncomingMessage and ClientRequest objects. The goal here is to make the async resource unique for each async operatio Refs: #24330 Refs: nodejs/diagnostics#248 Refs: #21313 Co-authored-by: Matheus Marchini <mat@mmarchini.me> PR-URL: #25094 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

As discussed in nodejs/diagnostics#248, nodejs#21313 and https://docs.google.com/document/d/1g8OrG5lMIUhRn1zbkutgY83MiTSMx-0NHDs8Bf-nXxM/preview reusing the resource object is a blocker for landing a resource based async hooks API and get rid of the promise destroy hook. This PR ensures that HttpAgent uses the a new resource object in case the socket handle gets reused.

As discussed in nodejs/diagnostics#248, #21313 and https://docs.google.com/document/d/1g8OrG5lMIUhRn1zbkutgY83MiTSMx-0NHDs8Bf-nXxM/preview reusing the resource object is a blocker for landing a resource based async hooks API and get rid of the promise destroy hook. This PR ensures that HttpAgent uses the a new resource object in case the socket handle gets reused. PR-URL: #27581 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

mcollina requested review from AndreasMadsen and bmeurer June 13, 2018 13:50

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Jun 13, 2018

async_hooks: add currentResource

15ffe72

Remove the need for the destroy hook in the basic APM case.

mcollina force-pushed the current-resource branch from 528f321 to 15ffe72 Compare June 13, 2018 13:51

mcollina added the async_hooks Issues and PRs related to the async hooks subsystem. label Jun 13, 2018

mcollina mentioned this pull request Jun 13, 2018

Node.js Foundation Diagnostics WorkGroup Meeting 2018-06-13 nodejs/diagnostics#204

Closed

mcollina requested review from watson and ofrobots June 13, 2018 14:15

addaleax reviewed Jun 13, 2018

View reviewed changes

bmeurer approved these changes Jun 13, 2018

View reviewed changes

apapirovski approved these changes Jun 13, 2018

View reviewed changes

Flarna reviewed Jun 13, 2018

View reviewed changes

mike-kaufman reviewed Jun 13, 2018

View reviewed changes

vsemozhetbyt reviewed Jun 13, 2018

View reviewed changes

mike-kaufman reviewed Jun 13, 2018

View reviewed changes

addaleax requested changes Jun 13, 2018

View reviewed changes

AndreasMadsen suggested changes Jun 14, 2018

View reviewed changes

Fishrock123 mentioned this pull request Jun 21, 2018

async_hooks: add getActiveResources (prototype) #21453

Closed

4 tasks

mcollina closed this Aug 21, 2018

ofrobots mentioned this pull request Oct 17, 2018

Deep dive discussion on {Async,Promise}_hooks & performance nodejs/diagnostics#248

Closed

mmarchini mentioned this pull request Nov 13, 2018

WIP http_parser: don't reuse HTTPParser #24330

Closed

3 tasks

mmarchini mentioned this pull request Nov 13, 2018

Node.js Foundation Diagnostics WorkGroup Meeting 2018-11-14 nodejs/diagnostics#252

Closed

Drieger mentioned this pull request Dec 17, 2018

src: do not reuse HTTPParser #25094

Closed

4 tasks

addaleax mentioned this pull request Feb 28, 2019

node::MakeCallback(), napi_make_callback() requires recv to be an object / not empty #26342

Closed

Flarna mentioned this pull request Mar 14, 2019

async-hooks: introduce async-storage API #26540

Closed

4 tasks

Flarna mentioned this pull request May 6, 2019

async_hooks: don't reuse resource in HttpAgent #27581

Closed

3 tasks

Qard mentioned this pull request Dec 14, 2019

async_hooks: add executionAsyncResource #30959

Closed

4 tasks

vdeturckheim mentioned this pull request Feb 4, 2020

Executive summary: Introducing a CLS-like API to Node.js core nodejs/TSC#807

Closed

github-actions bot mentioned this pull request Apr 24, 2024

Adding VersionStream for nodejs-22 wolfi-dev/os#17646

Merged

async_hooks: add currentResource #21313

async_hooks: add currentResource #21313

Conversation

mcollina commented Jun 13, 2018

Checklist

nodejs-github-bot commented Jun 13, 2018

mcollina commented Jun 13, 2018

addaleax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmeurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apapirovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flarna left a comment

Choose a reason for hiding this comment

Flarna commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

AndreasMadsen left a comment • edited Loading

Choose a reason for hiding this comment

Fishrock123 commented Jun 20, 2018

mcollina commented Aug 21, 2018

AndreasMadsen commented Aug 21, 2018

jasnell commented Aug 21, 2018

mcollina commented Aug 21, 2018

AndreasMadsen commented Aug 21, 2018

jasnell commented Aug 21, 2018

bmeurer commented Aug 21, 2018

AndreasMadsen commented Aug 22, 2018

Flarna commented Sep 20, 2018

AndreasMadsen left a comment •

edited

Loading