Segment stops responding

[mig42]

Hi,

We have an ASP.NET Core server running on Azure with analytics.NET 3.3.0-alpha, and it's configured to send user events to Segment using Analytics.Client.Track(). It's initialized using this line:

Analytics.Initialize(
    apiKey,
    new Config().setAsync(true).SetTimeout(100).SetMaxQueueSize(10000));

After a while (around 1h or so) segment stops confirming the reception of the tracked events. There are no new entries in the debug panel, meaning they don't arrive to Segment. It all starts working when we republish the app, probably because the program restarts.

For instance, this is the last event that was successfully sent a couple of days ago:

2018-07-03 07:53:42,915 36 INFO  Analytics - Segment.io request successful.
  batch id: c5a1b751-c683-430b-b31d-5c30bb0eacf0
  duration (ms): 87
2018-07-03 07:53:42,916 5 DEBUG Analytics - Queue is empty, flushing is finished.
2018-07-03 07:53:43,738 59 DEBUG Analytics - Blocking flush waiting until the queue if fully empty ..
2018-07-03 07:53:43,739 59 DEBUG Analytics - Blocking flush completed. 
2018-07-03 07:53:55,741 59 DEBUG Analytics - Enqueued action in queue.
  queue size: 1
2018-07-03 07:53:55,741 5 DEBUG Analytics - Dequeued action in async loop.
  message id: 4766d95e-d101-4215-a935-ca942e955559
  queue size: 0
2018-07-03 07:53:55,742 5 DEBUG Analytics - Created flush batch.
  batch size: 1
2018-07-03 07:53:55,743 59 DEBUG Analytics - Enqueued action in queue.
  queue size: 1

After that, no matter what we do, segment says "sending request" but never says "request successful" again:

2018-07-03 07:53:55,743 5 INFO  Analytics - Sending analytics request to Segment.io ..
  batch id: cd65ad4c-6ea3-43c9-b030-22205ac7a7a2
  json size: 448
  batch size: 1

Could this be related to #83 ? We also contacted Segment support, so I'll update this issue when we have more details.

Thanks!
Miguel

[mig42]

We've modified our app to create a Segment.Client instance (instead of relying on the static Analytics.Client) and we're disposing it and creating a new one every 30 minutes. So far events keep reaching Segment after 4 hours of server uptime, so there's probably some issue with the static way of working. Chances are we were hitting the issue fixed by pull request #83...

[daletskyi]

Hi @mig42 We faced exactly the same issue and #83 fixed it. Also #84 fixed error logging, so you can see failed requests in logs. However you need to wait until new nuget is released or build custom dll from master.

[mig42]

Hi @daletskyi, thanks a lot! We'll consider building our own dll, in the meantime I'd like to keep the thread open in case someone from the Segment dev team can confirm the bug in their library, unless it's too obvious!

Thank you!!

[f2prateek]

we plan on releasing the fix in #83 soon, waiting to hear back from a few folks who are testing it out as well

[f2prateek]

i’ve released 3.3.1-alpha with the PR in question so please try it out!

[mig42]

Thanks @f2prateek ! We'll upgrade ASAP and I'll update this thread with the results.

[f2prateek]

Thanks - please let us know once you've had a chance to try out the fix.

[PangbornIdentity]

Hello, we are using 3.3.1-alpha now in production, and with .net (not core), we find that we are no longer having issues with events stopping posting to segment. However, we are having an issue now where our cpu usage steadily climbs to dangerous level throught the day. Is it possible there is a GC issue with timers? or not using a static httpclient (microsoft/ApplicationInsights-dotnet#594)? its not related to use spikes or any blocked requests

[PangbornIdentity]

This was resolved by moving our service to a singleton scope to match that of the static analytics client

[f2prateek]

I see - so were you previously creating multiple new instances of the Analytics client?

[f2prateek]

(I'm going to close this issue as the original problem was solved by #83)