-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stripe CLI is not receiving all events #600
Comments
Hello @jaredlt, When the CLI creates endpoints it creates one for the latest Stripe API version and your default Stripe API Version. The CLI running on your device will receive 2 versions of each event. And the CLI locally filters one of them based on whether the --latest-api-version flag is used or not. This is so you can use the CLI on the latest version. If you upgrade your Stripe API version you won't see that log error anymore. We don't think those messages are related to the dropped events and are looking into this more. Can you tell us what webhooks you were expecting that didn't come through? |
I also encountered this issue and can provide logs. This is for a
Command:
(Note there is a typo in the prop |
@ctrudeau-stripe I'm expecting I've done some more testing and I have found:
Command used:
But I also experienced the same issue without setting the Below shows debug logs for the last forwarded
NB. When I view the Events within the Stripe Dashboard, I can see both, and when I click each event to view their |
@jameskeiger @jaredlt do things work as expected if you run listen with To share a little bit of context on what might be happening here: webhook endpoints are pinned to specific API versions when they're created and Stripe will only send each endpoint the event for the right API version shape. So, for example: if you have a webhook endpoint that's pinned on |
@tomer-stripe Thanks for looking. Removing the |
No, still have the same issue
No, I don't believe so. I'm using the stripe-ruby Gem
No, I'm only using |
Thanks @jaredlt! I'm going to keep looking at this and see if I can repro this locally with the same setup you've got. |
@tomer-stripe 👍 If I can help with anything further just let me know. FYI usually I get a failure within 10 Checkouts but one time I had to test ~35 Checkout payments before it failed :( Almost questioned my life choices on that one! |
@jaredlt sorry for the delay here! I can repro this, it looks like even just running |
I'm getting this too. It's causing frequent test failures in our CI. When things are working, the Stripe CLI reports two FWIW our Stripe settings specify our API version as 2020-08-27, and all our requests to Stripe set |
Right around the time of the failed checkout session I do get
Perhaps this disconnect/reconnect is causing the relevant event to be dropped? If the event is redelivered later that wouldn't help because we make the test time out after a minute. It's not clear to me why the socket would have dropped here. It's happily receiving and sending messages up to the moment of the I/O error and the redial succeeds instantly. |
I have the same setup and issue as @rocallahan (2020-08-27 setup in the dashboard and specified in the Node.js Stripe library constructor). I also observed those dropouts in connection:
Then the next event is dropped due to mismatching API versions (however the event body seems to indicate the correct API version):
Stripe CLI installed on macOS via Homebrew, version 1.5.12. |
Same thing here. The connection is dropped after a successful checkout on the dashboard. $HOME/opt/stripe listen --log-level debug [Wed, 14 Apr 2021 13:14:00 EDT] DEBUG websocket.Client.writePump: Sending ping message Environment: Ubuntu 20.04LTS, stripe version 1.5.12 |
We're working around this in our CI by scanning the stripe-cli debug log for "Disconnected" messages and if we see one, abandoning the test without reporting a failure. |
Hey everyone, sorry about the delay here. We're investigating some possible leads on our side and think this might have to do with the reconnection logic we've got on the client and server. We're trying to confirm this as the problem and work on ways to improve it. |
FWIW I am seeing cases where an event is being totally ignored because "Received event with non-default API version", but there is no disconnection recorded in the log :-(. |
Hi all, want to share some updates here and a few areas we're exploring.
I'll post another update on Friday this week. I'm also going to pin this issue and rename it to "Stripe CLI is not receiving all events" so that if other folks are hitting the same problem we can centralize the discussion. edit: original title for posterity was |
As requested, here's a |
Thanks @kylefox this is very helpful! |
Quick update here -- we've made several improvements to our reconnection times which should help. We're still chasing down several larger culprits that are likely introducing race conditions but have started work to improve those conditions as well. In the meantime, if anyone notices any dropped events please give us the event id! |
Similar to all the other ppl, |
Thanks @goetas! The team's digging into that one too. |
same issue here. |
In addition to the issues described above, I also experience dropped events using stripe events resend evt_xxxx. |
Thanks @codeofsumit! I've shared with the team. @fredrikholm would you happen to have an event id associated with that? Also do you see the event show up in the dashboard as resent? |
@tomer-stripe - sorry, I don't have any event ids, but I will keep a record of those that fail from now on and post them here. And a question: is the listen --forward-to command supposed to run until canceled, or is there a silent timeout? Good luck with the bug hunting! |
@tomer-stripe Would it be possible to move the reporting of specific events IDs somewhere else besides this GitHub thread? Can we submit them directly to Stripe via email or another channel? |
@kylefox sure, feel free to email me any new event ids: tomer@stripe.com |
I am experiencing the same issue. Latest CLI version 1.6.0, updated just now. |
Hey @ctrudeau-stripe @tomer-stripe @pepin-stripe — any updates on this? 😄 Event delivery appeared to get more reliable, so I assumed the issue had been fixed. But it's recently become unreliable again. Not sure if that's due to changes Stripe is making, or if it's just the randomness of life 🤷🏻♂️ Is there anything contributors (i.e. non-Stripe employees) can do to help? From the sounds of it, this issue requires server-side troubleshooting/fixes rather than client-side? |
Hey @kylefox! I don't have any big updates for you but we've been pretty heads down on over this for a few weeks. Would you mind sharing when you noticed unreliability or any event ids? Part of what we're trying to do is tease apart issues with the CLI vs general webhook delivery slowness. |
Hello everybody,
|
Hello everyone! Just had to deal with a combination of all the problems discussed here, recently. It seems there are multiple causes at play. On the one hand, there were events missed because of reconnection issues which has been discussed at the top of the thread and probably fixed. On the other hand, since stripe-cli 1.6.0, Stripe seems to have added payload validation for events which does not deal well with payloads from much older api versions. For historical reasons, our default api version is stuck at the end of 2015 and we were also missing customer.subscription.updated events. Logging in debug mode reported a malformed event on the 2015 api version and discarded the message. The latest api version message was also discarded because our default api version is not up to date. The end result is nothing got through to our internal development endpoint. The quick fix until Stripe deals with this is to revert to stripe-cli v 1.5.14. @stripe will you look at improving the web hook structure validation to cover for all api versions or is this a nudge for everyone to upgrade? |
For me, it seems like I stop receiving events when the system has been idle for a period of time. |
I'm also having this issue. I'm using Stripe Checkout for subscriptions. I received events for:
Notably missing is... customer.subscription.created . The event ID for that one that never got through is |
@ctrudeau-stripe @tomer-stripe I hate to nag, but is this issue being actively worked on? Reliable If it's helpful, here's some more output from a specific instance I encountered today (using
For reference, here's the command I ran to listen: stripe listen \
--skip-verify \
--log-level debug
--forward-to https://api.lvh.me:5001/webhooks/stripe \
--forward-connect-to https://api.lvh.me:5001/webhooks/stripe/connect This is the concerning line from the output:
When I check the associated Stripe accounts, they're all using (I am not explicitly setting an API version in code, i.e. Is it possible there's a bug in client code that filters out webhooks due to API versions? Because it sounds like other people in this thread are also seeing unexpected behavior with Thanks! |
I also ran the above command with the
I'm not a Go programmer, but is it possible Lines 263 to 282 in 959df58
|
Ubuntu 20.04 using CLI 1.7.0 here. It looks like the websocket died on me on connection reset. The cli had been running successfully and had reset just four minutes before this. The last successful event was
|
Hi! I went here from #487. I was experiencing similar issue with Stripe CLI Docker on Windows. I turned out that disabling firewall solved the problem. Not an elegant solution but works for now 😉 |
Looks like in my case, "resetting the connection" just died for some reason. I started up a new instance of my app and worked on other things while waiting for everything to spin up. By the time I got back (about 5 minutes later)
if it makes any difference, we're running stripe CLI 1.7.0 in docker on an ubuntu 18 host. |
Using --latest flag seems to solve some missed calls but after being idle for some time requires restarting listener service |
Posting here in case someone runs into a similar thing that I did. We were testing out the new orders_beta v2 and noticed that we were not receiving any of these new events using What I did notice was the events were still firing in our test mode dashboard. So as a workaround I used Ngrok to expose our local API as an endpoint and configured that as a direct endpoint in stripe. While configuring the endpoint in stripe you can specify API version as well as specify event types to send.
After doing this our API started receiving all of the events. This isn't ideal as I think we'd all prefer using the stripe-cli for this, but hope that this helps someone! |
Hey @Cavallando, Thanks for sharing. The StripeCLI, nor any of the client libraries support beta features or events. Cheers. |
Encountered this issue today swapping from |
I'm going to close this issue since we've made significant progress here over the years. We rewrote the underlying system to better handle volume and concurrency, and we've also added more monitoring (and canary's) in place to alert us if the deliveries are dropping. |
Issue
I have noticed that some webhooks fail to forward to my localhost. It seems that all the failures occur because of a proxy filter within Stripe CLI saying
Received event with non-default API version, ignoring
(discovered when setting the--log-level debug
flag).NB. I am testing the Checkout flow and forwarding webhooks to my localhost eg.
stripe listen --forward-to localhost:4040/webhooks/stripe
This looks like the same issue as #435
Expected Behavior
All webhooks to be forwarded to my localhost.
Steps to reproduce
I cannot reliably reproduce this failure on the first try. But I can reliably reproduce the error. I just try multiple checkout payments and one of them will eventually not send.
Traceback
I have tidied up all the
Incoming message
lines to make the logs a bit easier to read. This is all against the testing gateway.Environment
Windows via WSL1
Notes
This was tough to initially debug as there was nothing in the default output. The expected webhooks just didn't display. At first I thought that maybe there was some conflict because I also have my staging environment configured to receive the testing webhooks but I believe that was a red herring. I only discovered the
--log-level debug
config after some digging. So just to say perhaps there is something to consider around displaying something about the webhook with the default log-level even if it's not sent?The text was updated successfully, but these errors were encountered: