Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error parsing JSON returned from webhook #1402

Closed
steve-tapley opened this issue Nov 23, 2018 · 19 comments · Fixed by #1416
Closed

Error parsing JSON returned from webhook #1402

steve-tapley opened this issue Nov 23, 2018 · 19 comments · Fixed by #1416

Comments

@steve-tapley
Copy link

steve-tapley commented Nov 23, 2018

Hi,
I've recently moved to version 21.4.1 from 19.x, and now I'm receiving errors when parsing some webhook events.

Below is some JSON that caused an error in the parser. It looks like it falls over trying to generate a request object - from what it looks like to be an Id only,

I've tried both the ParseEvent and ConstructEvent methods - both fail identically.

{ "id": "evt_1DZUr64WLhpec2eZ8YxDi4sm", "object": "event", "api_version": "2014-06-17", "created": 1542941840, "data": { "object": { "id": "card_1DZUqq4WLhpec2eZr2GUSqJW", "object": "card", "address_city": null, "address_country": null, "address_line1": null, "address_line1_check": null, "address_line2": null, "address_state": null, "address_zip": null, "address_zip_check": null, "brand": "Visa", "country": "US", "customer": "cus_4iIYzLmgQznYlK", "cvc_check": "pass", "dynamic_last4": null, "exp_month": 9, "exp_year": 2019, "fingerprint": "tXS6DZYDEDNFUeFL", "funding": "credit", "last4": "4242", "metadata": { }, "name": "Gabe Newell", "tokenization_method": null } }, "livemode": false, "pending_webhooks": 1, "request": "req_7hFEZuZn4RwSm6", "type": "customer.card.deleted" }

Newtonsoft.Json.JsonSerializationException
Message = Error converting value "req_7hFEZuZn4RwSm6" to type 'Stripe.EventRequest'. Path 'request', line 36, position 33.

Hack "fix" until this has been sorted out:

var jObject = Newtonsoft.Json.JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(json);`
if (jObject["request"] != null)`
{
    jObject["request"] = null;
    json = jObject.ToString();
}

Cheers,
Steve

@ob-stripe
Copy link
Contributor

Hi @steve-tapley. The Stripe.net library is pinned to a specific API version, and expects the objects it decodes to be formatted according to that version.

In this case, Stripe.net v21.4.1 is pinned to API version 2018-11-08 (cf. https://github.com/stripe/stripe-dotnet/blob/v21.4.1/src/Stripe.net/Infrastructure/Public/StripeConfiguration.cs#L10). According to the api_version key in the JSON object you shared, the event object you're trying to decode was formatted with API version 2014-06-17. As such, it's not unexpected that trying to decode that object would fail. The exact cause is the change in the request property introduced in API version 2017-05-25.

When sending requests to Stripe's API, the Stripe.net library uses the versioning header to force a specific API version. However, event objects are always formatted according to your Stripe account's default API version. So when using webhooks with the Stripe.net library, it's important that your account's default API version matches the API version expected by the Stripe.net library.

You can upgrade to the latest API version (which is currently 2018-11-08, the version expected by Stripe.net v21.4.1) from your dashboard at https://dashboard.stripe.com/developers.

In short, this is expected behavior. I think we can improve the developer experience with a clearer error message though, so I'll leave this issue open for now.

@steve-tapley
Copy link
Author

steve-tapley commented Nov 26, 2018

Hi @ob-stripe ,
Thanks for the info. Yes, a better error message would have saved me some time.

Theres a few other issues here too, I think.

It would be nice to know what API versions a particular Stripe.net version is compatible with (e.g 2018-08-23 thru to 2018-11-08 for example).

At the moment, its difficult to test upgrades of Stripe.Net, because I cant set my test environment to be a different version of API to the live version.
I also cant set it to be a specific version, just the latest version, which isn't very helpful at all when it comes to testing. Sure, I can send a test webhook, but that doesn't exercise all the code paths I need/want, and also doesnt help me for any integration testing I want to perform.

The docs at https://stripe.com/docs/upgrades recommend updating the webhook code to handle both old and new version of each objects - are they really implying having two versions of the Stripe.net library? Sure, I could do that by having multiple assemblies, but I really think thats something better done by Stripe.net + Stripe API.

@phil118
Copy link

phil118 commented Nov 26, 2018

Hi, we are experiencing a similar issue to what Steve explained. I have upgraded the api version to 2018-11-08 and on stripe.net version 21.4.1 but all our Webhooks are failing.

public Task HandleAsync(string jsonEvent)
{
    EnsureArg.IsNotNullOrEmpty(jsonEvent);
    var stripeEvent = EventUtility.ParseEvent(jsonEvent);
    return HandleAsync(stripeEvent);           
}

public Task HandleAsync(Event stripeEvent)
{
   EnsureArg.IsNotNull(stripeEvent);
   var eventData = Mapper<TEventData>.MapFromJson(stripeEvent.Data.Object.ToString());
   return HandleEventAsync(eventData); 
}
{"message":"An error has occurred.","exceptionMessage":"Unexpected character encountered while parsing value: S. Path '', line 0, position 0.","exceptionType":"Newtonsoft.Json.JsonReaderException","stackTrace":" at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.JsonTextReader.Read()\r\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonConverter[] converters)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonConverter[] converters)\r\n at Stripe.Mapper`1.MapFromJson(String json, String p...

@ob-stripe
Copy link
Contributor

@phil118 As of v20.0.0, the Data.Object property of Event objects is directly instantiated to the correct type. You can either use pattern matching or use the Event object's Type property to cast to the correct type. You can find more information here: https://github.com/stripe/stripe-dotnet/wiki/Migration-guide-for-v20#interfaces-for-polymorphic-resources.

@ob-stripe
Copy link
Contributor

Hi @steve-tapley.

It would be nice to know what API versions a particular Stripe.net version is compatible with (e.g 2018-08-23 thru to 2018-11-08 for example).

The only API version a particular Stripe.net version is 100% compatible with is the one it's pinned to. Trying to use a different API version may work for some resources, but not for others. Depending on the exact changes, it may fail silently (e.g. deserialization would appear to work, but some fields would not be filled because the API didn't return them and other fields would be missing because the API did return them but they're not present in the library).

At the moment, its difficult to test upgrades of Stripe.Net, because I cant set my test environment to be a different version of API to the live version.

You can create a test webhook endpoint with the latest API version, without having to upgrade the API version for the entire account. Once the endpoint is created, you can create test events "organically" (i.e. by sending requests that would result in the events you want, not by using the "Send test webhook" button) and those events will be formatted with the latest API version.

The docs at https://stripe.com/docs/upgrades recommend updating the webhook code to handle both old and new version of each objects - are they really implying having two versions of the Stripe.net library? Sure, I could do that by having multiple assemblies, but I really think thats something better done by Stripe.net + Stripe API.

Right, I think the advice in those docs is mostly applicable to our client libraries that are not pinned to a specific API version, like Ruby or PHP. Things are a lot easier there, because those libraries don't care about specific resource formats.

For Stripe.net, my recommendation would be to do something like this:

  • set up a separate test server with the latest Stripe.net version
  • create a test webhook endpoint with the latest API version, pointing to your test server
  • test your integration until you're confident everything works correctly
  • at this point, upgrade the API version for your entire Stripe account and upgrade your production server with the latest Stripe.net and whatever changes you made to your integration

That last step is difficult to do in one go, so you may want to temporarily force your webhook handler to return a non-2xx status code temporarily so that Stripe will retry those events ~one hour later.

I realize all this is not ideal and we can definitely do better to improve the experience. I'll share your feedback internally.

@steve-tapley
Copy link
Author

steve-tapley commented Nov 27, 2018

Thanks @ob-stripe ,
I've updated the test version of the webhook to the latest version (which stripe.net supports), and I can continue with some testing...

However, another thing to raise internally the language used when setting the version number is very confusing.
The choices are either "latest" or "default". If I choose latest, does this mean I will always be on the latest version? If thats the case, that does not work at all for a C# library, because I would be forced to immediately upgrade my stripe.net version every time stripe updates the API. If not, then the language used in the dialog needs clarifying (please!).

The other issue I have is that I only have two choices. How does someone working with older versions of Stripe.Net set their API level to match their library? Think about a long testing cycle where we are testing against one version, and then Stripe may release a new version of their API during this time. We cant release this version, because the APIs are now different - and we can't specify the older version in your UI either :(

@Mkey85
Copy link

Mkey85 commented Nov 28, 2018

I'm facing similar issues.

Here my json file:

{
   "id":"evt_1DbB8SJWTSmTXxfubvsUk8OS",
   "object":"event",
   "api_version":"2018-11-08",
   "created":1543342692,
   "data":{
      "object":{
         "client_reference_id":"Marcel",
         "display_items":[
            {
               "currency":"usd",
               "amount":79900,
               "type":"sku",
               "sku":"sku_E3B0aWd4SPPNcT"
            }
         ],
         "livemode":false,
         "payment_intent":"pi_1DbB7lJWTSmTXxfuAJcTNZNR"
      }
   },
   "livemode":false,
   "pending_webhooks":1,
   "request":{
      "id":"req_xsqB6ODvOyoolS",
      "idempotency_key":null
   },
   "type":"checkout_beta.session_succeeded"
}
var stripeEvent = EventUtility.ParseEvent(json);

Exception:

Exception thrown: 'System.ArgumentNullException' in Newtonsoft.Json.dll
An exception of type 'System.ArgumentNullException' occurred in Newtonsoft.Json.dll but was not handled in user code
**Value cannot be null.**

I also can't find any support for the "checkout_beta.session_succeeded" event, do I miss something here?
I'm using Stripe API 2018-11-08 and Stripe.net version 21.4.1 (also tried 21.6.0)

Same json object gets passed with stripe.net 19.7.0

Marcel

@Mkey85
Copy link

Mkey85 commented Nov 28, 2018

Removing the data object is parsing the json string correctly.
Can it be that there is currently no support for the the stripe checkout method??!

Please add support for the checkout_beta.session_succeeded event.

@remi-stripe
Copy link
Contributor

@Mkey85 The event checkout_beta.session_succeeded is temporary for the beta so we will not add it to the library since it will go away in a few months. It's a string though so you should be able to match based on the raw string instead of relying on the constant!

@Mkey85
Copy link

Mkey85 commented Nov 29, 2018

It's okay, I added support for checkout beta event in my local repo. Thanks a lot.

@dpellizza
Copy link

dpellizza commented Dec 19, 2018

Hi,
I've recently moved to version 21.7.0 and now I'm receiving errors when parsing some webhook events.

public async Task<HttpResponseMessage> Post(Stripe.Event stripevent)

The stripevent doesn't have the data.object property filled!!!! Why??? I can't cast to any object :(

Under the test event

{
  "id": "evt_1Dj7qrI3m27K9pdzmzaZr7x8",
  "object": "event",
  "api_version": "2018-11-08",
  "created": 1545236693,
  "data": {
    "object": {
      "id": "ch_1DecezI2m6fK9pdz8ul444Y3",
      "object": "charge",
      "amount": 100,
      "review": null,
      "shipping": null,
      "source": {
        "id": "card_1DecnvI2m6yu9pdz0prJbiPJ",
        "object": "card",
        "address_city": null,
        "address_country": null,
        "address_line1": null,
        "address_line1_check": null,
        "address_line2": null,
        "address_state": null,
        "address_zip": null,
        "address_zip_check": null,
        "brand": "MasterCard",
        "country": "AU",
        "customer": "cus_E6qUyYXgtZyGEZ",
        "cvc_check": "pass",
        "dynamic_last4": null,
        "exp_month": 7,
        "exp_year": 2022,
        "fingerprint": "vP1isMjJgtHz17sJt",
        "funding": "credit",
        "last4": "8908",
        "metadata": {
        },
        "name": null,
        "tokenization_method": null
      },
      "source_transfer": null,
      "statement_descriptor": "DOCdfsgsdfgPC",
      "status": "succeeded",
      "transfer": "tr_1Deco0I456fK9pdzxgE970dI",
      "transfer_group": "group_ch_1DecnzI2m6fK9pdz8ul444Y3"
    },
    "previous_attributes": {
      "amount_refunded": 0,
      "refunded": false,
      "refunds": {
        "data": [
        ],
        "total_count": 0
      }
    }
  },
  "livemode": false,
  "pending_webhooks": 1,
  "request": {
    "id": "req_v2abI45LNL6wAo",
    "idempotency_key": null
  },
  "type": "charge.refunded"
}

Thanks in advance,
Davide.

@ob-stripe
Copy link
Contributor

Hi @dpellizza, can you share your code for creating the Stripe.Event instance exactly, as well as casting the instance's Data.Object property?

@dpellizza
Copy link

dpellizza commented Dec 20, 2018

Hi @ob-stripe ,
this is the code I use to receive stripe events:

ASP.NET Web API 5.2.7 C#
Newtonsoft.Json 12.0.1
stripe.net 19.10 to stripe.net 21.7.0

    public class AccountEventController : BaseApiController
    {
        private static readonly ILog Log = LogManager.GetLogger("AccountEvent");

        public async Task<HttpResponseMessage> Post(Stripe.Event stripevent)
        {
            try
            {
                switch (stripevent.Type)
                {
                    case Stripe.Events.PayoutPaid:
                        Stripe.Payout payout = stripevent.Data.Object as Stripe.Payout;
						
						...

                        break;

                    case Stripe.Events.ChargePending:

                        Stripe.Charge chrPen = stripevent.Data.Object as Stripe.Charge;
                        
						...

                        break;

                    case Stripe.Events.ChargeSucceeded:

                        Stripe.Charge chrSuc = stripevent.Data.Object as Stripe.Charge;
                        ...
						
                        break;

                    case Stripe.Events.ChargeFailed:

                        Stripe.Charge chrFail = stripevent.Data.Object as Stripe.Charge;
                        ...
						
                        break;

                    case Stripe.Events.ChargeRefunded:

                        Stripe.Charge chrRefunded = stripevent.Data.Object as Stripe.Charge;
						...

                        break;

                    case Stripe.Events.ChargeDisputeCreated:

                        Stripe.Dispute chrDispCreate = stripevent.Data.Object as Stripe.Dispute;
                        ...

                        break;

                    case Stripe.Events.ChargeDisputeClosed:
                        Stripe.Dispute chrDisputeClose = stripevent.Data.Object as Stripe.Dispute;
                        ...

                        break;

                    default:
                        break;
                }

                ...
				
                return Request.CreateResponse(HttpStatusCode.OK);

            }
            catch (Exception ex)
            {
                Log.Error(string.Format("Webhook AccountEvent Error: {0} StackTrace: {1}", ex.Message, ex.StackTrace));
                return Request.CreateResponse(HttpStatusCode.InternalServerError);
            }

        }

    }

and this is one of the stripe event I had tried to consume (dispute event):

{
  "id": "evt_1Dj5pwI2m6fK9pdzk7MzLfJv",
  "object": "event",
  "api_version": "2018-11-08",
  "created": 1545228948,
  "data": {
    "object": {
      "id": "dp_1Dj5pwI2m6fK9pdzX7pXKbtA",
      "object": "dispute",
      "amount": 5600,
      "balance_transaction": "txn_1Dj5pwI2m6fK9pdzmONv85Rb",
      "balance_transactions": [
        {
          "id": "txn_1Dj5pwI2m6fK9pdzmONv85Rb",
          "object": "balance_transaction",
          "amount": -5600,
          "available_on": 1545782400,
          "created": 1545228948,
          "currency": "eur",
          "description": "Chargeback withdrawal for ch_1Dj5pvI2m6fK9pdztBJl66B3",
          "exchange_rate": null,
          "fee": 1500,
          "fee_details": [
            {
              "amount": 1500,
              "application": null,
              "currency": "eur",
              "description": "Dispute fee",
              "type": "stripe_fee"
            }
          ],
          "net": -7100,
          "source": "dp_1Dj5pwI2m6fK9pdzX7pXKbtA",
          "status": "pending",
          "type": "adjustment"
        }
      ],
      "charge": "ch_1Dj5pvI2m6fK9pdztBJl66B3",
      "created": 1545228948,
      "currency": "eur",
      "evidence": {
        "access_activity_log": null,
        "billing_address": null,
        "cancellation_policy": null,
        "cancellation_policy_disclosure": null,
        "cancellation_rebuttal": null,
        "customer_communication": null,
        "customer_email_address": "d.p01@t.it",
        "customer_name": "d.p01@t.it",
        "customer_purchase_ip": "212.106.210.242",
        "customer_signature": null,
        "duplicate_charge_documentation": null,
        "duplicate_charge_explanation": null,
        "duplicate_charge_id": null,
        "product_description": null,
        "receipt": null,
        "refund_policy": null,
        "refund_policy_disclosure": null,
        "refund_refusal_explanation": null,
        "service_date": null,
        "service_documentation": null,
        "shipping_address": null,
        "shipping_carrier": null,
        "shipping_date": null,
        "shipping_documentation": null,
        "shipping_tracking_number": null,
        "uncategorized_file": null,
        "uncategorized_text": null
      },
      "evidence_details": {
        "due_by": 1546041599,
        "has_evidence": false,
        "past_due": false,
        "submission_count": 0
      },
      "is_charge_refundable": false,
      "livemode": false,
      "metadata": {
      },
      "reason": "fraudulent",
      "status": "needs_response"
    }
  },
  "livemode": false,
  "pending_webhooks": 1,
  "request": {
    "id": "req_3R725wWXvJOYOJ",
    "idempotency_key": null
  },
  "type": "charge.dispute.created"
}

And this is the object values after Stripe.Event cast (Object is null!!)

2018-12-20_09h31_31

Thanks a lot,
Davide.

@ob-stripe
Copy link
Contributor

ob-stripe commented Dec 20, 2018

Hi @dpellizza, thanks for the detailed information. I don't have much experience with ASP.NET, but I believe this issue is similar to #1361. Basically, ASP.NET won't automatically fill Data.Object because doing so requires using the StripeObjectConverter custom JSON converter, which ASP.NET doesn't know about.

This issue should be fixed in the upcoming v22 version (cf. #1396). In the meantime, you can work around the issue by deserializing the event manually using the Stripe.EventUtility.ParseEvent method. Something like this should work:

public async Task<HttpResponseMessage> Post()
{
    string rawBody = response.Content.ReadAsStringAsync().Result;
    var stripeEvent = Stripe.EventUtility.ParseEvent(rawBody);
    ...
}

(Not directly related to your issue, but ideally you should use the Stripe.EventUtility.ConstructEvent method so that you verify the webhook signature while constructing the event.)

Let me know if that helps!

@dpellizza
Copy link

Hi @ob-stripe,
this solves my problem and more now I manage webhook security ;)
Thanks!

@ob-stripe
Copy link
Contributor

@dpellizza Awesome! Glad you were able to fix the problem.

@ob-stripe
Copy link
Contributor

This is "fixed" in 22.0.0. The library will now correctly deserialize event objects regardless of their API versions, but raise an exception if an API version mismatch is detected (as the nested object might not be correctly deserialized). You can find more information in the migration guide for v22: https://github.com/stripe/stripe-dotnet/wiki/Migration-guide-for-v22#event-deserialization-with-eventutilityconstructevent--eventutilityparseevent.

@leinzoeten
Copy link

Not sure if this is working with the new version of stripe or that I am missing something. I have a . net 4.6 environment upgraded to the latest version of Stripe.net for .net and also updated my version to 2019-03-14 on the dashboard, I have:
Stream req = Request.InputStream;
req.Seek(0, System.IO.SeekOrigin.Begin);
string json = new StreamReader(req).ReadToEnd();
(So far its working and if I would write the Json string to for example a database its as the event that was send from the dasboard)
The next step always returns an error and stops there, with or without the , true/false:
Event stripeEvent = EventUtility.ParseEvent(json);

Any idea?

@niemyjski
Copy link

niemyjski commented Jun 4, 2019

I'm also seeing null reference exceptions, I'm just trying to construct some events and I saved the raw Json event that I saw in my dashboard Response Body but that throws a null reference as shown above, What do I need to do to parse one of those events from the stripe ui?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants