batch create_entity error 400

[kovapatrik]

Is it possible to create a batch of create_entity requests?
I've got an error 400 status code for an attempt, but I'm not sure if the library doesn't allow to do this, or our organization's SAP system.
I'm guessing the first one because I didn't find any example of doing this.

[phanak-sap]

Working with batch request is documented here:
https://github.com/SAP/python-pyodata/blob/master/docs/usage/advanced.rst

Should work, we use that. If you are still having issues, we would need much more information than HTTP 400 - ideally test that could be reproduced locally. At least enable logging to DEBUG level (same doc page) and provide the log file - scan first for credentials that you do not want to post to the internet.

[kovapatrik]

Yeah I saw that part of the documentation, I meant that there is no example for the create_entity function. I have something in my mind that could go wrong. I will check it tomorrow, and I will post if it's solved or not.

[kovapatrik]

I tried it and sadly it did not work. Could it be a setting on the server side which doesn't allow me to create a batch request? I tried it with get_entity methods too, and it didn't work.

[phanak-sap]

@kovapatrik Did you read my comment - asking for some facts and details? I am sorry, but at the moment there is literally nothing we can do on our side. We cannot comment on configuration of unknown backend service (nor we can really support any backend service "to work") and there are no information on the error on the pyodata side - no stacktrace at least, no complete log, no test/sample code to reproduce the error somehow.

• Are you at least able to correctly create your entity with direct request, while failing inside batch request?
• Is your backend service even Odata V2 (and not for example v3/v4, which is not yet supported)?
• With what version of pyodata are you having the problem?
• Check existing better, more actionable issues here, with sample code provided. For example here - Cannot access entity with Edm.Binary key #187. This issue provided at least some starting point to investigate possible bug. Create therefore some reproducer code as shown here, that would for example initialize the pyodata with your metadata as local file and then provide the code to create your entity, so can be debugged the generated HTTP Request.
  But if it will look OK according to odata specification then it is just a guessing game why the server returns HTTP 400, that's for investigation on your end.

[kovapatrik]

I didn't even expect you to solve my problem with the information I was given so you don't have to be sarcastic. I just asked a question about a backend service I have no experience with, before we thinking about a bug in the pyodata library itself. I thought an SAP service is standard so there are standard possible configurations. But in any case, here is the code I'm using:

s = requests.Session()
s.auth = HTTPKerberosAuth(mutual_authentication=OPTIONAL)
s.headers = {'x-csrf-token': 'fetch'}
s.verify = ca_bundle_location

res = s.get(SERVICE_URL)
csrf_token = res.headers.get("x-csrf-token", "")
s.headers.update({"x-csrf-token": csrf_token})

client = pyodata.Client(SERVICE_URL, s)

batch = client.create_batch()

rq1 = client.entity_sets.LeaveRequestSet.create_entity().set(**d1)
rq2 = client.entity_sets.LeaveRequestSet.create_entity().set(**d2)

batch.add_request(rq1)
batch.add_request(rq2)

batch.execute()

For the batch.execute() command I get this error:
HttpError: HTTP POST for multipart request 1989_1874_3537 failed with status code 400

And here is the debug output.

Remark: If I create requests separately using the same data given in the batch requests, then the execution will be successful.

[phanak-sap]

Hi @kovapatrik, thx for provided info.

The HTTP 400 Bad requests is obvious. The correct creation without batch requests is a valuable fact.

What catched my eye in the provided python script is missing changeset wrapping. My current understanding of odata is that each create call should be in its own changeset = in your example 1 batch request, inside 2 changesets. Your script should IMHO work correctly if it would be just two entity queries, then the changesets should not be needed.

Please try to modify it according to second sample from pyodata documentation

I hope it will help and it is just that simple. If not, we will need to investigate deeper and consider this a possible bug.

links:

• https://pyodata.readthedocs.io/en/latest/usage/advanced.html?highlight=batch#batch-requests
• https://blogs.sap.com/2018/05/06/batch-request-in-sap-gateway/
• https://www.odata.org/documentation/odata-version-2-0/batch-processing/

[phanak-sap]

Note: I also noticed that we still have pending Enhancement for handling partial success scenario in Batch, so be aware - issue #64

For that particular reason, you may want to stick to simple create requests in a loop if the purpose is for example for data creation. From performance standpoint, as alternative to Batches - either use standard Python library threading module, or we use gevent + monkey patching in a project built upon pyodata (odfuzz, yeah ugly codebase, but somehow it works :) ) .

from gevent import monkey
monkey.patch_all()

Since we officially support primarily requests library (with possibility to inject other requests-like library, but never practically tried), there is thread blocking anyway. You may find this comment interesting if performance gain was the reason for Create entities in Batches.

[kovapatrik]

The changset wrapping indeed solves the problem. But I had to wrap each create_entity request within it's own changeset, because if I added 2 requests into 1 changeset, the server responded with an error like the server only allows 1 change per changeset.

I didn't meet with odata before, so IMO it would be nice if there is an example in the docs also for create_entity requests in batch including the changeset wrapping.

So thanks for the help!

My last question: performance-wise, is it better to send requests in batch or it doesn't matter?

[kovapatrik]

Oh I see you answered my question before I even asked :D I will look up the module you linked, thank you!

[phanak-sap]

1. if the examples in Advanced.rst were not helpful for you from start, I will be happy to review PR with update of this file with example which would be for you more understandable.
2. "Example of batch request that contains simple entity query as well as changest consisting of two requests for entity modification" - yeah that a bit expects you know about Batches more than just what is stated in this line only. It is true that user of pyodata package, even that is somehow shielded from for example the precise URL generation, is still somehow expected to understand basics of odata protocol.
3. It could be a nice to have enhancement tho, to check the Batch content before sent, if changesets are missing and needed.

[phanak-sap]

My last question: performance-wise, is it better to send requests in batch or it doesn't matter?

There will be definitely some difference between X create requests sent in a simple loop (due synchronous wait on response) and sending the same X create requests in a Batch. But the Batch primarily saves networking resources (for example in browser apps, Fiori), the HTTP traffic is using less bytes. From time standpoint, the server will still need to use basically same resources to actually create the entities somewhere in DB. And you need to handle possibility of some failures inside the X create requests.

The gevent/multihreading will remove the waiting time a bit, so for big test data creation script it will end definitely sooner, that's for sure. But check first if it is even worth the hassle :) Maybe requests sent in a simple loop you will still create enough data in seconds or minutes and you find no need to actually optimize. Or you want to create huge batch of data, but no problem if executed overnight, once a year.

[phanak-sap]

The changeset wrapping indeed solves the problem. But I had to wrap each create_entity request within it's own changeset, because if I added 2 requests into 1 changeset, the server responded with an error like the server only allows 1 change per changeset.

Yes, this was exactly what I meant, that this is IMHO mandatory in the odata protocol. One batch, two changesets in your example.

[kovapatrik]

It will be likely used once per month so I think I won't create any threading and use the simple create requests command.

For the docs change: Now I understand that I have to know something about the odata protocol before using pyodata, so I guess if I knew before that a create request must be in a changeset, then the docs are clear on how to use changesets.

[phanak-sap]

Closing as resolved.