Add identification headers to SDK communication with servers #754

jreiberkyle · 2022-11-10T04:15:37Z

Related Issue(s):

Closes #739

Proposed Changes:

For inclusion in changelog:

Client identification headers are now sent with all requests.

Not intended for changelog:

The Request model has been removed and Sessions.request() and Sessions.stream() now take in arguments such as url, method, json, etc, and create the request internally.
All clients now have their methods call Session.request() directly instead of all going through a private method of the class
The Paged class in models now takes in the first page response as an argument instead of starting with a Request object and submitting the request and handling the first page internally. This was necessitated by the removal of the Request object but also has the benefit of earlier notification to the user of an error when the first page request was unsuccessful. Previously, the first page was only requested (and error thrown) when the user started using the generator.
Adds tests for confirming that the headers are sent in requests and a test for orders client to confirm the expected request is sent to the server.

Diff of User Interface

Old behavior:

From the code and output below, this is the relevant line:

2022-11-09 20:07:38,294 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - Headers({'host': 'api.planet.com', 'content-type': 'application/json', 'content-length': '158', 'authorization': '[secure]'})

Note that the headers 'x-planet-app' and 'user-agent' are not included.

planet-client-python$> planet orders request --name test --id 20221109_052227_10_2432 PSScene analytic_udm2 | planet --verbosity debug orders create
2022-11-09 20:07:38,281 - asyncio - DEBUG - Using selector: KqueueSelector
2022-11-09 20:07:38,282 - planet.auth - DEBUG - Reading from /Users/jennifer.kyle/.planet.json
2022-11-09 20:07:38,282 - planet.auth - DEBUG - Auth read from secret file /Users/jennifer.kyle/.planet.json.
2022-11-09 20:07:38,282 - planet.http - INFO - Session read timeout set to 30.0.
2022-11-09 20:07:38,294 - planet.http - DEBUG - Throttling cadence set to 0.1s.
2022-11-09 20:07:38,294 - planet.http - DEBUG - Workers capped at 50.
2022-11-09 20:07:38,294 - planet.http - DEBUG - Worker acquired.
2022-11-09 20:07:38,294 - planet.http - INFO - POST https://api.planet.com/compute/ops/orders/v2 - Sent
2022-11-09 20:07:38,294 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - Headers({'host': 'api.planet.com', 'content-type': 'application/json', 'content-length': '158', 'authorization': '[secure]'})
2022-11-09 20:07:38,295 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - b'{"name": "test", "products": [{"item_ids": ["20221109_052227_10_2432"], "item_type": "PSScene", "product_bundle": "analytic_udm2"}], "metadata": {"stac": {}}}'
2022-11-09 20:07:39,033 - httpx._client - DEBUG - HTTP Request: POST https://api.planet.com/compute/ops/orders/v2 "HTTP/1.1 202 Accepted"
2022-11-09 20:07:39,033 - planet.http - INFO - POST https://api.planet.com/compute/ops/orders/v2 - Status 202
2022-11-09 20:07:39,033 - planet.http - DEBUG - Headers({'content-type': 'application/json', 'date': 'Thu, 10 Nov 2022 04:07:39 GMT', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'vary': 'Origin', 'x-planet-span-id': '3178ea59-5cb8-444e-a272-ce0812008a28', 'x-planet-trace-id': '369209dd-9d29-43ae-8d98-a0281e552530', 'x-request-id': '369209dd-9d29-43ae-8d98-a0281e552530', 'content-length': '445', 'via': '1.1 edge, 1.1 google', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2022-11-09 20:07:39,033 - planet.http - DEBUG - Worker released.
{"_links": {"_self": "https://api.planet.com/compute/ops/orders/v2/ce65731e-ee90-4dd4-8950-29ba5496b47f"}, "created_on": "2022-11-10T04:07:38.742Z", "error_hints": [], "id": "ce65731e-ee90-4dd4-8950-29ba5496b47f", "last_message": "Preparing order", "last_modified": "2022-11-10T04:07:38.742Z", "metadata": {"stac": {}}, "name": "test", "products": [{"item_ids": ["20221109_052227_10_2432"], "item_type": "PSScene", "product_bundle": "analytic_udm2"}], "state": "queued"}

New behavior:

From the code and output below, this is the relevant line:

2022-11-09 20:08:58,295 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - Headers({'host': 'api.planet.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'planet-client-python/2.0a6dev', 'x-planet-app': 'python-cli', 'content-type': 'application/json', 'content-length': '158', 'authorization': '[secure]'})

Note that the headers 'x-planet-app' and 'user-agent' are now included.

planet-client-python$> planet orders request --name test --id 20221109_052227_10_2432 PSScene analytic_udm2 | planet --verbosity debug orders create
2022-11-09 20:08:58,282 - asyncio - DEBUG - Using selector: KqueueSelector
2022-11-09 20:08:58,282 - planet.auth - DEBUG - Reading from /Users/jennifer.kyle/.planet.json
2022-11-09 20:08:58,282 - planet.auth - DEBUG - Auth read from secret file /Users/jennifer.kyle/.planet.json.
2022-11-09 20:08:58,282 - planet.http - INFO - Session read timeout set to 30.0.
2022-11-09 20:08:58,294 - planet.http - DEBUG - Throttling cadence set to 0.1s.
2022-11-09 20:08:58,294 - planet.http - DEBUG - Workers capped at 50.
2022-11-09 20:08:58,295 - planet.http - DEBUG - Worker acquired.
2022-11-09 20:08:58,295 - planet.http - INFO - POST https://api.planet.com/compute/ops/orders/v2 - Sent
2022-11-09 20:08:58,295 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - Headers({'host': 'api.planet.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'planet-client-python/2.0a6dev', 'x-planet-app': 'python-cli', 'content-type': 'application/json', 'content-length': '158', 'authorization': '[secure]'})
2022-11-09 20:08:58,295 - planet.http - DEBUG - POST https://api.planet.com/compute/ops/orders/v2 - b'{"name": "test", "products": [{"item_ids": ["20221109_052227_10_2432"], "item_type": "PSScene", "product_bundle": "analytic_udm2"}], "metadata": {"stac": {}}}'
2022-11-09 20:08:58,986 - httpx._client - DEBUG - HTTP Request: POST https://api.planet.com/compute/ops/orders/v2 "HTTP/1.1 202 Accepted"
2022-11-09 20:08:58,986 - planet.http - INFO - POST https://api.planet.com/compute/ops/orders/v2 - Status 202
2022-11-09 20:08:58,986 - planet.http - DEBUG - Headers({'content-encoding': 'gzip', 'content-type': 'application/json', 'date': 'Thu, 10 Nov 2022 04:08:59 GMT', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'vary': 'Origin', 'x-planet-span-id': 'b8141866-f147-4e5c-867f-928c87e7f3c9', 'x-planet-trace-id': '8f59f424-7a2d-4dfa-aeed-281c1cb5dd6f', 'x-request-id': '8f59f424-7a2d-4dfa-aeed-281c1cb5dd6f', 'via': '1.1 edge, 1.1 google', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'transfer-encoding': 'chunked'})
2022-11-09 20:08:58,988 - planet.http - DEBUG - Worker released.
{"_links": {"_self": "https://api.planet.com/compute/ops/orders/v2/93fb9b71-cbd7-4ed7-be8b-c04c0fc08ec2"}, "created_on": "2022-11-10T04:08:58.733Z", "error_hints": [], "id": "93fb9b71-cbd7-4ed7-be8b-c04c0fc08ec2", "last_message": "Preparing order", "last_modified": "2022-11-10T04:08:58.733Z", "metadata": {"stac": {}}, "name": "test", "products": [{"item_ids": ["20221109_052227_10_2432"], "item_type": "PSScene", "product_bundle": "analytic_udm2"}], "state": "queued"}

PR Checklist:

This PR is as small and focused as possible
The title of this PR and/or the list of proposed changes are ready for inclusion in the Changelog or I have determined a Changelog entry is not required
I have updated docstrings for function changes and docs in the 'docs' folder for user interface / behavior changes
This PR does not break any examples or I have updated them

(Optional) @mentions for Notifications:
@cholmes @sgillies

Removes the Request class from model module. Paged now is initialized with the first page JSON and a callable to retrieve subsequent pages JSON. Response now only stores what is necessary from the http response and does not store the reqest. Adds StreamingResponse to support StreamingBody with additional properties. Removes Response last_modified property as it is unused.

remove Response class and have Session make requests Paged is initiated with a response instead of a request Remove Stream class and have Session.stream be a context manager yielding a streaming response move orders client to new request/response model

… to use new sessions.request format

… used

… data

sgillies · 2022-11-10T15:24:44Z

planet/clients/data.py

-        response = await self._do_request(request)
+        response = await self._session.request(method='PUT',
+                                               url=url,
+                                               json=request)


@jreiberkyle @kevinlacaille tangential comment here: the code might be more readable if we didn't use "request" as both a verb and a noun. Maybe self._session.request could be changed to self._session.send someday?

Ohh yeah I fully agree here! Right now Session has two methods that communicate with the servers: request() and stream(). Both of these suffer from verb/noun ambiguity. This was some of the satisfaction in removing the Request and Stream classes. At least we no longer have noun representation in the code base.

sgillies · 2022-11-10T15:28:36Z

planet/clients/subscriptions.py

+                                                   url=url,
+                                                   params=params)
+            async for sub in _SubscriptionsPager(response,
+                                                 self._session.request,


sgillies · 2022-11-10T15:34:23Z

tests/unit/test_models.py

-            }, 'items': [1, 2]
-        })
+    async def get_response(url, method):
+        return resp


I like that the pager change makes testing more clear.

yeah, the change was actually partially informed by the test formation. That was nice. Revisiting one's old code and making it smoother with the benefit of hindsight is very satisfying.

sgillies

@jreiberkyle excellent 🚀

jreiberkyle added 13 commits November 4, 2022 21:43

Paged initiated with a response instead of json

fdf9c12

use session.request directly instead of client method

0fb1ea1

update data client to use session.request directly

b8f23be

update subscriptions client to have session be a private property and…

9b971f2

… to use new sessions.request format

remove empty Request class placeholder because the class is no longer…

cb74136

… used

Merge branch 'main' into headers-fix-739

b2b86d0

type hinting fixes

becfe13

remove unused portions of tests

0f62a90

add test for request headers and content

1e78a7b

add test for identifying headers

e15bfc7

fix create_order not sending data with the request and add a test for…

2a13a07

… data

jreiberkyle requested review from kevinlacaille and sgillies November 10, 2022 04:27

sgillies reviewed Nov 10, 2022

View reviewed changes

sgillies approved these changes Nov 10, 2022

View reviewed changes

jreiberkyle merged commit bbb4f28 into main Nov 10, 2022

jreiberkyle deleted the headers-fix-739 branch November 10, 2022 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add identification headers to SDK communication with servers #754

Add identification headers to SDK communication with servers #754

Uh oh!

jreiberkyle commented Nov 10, 2022

Uh oh!

sgillies Nov 10, 2022

Uh oh!

jreiberkyle Nov 10, 2022

Uh oh!

sgillies Nov 10, 2022

Uh oh!

sgillies Nov 10, 2022

Uh oh!

jreiberkyle Nov 10, 2022

Uh oh!

sgillies left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add identification headers to SDK communication with servers #754

Add identification headers to SDK communication with servers #754

Uh oh!

Conversation

jreiberkyle commented Nov 10, 2022

Uh oh!

sgillies Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

jreiberkyle Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

sgillies Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

sgillies Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

jreiberkyle Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

sgillies left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants