[RFC] Working with cookies

[alexeyzimarev]

Separating the cookie discussion from #1786

How would the authenticator change the token though? Since the authenticator is called on a per request basis, unless the tokens are static it needs to do something to configure them.

It can be done in the authenticator itself as I've done in the sample https://restsharp.dev/usage.html#authenticator

I would suggest Encode and EncodeQuery become part of the Options and would not be changed for a client instance.

Agreed

I am not sure it's desirable to have a shared cookie container for a single RestClient instance it that thing is cached forever.

I am not sure either as I never used it. But it always been a part of the client (or request, don't remember) for some reason. Yes, I think it was on the request, but since it is used for the message handler instantiation and cannot be changed, I had to move it to the client. I guess the discussion if it was a good decision, maybe not.

If there would be a way to specify cookies per request using headers, the cookie container could stay as-is, but AddCookie in the client can be removed, or converted to AddDefaultCookie, and we can add AddCookie to the request, where it was previously.

[kendallb]

Separating the cookie discussion from #1786

How would the authenticator change the token though? Since the authenticator is called on a per request basis, unless the tokens are static it needs to do something to configure them.

It can be done in the authenticator itself as I've done in the sample https://restsharp.dev/usage.html#authenticator

I don't agree that is a viable option. The reason is that your Twitter example where you need to get and cache at runtime a token, based on a static clientId and clientSecret. Those are configured once and the authenticator then dynamically calls Twitter to get the value and then stores it for later use. So nothing in that scenario allows for that tokens to be dynamically managed on a per used basis which is needed for some API's in multi user environments like web apps.

BTW, this brings up something else I noticed. The function to get the GetAuthenticationParameter() ends up needing it's own RestClient internally to call Twitter the first time. The actual function to process authorization is passed the current RestClient and RestRequest, but those are NOT passed onto the GetAuthenticationParameter() function:

    public async ValueTask Authenticate(RestClient client, RestRequest request)
        => request.AddOrUpdateParameter(await GetAuthenticationParameter(Token).ConfigureAwait(false));

I think that function signature should be changed (or have another overload for it) that would be passed the RestClient and RestRequest. That way you can avoid having to create a short lived RestClient for the Twitter token that needs to be disposed of, but more importantly the authenticator class CAN inspect the RestRequest to get request specific stuff out of it.

Currently I am not sure where you would put the request specific stuff in the RestRequest property bag, but it could be retrieved from a header or something.

Or perhaps the correct solution is a combination of a) adding an overload for GetAuthenticationParameter() that accepts the RestClient and RestRequest, as well as hooking up an optional version of that in the RestRequest itself so the caller can construct something to be done at request time. That way the user has two ways to wire up an Authenticator. Wire up a static one that is the same across all API calls in the RestClient, or wire up a request specific one in RestRequest.

That was kind of the approach I was planning to take in the PR I am working on.

Although this is such an important issue IMHO to avoid folks trying to set this on a single instance RestClient at request time and having threading issues, that I almost feel breaking the API and moving it to the request is the correct approach. Otherwise you will end up with two problems with existing code being ported to v107:

1. They port to v107 and continue to use a RestClient per request, as EasyPost did as stuff like configuring the Authenticator in the client kinda indicates it's request specific.
2. They correctly change to using a shared instance of RestClient per API entry point, but override the Authenticator on a per request basis (as it's entirely possible to do that) and end up with really wonky runtime threading issues (or cross request authentication bugs).

I am not sure it's desirable to have a shared cookie container for a single RestClient instance it that thing is cached forever.

I am not sure either as I never used it. But it always been a part of the client (or request, don't remember) for some reason. Yes, I think it was on the request, but since it is used for the message handler instantiation and cannot be changed, I had to move it to the client. I guess the discussion if it was a good decision, maybe not.

Actually it was always on the client, which is another reason everyone (including myself) used a RestClient per request. I am in the process of re-writing the current code that uses that right now. In our case, we manage our own cross request CookieContainer and setup a new cookie container on every request with just what is needed. I think the correct place for the cookie container is in the RestRequest, not RestClient as it needs to be request specific and not shared between calls.

If there would be a way to specify cookies per request using headers, the cookie container could stay as-is, but AddCookie in the client can be removed, or converted to AddDefaultCookie, and we can add AddCookie to the request, where it was previously.

Actually I would argue that having a common cookie container shared across requests with different threads is extremely dangerous due to cookies leaking between requests. It only worked previously on the existing RestClient as everyone used a new one per request. Removing it from the client and moving it to the RestRequest is the correct solution and while a breaking change, a necessary one to avoid security problems with cookie leakage across requests.

I suspect most folks never use it for 99% of REST requests, and in the few cases they do use it I expect they would be asking the same questions I have, which is how do we avoid cross contamination between requests. In other words, anyone already moved to v107 that has not complained about this is most likely not using cookies.

[alexeyzimarev]

So nothing in that scenario allows for that tokens to be dynamically managed on a per used basis which is needed for some API's in multi user environments like web apps.

That's correct, but not all the apps out there are user-facing apps. The OAuth2 token I used is the app token for machine-to-machine integration, where the current solution works.

Actually it was always on the client, which is another reason everyone (including myself) used a RestClient per request.

That basically forced HttpWebRequest to create a new instance of HttpMessageHandler, causing hanging connections and breaking the connection pooling. So, it was broken from the moment Microsoft re-implemented HttpWebRequest using HttpClient.

Actually I would argue that having a common cookie container shared across requests with different threads is extremely dangerous due to cookies leaking between requests.

"It depends" :) It works totally fine for machine-to-machine integration. But yes, it will hurt when used in user-facing apps, where each user has its own creds. No doubt.

[kendallb]

Related to cookies as mentioned in the other RFC, Microsoft says do not use them with HttpClient:

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0#cookies

Cookies
The pooled HttpMessageHandler instances results in CookieContainer objects being shared. Unanticipated CookieContainer object sharing often results in incorrect code. For apps that require cookies, consider either:

• Disabling automatic cookie handling
• Avoiding IHttpClientFactory

[alexeyzimarev]

Correct. It explicitly mentions the pooled handler. RestSharp doesn't use pooled handlers, if it is used as a singleton, as part of some typed API client, it works fine as cookies aren't shared outside of that client. That implies that we are talking about a single-user app or machine-to-machine API calls.

[kendallb]

Right, a single instance client won't work with shared cookies though for a web app, and we use cookies for some of our usages for RestClient (not for our API calls, but when using it for accessing third party web sites for scraping etc as we need to perform a login). So for that, a shared cookie container won't work :(

[alexeyzimarev]

That's right. What I mean by this issue is not to remove the cooking container but put a big warning to the docs. Also, remove the AddCookie implementation that adds a cookie to the container (maybe) and bring back the cookie parameter. The issue with the old cookie parameter was that it added cookies to HttpWebRequest and I think it used a cookie container. I need to look at that .NET issue to better understand how to pass cookies as headers. Plus, we need to ensure to properly get cookies from the response. Currently, I query the container for the request domain, so it doesn't return other cookies, but, as you rightfully mentioned, cookies for multiple users might be overriding each other and it will be madness at some point.

[kendallb]

Right. Sending the cookies in the header is easy, but then as you mentioned you also need to extract the cookies from the response and put them back into the jar.

[alexeyzimarev]

Right now, the client always uses a cookie container. If it is gone, we need to find a way to get the cookies.

[kendallb]

Right the cookie container is in RestClient and is attached to the HttpClientHandler of HttpClient. And even using HttpClientFactory, as Microsoft says you should not rely on it for cookies as those handlers get reused across requests and they do nothing to ensure the cookie container is not reused. I don't think you can change the container either after HttpClient has been created (or really after the handler has been created).

I think the only viable way to do this correctly with a single instance RestClient is to have a CookeContainer attached to the request object (provided by the caller if needed), and not RestClient. Then on each request, we manually serialize the cookies into the headers (pretty easy), and then when we process the response we manually parse the cookies from the response headers. That way it's all per-request and no cross contamination of cookies.

I spent a lot of time researching this a long time ago when we tried to change to using HttpClient for our web scraping code, and eventually gave up and instead layered it on top of RestClient. LOL. So now we are back at square one. As far as I can tell from googling what others have done, everyone using HttpClient is manually serializing and de-deserializing cookies from the headers.

[kendallb]

Updated cookie handling is fixed by this PR and does nothing else:

#1801

[kendallb]

Grrr, run into a major snag with this and using Cookies with RestSharp. As mentioned it's not possible to use cookies with RestSharp as it is currently set up in v107 as the cookies will get shared across requests (super bad). So the code I wrote works really well, EXCEPT when a request is actually a redirect as the cookies need to get picked up from the redirect and put into the cookie jar. Since redirect handling is all handled by HttpClient, the cookies get lost as they are no longer in the response headers when we get the final response back. It works only if HttpClient is handling the cookies.

The solution would be to process redirects in RestSharp itself, so that cookies would work but that is going to be quite a bit of work to make it correct. And the catch is I cannot just turn it off for a single request, because that is all configured on the HttpClient handler. So it either needs to be on or off completely for each client.

Should I try to add code to handle redirects internally in RestSharp to get cookies working?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.