Allow URLs with paths as base_url in ClientSession

[thewhaleking]

Is your feature request related to a problem?

Many sites have an API path in the URL. For example, Jira uses {jira_url}/rest/api/2/ as the base for its API requests.
The current implementation of the ClientSession only allows absolute URLs without a path. Thus, to make a number of Jira requests, you would do something like:

async with aiohttp.ClientSession(base_url="https://jira.com") as client:
    baz = await client.get("/rest/api/2/issue/ISSUE-1")
    bom = await client.get("/rest/api/2/project/ISSUE")

instead of the simpler

async with aiohttp.ClientSession(base_url="https://jira.com/rest/api/2") as client:
    baz = await client.get("/issue/ISSUE-1")
    bom = await client.get("/project/ISSUE")

This somewhat defeats the purpose of having the base_url in the first place.

Describe the solution you'd like

Allow using paths in the base_url, similar to how Requests and HTTPX do.

Describe alternatives you've considered

Just continue typing in the path or using variables with f-strings.

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[hhromic]

+1

I was also expecting base_url to allow for any base URL, not just absolute.
Moreover, I just tried using a URL object for base_url, and while the assertion doesn't trigger, it silently ignores the path part.

For example, this snippet makes a request to http://service:8080/orders/1 (wrongly) without any errors:

import asyncio
from aiohttp import ClientSession
from yarl import URL

async def test():
    base_url = URL("http://service:8080/api/v2")
    async with ClientSession(base_url=base_url) as session:
        async with session.get("/orders/1") as response:
            body = await response.text()

asyncio.run(main())

[buzanovn]

+1

This is the default behavior of urljoin from urllib.parse when the joined part starts with a slash, otherwise the url is joined as we want it, but path that does not start with slash is restricted by design here

aiohttp/aiohttp/client.py

Lines 355 to 361 in cc6dc0c

	def _build_url(self, str_or_url: StrOrURL) -> URL:
	url = URL(str_or_url)
	if self._base_url is None:
	return url
	else:
	assert not url.is_absolute() and url.path.startswith("/")
	return self._base_url.join(url)

I think that a removal of second assert condition makes the deal, and it does not violate the RFC as stated in base_url feature PR discussion #6129 (comment)

[bogdan-coman-uv]

+1
that would be really usefull

[andreaevooq]

+1

[AYMENJD]

Why is this not possible till now?

[Dreamsorcerer]

It would appear to be to get the feature merged in a timely manner without considering the extra edge cases:

We can improve the feature support in future
#6129 (comment)

#6129 (comment)

[mokko]

I also need base_url to include paths. In the meantime, can we document current behavior better? Do you want me to attempt an PR for the documentation?

[Dreamsorcerer]

Yep, that'd be great.

[mokko]

Thanks for taking my PR to improve the documentation.

Although perhaps obvious, let me just say: the workaround is not to use ClientSession's base_url, but to construct your urls yourself. In my case that leads to somewhat ugly code, but is simple enough to do.

[Dreamsorcerer]

So, from a discussion on yarl and reading the spec, I think the main complication for implementing this (and probably the reason it was skipped initially) is the behaviour of joining the URLs. This may also trip people up who are expecting URLs to be joined differently, as evidenced by comments above in this thread:

    base_url = URL("http://service:8080/api/v2")
    async with ClientSession(base_url=base_url) as session:
        async with session.get("/orders/1") as response:

The above code (if paths were supported) would make a request to http://service:8080/api/orders/1, but the user is probably expecting the request to go to http://service:8080/api/v2/orders/1 and should have used http://service:8080/api/v2/ as the base_url.
https://datatracker.ietf.org/doc/html/rfc3986.html#section-5.2.3

If this is the only issue, then maybe we should add support for paths only when they end with a /, to avoid user confusion.

[Cycloctane]

I would like to make this useful feature possible. But I am still not sure how we should handle the edge cases.
Other http client libraries like httpx automatically add the slash at the end of base url to make it works as mostly expected, while http client in C# follows the rfc and ignores the path fragement after the rightmost slash. I think either approach is acceptable as long as we document it clearly.

[Dreamsorcerer]

As mentioned above, I think it's confusing, so I'd be in favour of just rejecting any paths that don't end with a /. Just require the user to pass a path with a trailing slash and I think that should be good.