Large catalog async optimizations

[matthewhanson]

I've been playing with using asyncio/httpx/aiofiles/aioboto3 to improve the speed of reading and writing of large catalogs (10+ million). Right now I'm just wrapping around PySTAC but think some of this could be incorporated into it.

Opening up this ticket to see if anyone else has been working with async functions here.

[TomAugspurger]

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/CLIENT_LIBRARY_DEVELOPER.md has some information on how Azure designed their libraries for working with multiple HTTP networking libraries (including sync and async). https://devblogs.microsoft.com/azure-sdk/custom-transport-in-python-sdk-an-httpx-experiment/ is an example plugging in httpx through that mechanism.

[gadomski]

Closing after some discussion at today's stac-utils working group meeting. It seems as though writing your own IO functions that produce PySTAC data structures is the way to go: One example of this pattern is this method in stac-asset.