Project status

[jpbourgeon]

Hi

Sam, this is a very promising project. The goal is legitimate, the idea is simple and smart. In my opinion, it certainly deserves to become a viable alternative to the slow and bulky AWS SDK.

However, its status is unclear at this point. It generated some traction (94 stars), and at the same time, it hasn't merged any PR for two months now.

Could you clarify this project's purpose please? Is this a smart POC that will remain at the experimental level, or will it evolve into a production-ready community tool?

Anyway, keep up the good work 👏

[sam-goodwin]

Thanks for the feedback @jpbourgeon. I built this project as a POC with hopes of attracting collaborators to work with on fleshing out the rest of the services. I haven't had time to dedicate to it as I'm working on many other things right now. I'm still very interested in this project but need help to drive it forward.

[jpbourgeon]

That's good to read. I'm willing to help. But I need some guidance.

I created issues #22 and #23 so that every call to action in the repo are tracked.

What is the first action that you want to delegate ?

[sam-goodwin]

That's a good question - and thanks for helping! ❤️

I set up some very basic benchmarking here: https://github.com/sam-goodwin/itty-aws/tree/main/benchmark but it only really tests for cold start testing, which obviously this will do well at. A more important test would be sustained use - to test if performance degrades with usage due to poor connection management. I think that's likely the biggest risk right now - how well does the vanilla https/fetch libraries hold up against whatever AWS is doing?

Then, coverage would be next important. But, I don't think total coverage is important - there is definitely a long tail when it comes to usage in AWS. We should put together a short list of important services. I started this with some of the Issues. Services I personally use: CloudWatch, Scheduler, SNS, SQS, S3, DynamoDB, Event Bridge, Lambda, Step Functions, CloudFormation. But I've also heard requests for ECS, CodeBuild, EC2. I suspect there's a small group of services that can hit like 70% of use-cases?

[jpbourgeon]

This plan sounds good.
Let's focus on performance benchmarking for now, to ensure that we build the library on solid ground.
Then we will talk about the services priority list.
See you in #22

[jpbourgeon]

I compared the library with the AWS SDKs. Please review the results here and share your feedback.

I gained the following knowledge:

• itty-aws improves cold start function init duration by 50% and cold start function global duration by 25%.
• On warm starts, itty-aws is comparable to AWS SDKs.
• In most metrics, itty-aws is more stable and consistent across invocations and runtimes.
• During cold starts, the latency of itty-aws API calls is longer than its alternatives, which has a negative impact on the duration of the function. It seems that it is related to the time needed to establish the TCP connexion during HTTP requests, but it needs to be confirmed.

I recommend this next action:

• Try to improve API call latency. I suggest using a better HTTP library than node-fetch. undici is the package that powers the fetch function in Node.js >= 18.0. It has a reputation for being faster than node-fetch and is supported from Node.js >= 14.0. In particular, it supports connexion pooling and pipelining.
• Make itty-aws an event emitter to measure and compare HTTP request latency.

I think we have to talk about:

• Architect for growth: I recommend middleware architecture ala express.js like @aws-sdk. From this architecture, the core of the library will remain simple and all other aspects can be handled in the form of plugins: errors, adaptation of incompatible services (S3, etc.), ...
• There is more than the SDK. I'd like to experiment with a community runtime environment that provides NO AWS SDKs in the runtime environment.
• Monorepo refactoring to handle these changes: @itty-aws/sdk, @itty-aws/xxx-plugin, @itty-aws/xxx-runtime, etc.

[sam-goodwin]

Thanks for the detailed analysis - so good to have metrics and visualizations. Amazing work.

In most metrics, itty-aws is more stable and consistent across invocations and runtimes.

That is surprising!

During cold starts, the latency of itty-aws API calls is longer than its alternatives, which has a negative impact on the duration of the function. It seems that it is related to the time needed to establish the TCP connexion during HTTP requests, but it needs to be confirmed.

Interesting outcome here. I am not sure how this could be different between AWS's clients and ours. This seems important to dig into. Place to start would likely be the connection code deep in the AWS SDK. I've been down there before and didn't notice anything too strange, but I didn't look thoroughly.

Try to improve API call latency. I suggest using a better HTTP library than node-fetch. undici is the package that powers the fetch function in Node.js >= 18.0. It has a reputation for being faster than node-fetch and is supported from Node.js >= 14.0. In particular, it supports connexion pooling and pipelining.

I looked into undici because I was receiving errors in node-fetch and i discovered that undici is what drives node-fetch.

Problem with undici would be that it ties us to node? Would it also work in browser, deno, bun, cloudflare?

Have we tried using a custom HTTP agent? https://github.com/node-fetch/node-fetch#custom-agent

Architect for growth: I recommend middleware architecture ala express.js like @AWS-SDK. From this architecture, the core of the library will remain simple and all other aspects can be handled in the form of plugins: errors, adaptation of incompatible services (S3, etc.), ...

Are you suggesting a middleware layer for injecting interceptors into the request pipeline? I'm down for that, but i'm cruious what your use-case is?

There is more than the SDK. I'd like to experiment with a community runtime environment that provides NO AWS SDKs in the runtime environment.

Can you elaborate? Do you mean a custom AWS SigV4 signers or something else?

Monorepo refactoring to handle these changes: @itty-aws/sdk, @itty-aws/xxx-plugin, @itty-aws/xxx-runtime, etc.

Part of itty's value prop is the simplistic "itty" nature. But I am also down for mono repos if it makes sense - I use them for everything but I didn't see the value for itty. Can you expand on what the different packages would do?

[jpbourgeon]

I'm really excited about this project and can't wait to discuss these different subjects.
This conversation now deserves to be divided into different topics. @sam-goodwin, could you enable the discussion tab of the repository? To keep the issues clutter free, I'll move the content of this issue to respective discussion threads and answer there.

[sam-goodwin]

I enabled discussions. Sorry it took me so long!

[jpbourgeon]

Hi

I hope you are doing well. I posted different topics to continue our discussions. Maybe you can start with #26