feat: refacto (part 1)

[bodinsamuel]

I would advice reading the 2 commits separately.
Initially wanted to do more but let's not push too much in a single PR.

Changes

• a8c37e9: add metrics with datadog
• d71c2dd: use async/await syntax everywhere

Testing

Launch the 3 different status (bootstrap, replicate, watch), by simply commenting each of the steps.

Deploy

• Datadog: it will need to be installed with this tutorial (never tried though)

What's next (probably)

• Split code and rename some function (for clarity)
• Improve CHANGELOG
  JsDelivr returns a list of files with stats, we could use that as a main way to fetch CHANGELOG and rollback to old method if nothing found (some packages are not bundled with it). That could fasten the process.
• Parallelise some stuff
  Saving to algolia and fetching new package for example

[Haroenv]

🙌 I love the effort you put in this. I haven't yet tried it out, but made some comments on the code! Looking forward to see where we can improve things once we have the metrics

[bodinsamuel]

@Haroenv I answered all your question and let some unresolved where you could still have comments.

• Upgraded async
• Added more metrics
• Fixed a bug in bootstrap
• Fixed emojis
• Fixed naming

[MartinKolarik]

Improve CHANGELOG
JsDelivr returns a list of files with stats, we could use that as a main way to fetch CHANGELOG and rollback to old method if nothing found (some packages are not bundled with it). That could fasten the process.

We also recently discussed with @Haroenv over email the possibility of using this file listing endpoint for typings discovery instead of the current unpkg method, so if the same listing request results could be used for both, the performance improvement should be even bigger.

[bodinsamuel]

@MartinKolarik Thanks a lot Martin, did not think about that 🙏

I'm not sure how to achieve everything in one single call though.
If I call:

• Stats: https://data.jsdelivr.com/v1/package/npm/algoliasearch/stats I won't have .d.ts/changelog because no one loads them.
• Structure: https://data.jsdelivr.com/v1/package/npm/algoliasearch@3.33.0/tree I won't have stats
• HTML: https://cdn.jsdelivr.net/npm/algoliasearch@3.33.0/ it has everything but I need to parse HTML :'(

Do you have any advice on this? ☺️

Note, we currently download the whole jsdelivr stats in one call https://data.jsdelivr.com/v1/stats/packages/month/all but it seems to fail pretty often and it has a huge memory footprint, I will maybe change that to a call per package, do you think it will be okay? :)

[MartinKolarik]

As for stats, I believe the best way is to keep loading them in one call. We added that endpoint specifically for npm-search as it's way more effective than making a request for every package. It should also be stable now - do you have any stats about how often you see it fail?

There is one easy improvement to make though - I just noticed the endpoint returns stats for both npm packages (about 70k records) and GitHub packages (950k records). Since you only use npm packages stats, we could probably add something like https://data.jsdelivr.com/v1/stats/packages/npm/month/all, to reduce the response size by more than 90%.

For typings and changelogs, you would use https://data.jsdelivr.com/v1/package/npm/algoliasearch@3.33.0 (this endpoint also gets faster over time because we now store all listings in a persistent database after generating them on the first request).

[bodinsamuel]

do you have any stats about how often you see it fail?

Yesterday I constantly had 524 coming from cloudflare, I thought it was the endpoint doing that but it does not seems to happen anymore 🙌

to reduce the response size by more than 90%.

That would be awesome 😱

For typings and changelogs, you would use https://data.jsdelivr.com/v1/package/npm/algoliasearch@3.33.0

Thanks a lot, I will put that in place in a next PR 😍

[MartinKolarik]

https://data.jsdelivr.com/v1/stats/packages/npm/month/all

71 MB -> 4 MB

Thanks a lot, I will put that in place in a next PR 😍

👍 feel free to ping me if you need anything.

[bodinsamuel]

trully amazing, thank you !!!

[Haroenv]

Note that we fetch changelog from GitHub because sometimes it isn’t inclues in the npm package, and we want to have a fallback there

[Haroenv]

looks good to me, a solid base to continue on. Which setup do we need to do on datadog to set it all up for production?