Nondeterministic problems with private packages authentication with valid token

[kamilkazmierczakMtab]

I have weird issues with npm install that is called from size-limit-action
In 90% cases on first try I get
npm ERR! 401 Unauthorized - GET https://npm.pkg.github.com/download/..... - - unauthenticated: User cannot be authenticated with the token provided
It's always for private packages under same organization.
Token is 100% valid (It's not secrets.GITHUB_TOKEN because GITHUB_TOKEN can't access private packages) with all read and write access - even with token with absolutely all access issue occurs)

What is very surprising is that almost everytime on second try (re-run failed action) everything works fine.

I tried to live with it and almost everytime re-run failed action but sometimes I have bigger problems - e.g. I just updated my private package and of course it failed at first try but then on second attempt it failed after switching to main branch - so now I have no idea what to do to make it work.

I'm using latest version of size-limit-action (1.7.0) and I'm 100% sure that token is valid because I'm using it for npm install for many other workflows and everything works as expected so it must be problem with size-limit-action`

[kamilkazmierczakMtab]

Also after update of some dependencies it will always fail after switching to main branch so I have to manually update packages in main branch and only after that size-limit will not fail with 401 after switching to main