feat: Web3 RPC Handler (npm package) #1

Keyrxng · 2024-03-01T23:32:19Z

Resolves #3

My package is hosted at yarn @keyrxng/web3-rpc-racer which I'm using in the video where I had left off the now closed pr.

I've cleaned up the code
Now using tsc to export types properly
Cache refresh is configured when initializing
Removed ts-template skeleton
Few more tests

We should import the Chainlist repository as a submodule to stay in sync.

The URLs should be tested upon initialization (i.e. page load) via Promise.race

The class should rank sort based on latency.

The class should drop failed endpoints.

The class should have a wrapper method to make a network request using the optimal RPC endpoint, and to drop it/automatically rotate to the next best from the list if a network request fails.

Make a standalone and polymorphic (frontend and backend compatible) npm module.

Make sure to have unit tests in Jest.

All covered bar one

The wrapper method I haven't implemented as the spec has described it. The optimalRPC is outdated from call to call so getFastestRpcProvider() will retest either allNetworkUrls or just the successful cached rpcs, returning an updated provider which the invoking logic would handle making the call. This way the requested provider is always the quickest other than when calling getProvider() which will return the provider as-is.

the runtime rpcs are dropped as they fail for whatever reason so there is no need for rotation plus you always receive back the lowest latency provider.

Usage

const rpcHandler = RPCHandler.getInstance(networkId, 5);
return rpcHandler.getProvider();

just shows the bad rpcs being excluded until refresh

cache-refresh-every-5.mp4

Keyrxng · 2024-03-01T23:37:22Z

dont hold back @molecula451

0x4007 · 2024-03-01T23:42:44Z

.github/workflows/build.yml

@@ -28,8 +28,8 @@ jobs:
          yarn
          yarn build
        # env: # Set environment variables for the build
-          # SUPABASE_URL: "https://wfzpewmlyiozupulbuur.supabase.co"
-          # SUPABASE_ANON_KEY: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6IndmenBld21seWlvenVwdWxidXVyIiwicm9sZSI6ImFub24iLCJpYXQiOjE2OTU2NzQzMzksImV4cCI6MjAxMTI1MDMzOX0.SKIL3Q0NOBaMehH0ekFspwgcu3afp3Dl9EDzPqs1nKs"
+        # SUPABASE_URL: "https://wfzpewmlyiozupulbuur.supabase.co"


I guess it would be best not to publish these

0x4007 · 2024-03-01T23:43:42Z

build/esbuild-build.ts

-];
+import chainlist from "../lib/chainlist/constants/extraRpcs";
+
+const typescriptEntries = ["rpc-handler.ts", "constants.ts", "handler.ts", "services/RPCService.ts", "services/StorageService.ts"];


I'm surprised it passed kebab case test?

https://github.com/ubiquity/ts-template/tree/development/.github/workflows

ts-template doesn't have the kebab workflow in it by default, pay.ubq does tho. Just copy paste it over?

https://github.com/ubiquity/ts-template/tree/development/.github/workflows

ts-template doesn't have the kebab workflow in it by default, pay.ubq does tho. Just copy paste it over?

bring it over https://github.com/ubiquity/web3-rpc-racer/actions/runs/8123062862/job/22205701380?pr=1

0x4007 · 2024-03-01T23:45:18Z

dist/cjs/constants.d.ts

+  Anvil = 31337,
+}
+export declare enum Tokens {
+  DAI = "0x6b175474e89094c44da98b954eedeac495271d0f",


It would make more sense to associate tokens per network in case they aren't matching

Makes sense, should I add any more tokens in btw?

Not for now but I suppose this should be passed in as constructor arguments instead so that the application can handle this?

0x4007 · 2024-03-01T23:50:34Z

src/services/StorageService.ts

+export class StorageService {
+  static getLatencies(env: string): Record<string | number, number> {
+    if (env === "browser") {
+      return JSON.parse(localStorage.getItem("rpcLatencies") || "{}");


I don't think local storage modification should be within this module.

Instead the results should be stored on the object. The end user should manually handle local storage if it's necessary I think.

So it's both, I felt that would be better. The invoking logic doesn't need to do anything besides initialize and the rest is taken care of, it saves having to repeat code across multiple UI's and for node envs the state is still saved onto the object

what would you suggest, that I remove this and have each UI handle it's own storage?

I'd need to expose the runtimeRPCSs and latencies, then the UI would pull those from the object after init and then save them into local. Anytime getFastestProvider is called the UI will pull from local, pass them into the handler then save the output to local again?

You can consider decoupling the logic if you don't want each UI to handle it redundantly. Just have the localStorage logic in a separate method.

However I would argue that it may not be necessary to use localStorage at all. If we do a Promise.race once upon page load every time that's probably not bad and will accommodate for changing rpc conditions on a sporadic basis.

Since all the RPC endpoints fire off in parallel we shouldn't have any bottle necks initializing like this.

Like a boolean passed into init enabling local storage handling automatically then methods that just set/get local storage?

The only problem I think is, with every page refresh the class is re-initialized and because the class state of latencies is <= 1 then it'll always race all RPCs. On init I pull the latencies and cache refresh count from localStorage and with passing the refresh arg in on init the UI is able to maintain a cache, so I think local is needed unless I'm not seeing something

We already initialize like you say essence just with the cached rpcs (unless refreshing the list), I can't think how we'd persist state across page visits without using local

Yes I just think its not really necessary and adds complexity to making this polymorphic, or also being able to run in node.

rndquu

@Keyrxng Pls add a proper readme file which describes:

what the package does
how to install it
how to use it
how to run tests

Keyrxng · 2024-03-04T12:32:16Z

idk what the official name will be but I just went with rpc-handler

molecula451 · 2024-03-04T14:53:03Z

build still fails @Keyrxng

Keyrxng · 2024-03-04T15:08:19Z

build still fails @Keyrxng

Added the kebab scripts (forgot to when I added the workflow).

Deploying to cloudflare should be removed as there is nothing to deploy i.e no static folder which fails the run

and I think Knip is still not fully set up yet, is that right?

.github/workflows/scripts/kebab-case.sh

molecula451 · 2024-03-04T15:13:28Z

and I think Knip is still not fully set up yet, is that right?

yes you can skip/ignore knip

Keyrxng · 2024-03-04T15:36:08Z

Is the run failing maybe because the workflow isn't on the branch it's being merged into? I have no idea why it would fail when it's a 1:1 copy of the stuff on the pay.ubq repo

molecula451 · 2024-03-04T15:36:52Z

Is the run failing maybe because the workflow isn't on the branch it's being merged into? I have no idea why it would fail when it's a 1:1 copy of the stuff on the pay.ubq repo

you mean the build.yml?

Keyrxng · 2024-03-04T15:38:34Z

Is the run failing maybe because the workflow isn't on the branch it's being merged into? I have no idea why it would fail when it's a 1:1 copy of the stuff on the pay.ubq repo

you mean the build.yml?

The kebab case workflow, the debug run was a big help

molecula451 · 2024-03-04T15:51:17Z

The kebab case workflow, the debug run was a big help

yeah we are on a permissions issue: cc @gitcoindev

This is destructive and renames files so lets not use it.

Keyrxng · 2024-03-05T07:18:54Z

You are importing the unbuilt version, import from /dist/cjs it should work

rndquu · 2024-03-05T07:20:25Z

@Keyrxng

Trying to run this code:

import { RPCHandler } from "./src";

async function main() {
    const handler = new RPCHandler(100);    
}

main();

Getting this error:

rndquu@User-MBP web3-rpc-racer % npx tsx my.ts
/Users/rndquu/Public/projects/backend/web3-rpc-racer/src/constants.ts:78
  [networkIds.Mainnet]: ["https://rpc-pay.ubq.fi/v1/mainnet", ...(extraRpcs[networkIds.Mainnet] || [])],
                                                                  ^


ReferenceError: extraRpcs is not defined
    at extraRpcs (/Users/rndquu/Public/projects/backend/web3-rpc-racer/src/constants.ts:78:67)
    at Object.<anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/src/constants.ts:85:27)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Object.S (/Users/rndquu/Public/projects/backend/web3-rpc-racer/node_modules/tsx/dist/cjs/index.cjs:1:1292)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at <anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/src/index.ts:1:131)
    at Object.<anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/src/index.ts:16:28)

Node.js v20.11.1

What am I doing wrong?

rndquu · 2024-03-05T07:21:20Z

You are importing the unbuilt version, import from /dist/cjs it should work

Thanks, I'll try

molecula451 · 2024-03-05T07:22:37Z

What am I doing wrong?

extraRpcs it's part of chainlist submodule make sure you git recursively clone or init the module

rndquu · 2024-03-05T07:30:04Z

@Keyrxng

I've installed git submodules via git submodule update --init --recursive and rebuilt the project.

Trying to get RPCs for BNB chain via:

import { RPCHandler } from "./dist/cjs/rpc-handler";

async function main() {
    const handler = new RPCHandler(56);    
}

main();

Getting this error:

TypeError: Cannot read properties of undefined (reading '0')
    at new r (/Users/rndquu/Public/projects/backend/web3-rpc-racer/dist/cjs/rpc-handler.js:19:6622)
    at main (/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts:4:21)
    at <anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts:7:1)
    at Object.<anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts:7:6)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Object.S (/Users/rndquu/Public/projects/backend/web3-rpc-racer/node_modules/tsx/dist/cjs/index.cjs:1:1292)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Module._load (node:internal/modules/cjs/loader:1023:12)
    at cjsLoader (node:internal/modules/esm/translators:356:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:305:7)

What am I doing wrong?

rndquu · 2024-03-05T07:34:00Z

README.md

+- Re-test the cached RPCs
+
+```typescript
+const provider = handler.getFastestRpcProvider();


Suggested change

const provider = handler.getFastestRpcProvider();

const provider = await handler.getFastestRpcProvider();

rndquu · 2024-03-06T13:14:24Z

I assume you are just trying to run the script with tsx or something?

Yes

Now I'm getting the Module '"./dist/cjs/src/handler"' has no exported member 'HandlerConstructorConfig' error

How to reproduce:

Checkout to the latest commit from this PR
yarn install
git submodule update --init --recursive
yarn build
npx tsx my.ts

import { HandlerConstructorConfig } from "./dist/cjs/src/handler";
import { RPCHandler } from "./dist/cjs/src/rpc-handler";

const config: HandlerConstructorConfig = {
    autoStorage: true,
    cacheRefreshCycles: 5,
  };

async function main() {
    const networkId = 56;
    const handler = new RPCHandler(networkId, config);
    const provider = await handler.getFastestRpcProvider();
    console.log(provider);
}

main();

Keyrxng · 2024-03-07T13:46:58Z

Your screenshot made me realize two things:

type.d out was not right
I had added all the ids etc at the type level but no further

I've tested multiple times with fresh vsc instance, yarn, build, test and script and everything is good for me, it should be for you too

removed axios and just using built-in fetch
config is the only arg passed in now and the only mandatory prop is networkId
removed unused stuff like network explorer, currencies, tokens etc as they serve no purpose in racing the RPCs but a good thing to be exported from the package if it's going to be used across many repos
fixed the tests
found that it's the JsonRpcProvider that is returning the http:/\/localhost:8545"; so I've added the check back in

rndquu

If you build the project with yarn build then this line import { HandlerConstructorConfig } from "./dist/cjs/src/handler"; returns the Module '"./dist/cjs/src/handler"' has no exported member 'HandlerConstructorConfig' error. But if you build with yarn build:cjs there is no such error. Pls make sure that yarn build, yarn build:cjs and yarn build:esm are working the same way.
Getting this error:

rndquu@Users-MacBook-Pro web3-rpc-racer % npx tsx my.ts
node:internal/modules/cjs/loader:1144
  const err = new Error(message);
              ^

Error: Cannot find module './dist/cjs/src/rpc-handler'
Require stack:
- /Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts
    at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
    at a._resolveFilename (/Users/rndquu/Public/projects/backend/web3-rpc-racer/node_modules/tsx/dist/cjs/index.cjs:1:1729)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at <anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts:2:28)
    at Object.<anonymous> (/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts:16:6)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Object.S (/Users/rndquu/Public/projects/backend/web3-rpc-racer/node_modules/tsx/dist/cjs/index.cjs:1:1292)
    at Module.load (node:internal/modules/cjs/loader:1207:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/Users/rndquu/Public/projects/backend/web3-rpc-racer/my.ts' ]
}

Node.js v20.11.1

How to reproduce:

Checkout to the latest commit from this PR
yarn install
git submodule update --init --recursive
yarn build:cjs
npx tsx my.ts

import { HandlerConstructorConfig } from "./dist/cjs/src/handler";
import { RPCHandler } from "./dist/cjs/src/rpc-handler";

const config: HandlerConstructorConfig = {
    networkId: 100,
    autoStorage: false,
    cacheRefreshCycles: 5,
  };

async function main() {
    const handler = new RPCHandler(config);
    const provider = await handler.getFastestRpcProvider();
    console.log(provider);
}

main();

Keyrxng · 2024-03-10T10:13:37Z

Your build dir should look like this after running yarn build

yarn build is the only yarn command that should be run as postbuild handles the types

Vid below is me doing a fresh yarn > yarn build > npx tsx ./tests/script-test.ts

I do not understand why you are still getting this error @rndquu

0310.mp4

0x4007 · 2024-03-10T13:50:26Z

Just make a GitHub action to prove its not env related?

rndquu · 2024-03-11T07:04:59Z

@Keyrxng I haven't touched anything in the code since yesterday but today the build is performed without errors

Check this script:

import { HandlerConstructorConfig } from "../dist/cjs/src/handler";
import { RPCHandler } from "../dist/cjs/src/rpc-handler";

const config: HandlerConstructorConfig = {
  networkId: 314,
  autoStorage: false,
};

async function main() {
  const handler = new RPCHandler(config);
  const provider = await handler.getFastestRpcProvider();
  console.log(provider);
}

main();

It returns:

rndquu@User-MBP web3-rpc-racer % npx tsx tests/script-test.ts
[RPCService] Latencies object is empty
[RPCService] Failed to find fastest RPC

Although there are available RPCs for the Filecoin network:

Why is it happening?

Keyrxng · 2024-03-11T08:50:15Z

Happy days we are on the same page with it now tho good stuff

It's failing because I have the request timeout at 500ms, only three rpcs are returned for 314 that have data-tracking-none. Upping the timeout to 1500ms it succeeded, which raises a few questions doesn't it?

We want to keep the timeout low as in the sense that UBQ use it but it might be an idea for me to implement some sort of fallback where if none succeed then try again without the timeout? As we use 100 and 1 and they have lots of RPCS for both those chains the timeout should be kept in as I've seen calls take ^3s, so the timeout keeps things clean when there are lots of endpoints to test but in the case of there being very few endpoints or few good quality endpoints maybe it should be handled? Our use-case and the needs that UBQ have for it are covered anything else is more of a public service than anything I think is it not? :))

Let me know how to handle it and I'll push it today

$ npx tsx ./tests/script-test.ts
chainID 314:  filecoin
chain:  Filecoin
rpcss:  [
  'https://api.node.glif.io',
  'https://node.filutils.com/rpc/v1',
  'https://api.chain.love/rpc/v1'
]
Trace: r {
  _isProvider: true,

Provider available
Block number: 3728033
{
  'https://node.filutils.com/rpc/v1_314': 789.8341999999999,
  'https://api.node.glif.io_314': 1249.8516,
  'https://api.chain.love/rpc/v1_314': 1176.7801
}

rndquu · 2024-03-11T09:04:27Z

It's failing because I have the request timeout at 500ms

that have data-tracking-none

You're right, ubiquity case is covered, all these settings (timeout, data-tracking-none) could be moved to config in a separate issue

rndquu

Works fine

Keyrxng · 2024-03-11T09:05:20Z

It's failing because I have the request timeout at 500ms
that have data-tracking-none

You're right, ubiquity case is covered, all these settings (timeout, data-tracking-none) could be moved to config in a separate issue

I was going to suggest just that, add timeout etc into config and leave it to the user

package.json

README.md

molecula451 · 2024-03-14T05:59:47Z

Tested, working good, it looks like https://gnosis-pokt.nodies.app it's making the race at this time of writing, the README can be improved further with all the stuff that was tested here, as it it only available as a NPM installation on the current readme

molecula451 · 2024-03-14T06:02:30Z

Code looks good and well structured, the package it's working well, there are maybe things to improve that will only become handful as the package it's use, mergin this, good stuff @Keyrxng

* feat: web3 rpc handler npm package * feat: web3 rpc handler npm package * chore: configurable, add kebab, remove dist * chore: readme * chore: add workflow scripts * chore: remove build workflow * Delete .github/workflows/scripts/kebabalize.sh This is destructive and renames files so lets not use it. * chore: improved race, tests and readme, removed ubq rpcs * chore: script permission * chore: type the constants remove bad else * fix: type out path to src * chore: type the promises, add ts config and jest workflow * chore: final touch ups * chore: export all from index * fix: readme * Update package.json * Update README.md --------- Co-authored-by: Keyrxng <106303466+Keyrxng@users.noreply.github.com> Co-authored-by: アレクサンダー.eth <4975670+pavlovcik@users.noreply.github.com>

fix: workflow cypress

Keyrxng added 2 commits March 1, 2024 22:59

feat: web3 rpc handler npm package

64eb690

feat: web3 rpc handler npm package

7b3c409

0x4007 reviewed Mar 2, 2024

View reviewed changes

chore: configurable, add kebab, remove dist

b86112f

molecula451 requested review from rndquu and gentlementlegen March 3, 2024 20:40

rndquu requested changes Mar 4, 2024

View reviewed changes

chore: readme

01265b0

chore: add workflow scripts

0e3acc5

molecula451 reviewed Mar 4, 2024

View reviewed changes

.github/workflows/scripts/kebab-case.sh Show resolved Hide resolved

chore: remove build workflow

bce0803

molecula451 requested a review from gitcoindev March 4, 2024 15:53

Delete .github/workflows/scripts/kebabalize.sh

4bf8716

This is destructive and renames files so lets not use it.

rndquu self-requested a review March 5, 2024 06:55

rndquu mentioned this pull request Mar 5, 2024

CI: setup npm auto publish #2

Closed

rndquu reviewed Mar 5, 2024

View reviewed changes

chore: final touch ups

2f58ec9

Keyrxng added 2 commits March 7, 2024 13:48

chore: export all from index

e412248

fix: readme

ab3842d

rndquu requested changes Mar 10, 2024

View reviewed changes

rndquu approved these changes Mar 11, 2024

View reviewed changes

gentlementlegen approved these changes Mar 14, 2024

View reviewed changes

gentlementlegen requested changes Mar 14, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

molecula451 approved these changes Mar 14, 2024

View reviewed changes

molecula451 linked an issue Mar 14, 2024 that may be closed by this pull request

Implement Dynamic RPC pick handler #3

Closed

0x4007 reviewed Mar 14, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

Update package.json

3e22813

molecula451 reviewed Mar 14, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README.md

4167a85

molecula451 merged commit d003d97 into ubiquity:web3-rpc-racer-npm-branch Mar 14, 2024
1 of 2 checks passed

molecula451 mentioned this pull request Mar 14, 2024

Implement Dynamic RPC pick handler #3

Closed

Keyrxng mentioned this pull request Mar 14, 2024

TypeError: counter.plus is not a function ubiquibot/comment-incentives#31

Closed

github-actions bot mentioned this pull request Mar 22, 2024

chore(development): release 1.0.0 #7

Merged

github-actions bot mentioned this pull request Jul 31, 2024

chore(main): release 1.1.0 #34

Merged

0x4007 pushed a commit that referenced this pull request Oct 10, 2024

Merge pull request #1 from Meniole/development

3f35a2d

fix: workflow cypress

	const provider = handler.getFastestRpcProvider();
	const provider = await handler.getFastestRpcProvider();

feat: Web3 RPC Handler (npm package) #1

feat: Web3 RPC Handler (npm package) #1

Conversation

Keyrxng commented Mar 1, 2024

Usage

just shows the bad rpcs being excluded until refresh

Keyrxng commented Mar 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rndquu left a comment

Choose a reason for hiding this comment

Keyrxng commented Mar 4, 2024

molecula451 commented Mar 4, 2024

Keyrxng commented Mar 4, 2024

molecula451 commented Mar 4, 2024 • edited Loading

Keyrxng commented Mar 4, 2024

molecula451 commented Mar 4, 2024

Keyrxng commented Mar 4, 2024

molecula451 commented Mar 4, 2024

Keyrxng commented Mar 5, 2024

rndquu commented Mar 5, 2024

rndquu commented Mar 5, 2024

molecula451 commented Mar 5, 2024

rndquu commented Mar 5, 2024

Choose a reason for hiding this comment

rndquu commented Mar 6, 2024

Keyrxng commented Mar 7, 2024

rndquu left a comment

Choose a reason for hiding this comment

Keyrxng commented Mar 10, 2024 • edited Loading

0x4007 commented Mar 10, 2024

rndquu commented Mar 11, 2024

Keyrxng commented Mar 11, 2024 • edited Loading

rndquu commented Mar 11, 2024

rndquu left a comment

Choose a reason for hiding this comment

Keyrxng commented Mar 11, 2024

molecula451 commented Mar 14, 2024

molecula451 commented Mar 14, 2024

molecula451 commented Mar 4, 2024 •

edited

Loading

Keyrxng commented Mar 10, 2024 •

edited

Loading

Keyrxng commented Mar 11, 2024 •

edited

Loading