Add code documentation for data loader (#42)

* Add code documentation for data loader

* Update the documentation for data loaders

apps/docs/src/content/docs/guides/tg-bot.md

@@ -212,6 +212,6 @@ Your Telegram bot is now set up and will monitor blockchain transactions and sen
 
 In this guide, you've learned how to create a Telegram bot that monitors blockchain transactions and sends human-readable alerts. By using the Loop Decoder library, you can easily set up a bot to track specific contract addresses and chains without needing in-depth knowledge of EVM transaction decoding.
 
-Let us know on Twitter ([@3loop_io](twitter.com/3loop_io)) if you encounter any problems or have any questions, we'd love to help you!
+Let us know on X/Twitter ([@3loop_io](https://x.com/3loop_io)) if you encounter any problems or have any questions, we'd love to help you!
 
 Happy coding!

apps/docs/src/content/docs/reference/data-loaders.md

@@ -0,0 +1,33 @@
+---
+title: Data Loaders
+description: Data Loaders used to fetch ABI and Contract Metadata required for decoding transactions
+sidebar:
+  order: 2
+---
+
+Data Loaders are mechanisms designed to retrieve the required ABI and Contract Metadata for transaction decoding. Internally, they automatically batch and cache requests when processing logs and traces in parallel. Users are encouraged to create a persistent store to cache data, thus avoiding unnecessary API requests. Additionally, the custom store should include all proprietary contracts.
+
+The `AbiStore` and `ContractMetadataStore` accept strategies that resolve data from third-party APIs when the data is missing from the store. Upon successful resolution, the data is cached in the store to prevent unnecessary API requests in the future.
+
+The Loop Decoder implements various optimizations to reduce the number of API requests made to third-party APIs. For instance, if the ABI of a contract is resolved from Etherscan, the ABI is then cached in the store. Subsequently, if the ABI is requested again, the store will return the cached ABI instead of making another API request.
+
+## Contract metadata
+
+Contract metadata is a collection of information about a contract, such as the contract's name, symbol, and decimals.
+
+Loop Decoder provides some strategies out of the box:
+
+- `ERC20RPCStrategyResolver` - resolves the contract metadata of an ERC20 token from the RPC
+- `NFTRPCStrategyResolver` - resolves the contract metadata of an NFT token (ERC721, ERC1155) from the RPC
+
+## ABI Strategies
+
+ABI strategies will receive the contract address, and event or function signature as input and would return the ABI as a stringified JSON. Loop Decoder provides some strategies out of the box:
+
+- `EtherscanStrategyResolver` - resolves the ABI from Etherscan
+- `SourcifyStrategyResolver` - resolves the ABI from Sourcify
+- `FourByteStrategyResolver` - resolves the ABI from 4byte.directory
+- `OpenchainStrategyResolver` - resolves the ABI from Openchain
+- `BlockscoutStrategyResolver` - resolves the ABI from Blockscout
+
+You can create your strategy by implementing the `GetContractABIStrategy` Effect RequestResolver.

apps/docs/src/content/docs/reference/data-store.md

@@ -0,0 +1,90 @@
+---
+title: Data Store
+description: Data Storages are simple APIs that implement a key-value store for persistent cache.
+sidebar:
+  order: 1
+---
+
+Loop decoder relays on two data stores for caching the ABI and contract metadata. The user of the library is free to choose which data layer to use depending on their environment and requirements.
+
+If your application is designed to decode a fixed subset of contracts, you can provide a hardcoded map of ABIs and contract metadata. For more flexible applications, you can use any persistent database. When running in a browser, you can implement a REST API to fetch the ABIs and contract metadata from a server.
+
+A store also requires a set of strategies that we cover separately in the [Data Loaders](../data-loaders) section.
+
+## AbiStore
+
+ABI Store Interface requires 2 methods `set` and `get` to store and retrieve the ABI of a contract. Optionally you can provide a batch get method `getMany` for further optimization. Because our API supports ABI fragments, the get method will receive both the contract address and the fragment signature.
+
+```typescript
+interface GetAbiParams {
+  chainID: number
+  address: string
+  event?: string | undefined // event signature
+  signature?: string | undefined // function signature
+}
+```
+
+The `set` method will receive a map of ABIs indexed by the contract address, event signature, and function signature. You can choose to store the data in the best format that fits your database.
+
+```typescript
+type Address = string
+type Signature = string
+type ABI = string // JSON stringified ABI
+
+interface ContractABI {
+  address?: Record<Address, ABI>
+  func?: Record<Signature, ABI>
+  event?: Record<Signature, ABI>
+}
+```
+
+The full interface looks as follows:
+
+```typescript
+interface AbiStore {
+  readonly strategies: Record<ChainOrDefault, readonly RequestResolver.RequestResolver<GetContractABIStrategy>[]>
+  readonly set: (value: ContractABI) => Effect.Effect<void, never>
+  readonly get: (arg: GetAbiParams) => Effect.Effect<string | null, never>
+  readonly getMany?: (arg: Array<GetAbiParams>) => Effect.Effect<Array<string | null>, never>
+}
+```
+
+## ContractMetadataStore
+
+Similar to the ABI Store, the Contract Metadata Store Interface requires 2 methods `set`, `get`, and optionally `getMany` to store and retrieve the contract metadata.
+
+The `get` method will receive the contract address and the chain ID as input.
+
+```typescript
+interface ContractMetaParams {
+  address: string
+  chainID: number
+}
+```
+
+And, the `set` method will be called with 2 pramaters, the key in the same format as the `get` method, and the metadata value.
+
+Contract metadata is a map of the following interface:
+
+```typescript
+export interface ContractData {
+  address: string
+  contractName: string
+  contractAddress: string
+  tokenSymbol: string
+  decimals?: number
+  type: ContractType
+  chainID: number
+}
+```
+
+The full interface looks as follows:
+
+```typescript
+interface ContractMetaStore {
+  readonly strategies: Record<ChainOrDefault, readonly RequestResolver.RequestResolver<GetContractMetaStrategy>[]>
+  readonly set: (arg: ContractMetaParams, value: ContractData) => Effect.Effect<void, never>
+  readonly get: (arg: ContractMetaParams) => Effect.Effect<ContractData | null, never>
+  readonly getMany?: (arg: Array<ContractMetaParams>) => Effect.Effect<Array<ContractData | null>, never>
+}
+```

packages/transaction-decoder/src/abi-loader.ts

@@ -1,4 +1,4 @@
-import { Context, Effect, Either, RequestResolver, Request, Array, Console } from 'effect'
+import { Context, Effect, Either, RequestResolver, Request, Array, pipe } from 'effect'
 import { ContractABI, GetContractABIStrategy } from './abi-strategy/request-model.js'
 
 export interface GetAbiParams {
@@ -33,38 +33,104 @@ function makeKey(key: AbiLoader) {
   return `abi::${key.chainID}:${key.address}:${key.event}:${key.signature}`
 }
 
+const getMany = (requests: Array<GetAbiParams>) =>
+  Effect.gen(function* () {
+    const { getMany, get } = yield* AbiStore
+
+    if (getMany != null) {
+      return yield* getMany(requests)
+    } else {
+      return yield* Effect.all(
+        requests.map(({ chainID, address, event, signature }) => get({ chainID, address, event, signature })),
+        {
+          concurrency: 'inherit',
+          batching: 'inherit',
+        },
+      )
+    }
+  })
+
+const setOnValue = (abi: ContractABI | null) =>
+  Effect.gen(function* () {
+    const { set } = yield* AbiStore
+    // NOTE: Now we ignore the null value, but we might want to store it to avoid pinging the same strategy again?
+    if (abi) yield* set(abi)
+  })
+
+const getBestMatch = (abi: ContractABI | null, request: AbiLoader) => {
+  if (abi == null) return null
+
+  const { address, event, signature } = request
+
+  let result: string | null = null
+
+  const addressmatch = abi.address?.[address]
+  if (addressmatch != null) {
+    result = addressmatch
+  }
+
+  const funcmatch = signature ? abi.func?.[signature] : null
+  if (result == null && funcmatch != null) {
+    result = `[${funcmatch}]`
+  }
+
+  const eventmatch = event ? abi.event?.[event] : null
+  if (result == null && eventmatch != null) {
+    result = `[${eventmatch}]`
+  }
+
+  return result
+}
+
+/**
+ * Data loader for contracts abi
+ *
+ * The AbiLoader is responsible for resolving contracts ABI. The loader loads the ABI
+ * for one Event or Function signature at a time. Because the same signature can result
+ * in multiple ABIs, the loader will prioritize fetching the ABI for the address first.
+ *
+ * We first attempt to load the metadata from the store. If the metadata is not found in
+ * the store, it falls back to user provided strategies for loading the metadata
+ * from external sources.
+ *
+ * **Strategies for Loading ABI**
+ *
+ * Users can provide external strategies that will be used to load the ABI
+ * from external sources. The strategies are executed sequentially until one of
+ * them resolves successfully. The Strategies are also implemented as
+ * Effect Requests. When a strategy resolves successfully, the result is stored in the store.
+ *
+ * Strategies are grouped by chainID, and a default scope. The default scope is intended
+ * for the APIs that are chain agnostic, such as 4byte.directory, while the chainID are API's
+ * that store verified ABIs for example Etherscan.
+ *
+ * **Request Deduplication**
+ *
+ * When decoding a transaction we will have multiple concurrent requests of the
+ * same function/event as we decode each trace and log in parallel.
+ *
+ * To optimize concurrent requests, the AbiLoader uses the RequestResolver
+ * to batch and cache requests. However, out-of-the-box, the RequestResolver does not
+ * perform request deduplication. To address this, we implement request deduplication
+ * inside the resolver's body. We use the `makeKey` function to generate a unique key
+ * for each request and group them by that key. We then load the ABI for the unique
+ * requests and resolve the pending requests in a group with the same result.
+ *
+ * Further improvements can be made by extra grouping by address, to avoid extra
+ * requests for each signature.
+ */
 const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: Array<AbiLoader>) =>
   Effect.gen(function* () {
     if (requests.length === 0) return
 
-    const abiStore = yield* AbiStore
+    const { strategies } = yield* AbiStore
     // NOTE: We can further optimize if we have match by Address by avoid extra requests for each signature
     // but might need to update the Loader public API
     const groups = Array.groupBy(requests, makeKey)
     const uniqueRequests = Object.values(groups).map((group) => group[0])
 
-    const getMany = (requests: Array<GetAbiParams>) => {
-      if (abiStore.getMany != null) {
-        return abiStore.getMany(requests)
-      } else {
-        return Effect.all(
-          requests.map(({ chainID, address, event, signature }) =>
-            abiStore.get({ chainID, address, event, signature }),
-          ),
-          {
-            concurrency: 'inherit',
-            batching: 'inherit',
-          },
-        )
-      }
-    }
-
-    const set = (abi: ContractABI | null) => {
-      // NOTE: Now we ignore the null value, but we might want to store it to avoid pinging the same strategy again?
-      return abi ? abiStore.set(abi) : Effect.succeed(null)
-    }
-
-    const [remaining, results] = yield* getMany(uniqueRequests).pipe(
+    const [remaining, results] = yield* pipe(
+      getMany(uniqueRequests),
       Effect.map(
         Array.partitionMap((resp, i) => {
           return resp == null ? Either.left(uniqueRequests[i]) : Either.right([uniqueRequests[i], resp] as const)
@@ -85,8 +151,6 @@ const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: Array<Ab
       },
     )
 
-    const strategies = abiStore.strategies
-
     // Load the ABI from the strategies
     const strategyResults = yield* Effect.forEach(remaining, ({ chainID, address, event, signature }) => {
       const strategyRequest = GetContractABIStrategy({
@@ -108,29 +172,12 @@ const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: Array<Ab
       strategyResults,
       (abi, i) => {
         const request = remaining[i]
-        const { address, event, signature } = request
-
-        let result: string | null = null
-
-        const addressmatch = abi?.address?.[address]
-        if (addressmatch != null) {
-          result = addressmatch
-        }
-
-        const funcmatch = signature ? abi?.func?.[signature] : null
-        if (result == null && funcmatch != null) {
-          result = `[${funcmatch}]`
-        }
-
-        const eventmatch = event ? abi?.event?.[event] : null
-        if (result == null && eventmatch != null) {
-          result = `[${eventmatch}]`
-        }
+        const result = getBestMatch(abi, request)
 
         const group = groups[makeKey(request)]
 
         return Effect.zipRight(
-          set(abi),
+          setOnValue(abi),
           Effect.forEach(group, (req) => Request.succeed(req, result), { discard: true }),
         )
       },