feat: a standard for accessing the transaction log

[roman-kashitsyn]

No description provided.

[sea-snake]

The Transaction record has a field kind, is this value limited to namespaced query/update method names only? Or is it also possible to have transactions for events that happened during for example a heartbeat.

The Transaction has an optional field with the same name as the kind value. I assume a response can be deserialized based on record { kind: text; } first and then based on this text value you know how to deserialize the whole response.

But what about services that want to show the whole log and don't have all the possible candid definitions? Without a candid definition for each possible response type, you can't deserialize the data to it's original value since a encoded response for example doesn't contain the original keys of a struct. Maybe the candid could be fetched by the service to resolve this (not sure if we can do this from an inter canister call). Or the transaction data could be a more generic format similar to the metadata in ICRC-1.

I'm not sure how a canister could implement certification for all transactions. If the transactions were grouped in blocks that are certified as seen in the ICP ledger, then the number of hashes would be significantly lower. Is certification something that should be part of ICRC-3 standard? Without certification, a web service would be required to make update calls.

[roman-kashitsyn]

Great questions, @sea-snake!

Our experience with the ICP ledger suggests that having a single API for accessing transactions from canisters and agents (e.g., from a Rosetta node) does not work well, partly because Candid is not a great encoding for certification.

So the WG agreed to split the log access into smaller specifications: one for canisters and another for agents. Canisters want structured data and don't care about certification. Agents want certification and don't care too much about Candid.

ICRC-3 is a specification for canisters.

The Transaction has an optional field with the same name as the kind value. I assume a response can be deserialized based on record { kind: text; } first and then based on this text value you know how to deserialize the whole response.

Not quite: you can decode the whole record at once. If your code does not know about some transaction type, the Candid decoder will omit this field, and your code won't be able to access it.

For example, if your code assumes this structure:

record {
  kind: text; icrc1_mint : opt ...; icrc1_burn : opt ...; icrc1_transfer : ```;
}

And the ledger gives you the following:

record { kind = "icrc2_approve"; icrc2_approve : opt record { ... } }

the decoder will succeed, and you'll get record { kind = "icrc2_approve"; icrc1_mint = null ; <other fields are also null> }.

I'm not sure how a canister could implement certification for all transactions.

ICRC-3 will not support certification; it's a canister-oriented API. Another specification based on CBOR encoding and representation-independent hashes will follow later. We will need such a spec to implement an efficient Rosetta node.

[plitzenberger]

The current architecture requires the service to move data to an archive before it runs out of memory. Wouldn't it be easier to have an orchestration canister that holds the information about which canisters hold which transaction ranges? Then, whenever the latest canister runs out of mem, a new one is created and set to the latest field on the orchestration canister. This way, the get_transactions request would always have the same interface independent of which range gets requested.

[plitzenberger]

The archived_transactions field feels a bit unrelated to me. I would expect this as a response of a meta data request. Or is this related to the specified requested range in some way?

[roman-kashitsyn]

Or is this related to the specified requested range in some way?

Yes, the ledger restricts entries in the archived transactions list to the range the client requested.

We didn't want to have a separate "metadata" endpoint that returns a range -> canister assignment because this API would suffer from race conditions: by the time you received the response, the assignment of the tail might have moved already.

[plitzenberger]

I am trying to understand why we have different interfaces for query transactions from the archive. This data should point to the suitable canister where I can call the get_transactions in the same way, IMO.

[roman-kashitsyn]

That's a possibility; this approach would allow for multi-stage archival.
The main problem with having nested get_transactions is that writing a client becomes harder: how many redirects would you want to follow?
With the current API, you won't have more than one redirect.

[dietersommer]

off-by-1 error: should probably be { /* transactions 0..1_999 */ } as only 2000 trx can be returned
analogous below for the remaining part of the example

[roman-kashitsyn]

I use Rust syntax for ranges here, which does not include the right bound: 0..2000 means [0, 2000) in standard math notation. We can also switch to the math notation to avoid confusion.

[dietersommer]

Ah, OK. I guess if we make it clear, it should be fine as well. But otherwise, it is confusing. I proposed a change to standard notation, feel free to reject it and make clear what kind of notation it is. Not sure what is less confusing to the majority of people.

[william-iclight]

We have a solution for this and have been running it for some time.

DRC202 is a standard for scalable storage of token transaction records. It supports multi-token storage, automatic scaling to create storage canisters (buckets), and automatic routing of query records.

• Automatic expansion ,the proxy contract uses BloomFilter
• Support AccountId/Principal/Txid/BlockIndex multiple ways to query transaction records

More information: https://github.com/iclighthouse/DRC_standards/blob/main/DRC202/DRC202.md