feat: implement token lists management with fetcher and parser support #10

saledjenic · 2025-09-01T16:31:37Z

Added new package tokenlists for managing token lists.
Implemented tokensList struct for thread-safe state management.
Introduced refreshWorker for background updates of token lists.
Created fetchers for remote token lists and individual token lists.
Added parsers for CoinGecko, Uniswap and Status list token formats.
Included configuration options for token lists management.
Implemented validation for token lists against JSON schemas.
Added comprehensive tests for all new functionalities.

codecov-commenter · 2025-09-01T16:35:39Z

Codecov Report

❌ Patch coverage is 77.20000% with 228 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.45%. Comparing base (d334ec9) to head (bd11449).

Files with missing lines	Patch %	Lines
pkg/tokenlists/tokenslist.go	62.98%	89 Missing and 25 partials ⚠️
pkg/tokenlists/test_helper.go	60.95%	28 Missing and 13 partials ⚠️
pkg/tokenlists/refresh.go	75.34%	13 Missing and 5 partials ⚠️
pkg/tokenlists/validate.go	79.77%	14 Missing and 4 partials ⚠️
pkg/tokenlists/httpclient.go	78.94%	8 Missing and 4 partials ⚠️
pkg/tokenlists/types.go	57.69%	11 Missing ⚠️
...g/tokenlists/remote_list_of_token_lists_fetcher.go	82.69%	6 Missing and 3 partials ⚠️
pkg/tokenlists/fetcher.go	93.47%	1 Missing and 2 partials ⚠️
pkg/tokenlists/test_data_lists_of_token_lists.go	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master      #10       +/-   ##
===========================================
+ Coverage   23.24%   46.45%   +23.20%     
===========================================
  Files          22       39       +17     
  Lines        1325     2325     +1000     
===========================================
+ Hits          308     1080      +772     
- Misses       1014     1185      +171     
- Partials        3       60       +57

Files with missing lines	Coverage Δ
pkg/tokenlists/config.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/defaults.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/parser_coingecko_all_tokens.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/parser_standard.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/parser_status.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/remote_token_lists_fetcher.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/test_data_status_token_list.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/test_data_uniswap_token_list.go	`100.00% <100.00%> (ø)`
pkg/tokenlists/test_data_lists_of_token_lists.go	`80.00% <80.00%> (ø)`
pkg/tokenlists/fetcher.go	`93.47% <93.47%> (ø)`
... and 7 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dlipicar

Thanks for splitting the logic into files. I feel we could go one step further and create packages with self-contained single-responsibility functionality, that way these modules become reusable in a more granular level.
I haven't though too much about it, but I imagine something like:

A package in charge of fetching/parsing lists of token lists
A package in charge of fetching/parsing token lists (doesn't care whether they come from, could be downloaded or a local file selected by the user)
A package in charge of merging multiple token lists

The "Service" part that runs the timers and triggers periodical fetches. Hmm I've avoided putting such stuff in the wallet-sdk for now, and I haven't thought too much about it yet, but my general feeling is that something like that belongs more in the consumer (ie status-go) than the wallet-sdk. Anyway, it's a discussion I'd like to have!

dlipicar · 2025-09-11T20:46:39Z

pkg/tokenlists/config.go

+	return c
+}
+
+func (c *Config) WithCommunityTokenStore(store CommunityTokenStore) *Config {


haven't finished reviewing, but this is a red flag. The concept of a community is something status-specific, it most likely doesn't belong in the wallet-sdk

The concept of a community is something status-specific

Yes it is, and the wallet-sdk is here to simplify Status' needs. Was thinking whether to keep it on the stautsgo side or not, but then moved it here and let tokenlists package to take care of everything token-related.

dlipicar · 2025-09-11T21:00:02Z

pkg/tokenlists/config.go

+
+	"go.uber.org/zap"
+)
+


the config struct should probably be defined here for clarity

It can be, but added it to types.go since all types for the package are there.

dlipicar · 2025-09-12T01:19:48Z

pkg/common/chainid.go

 package common

+const (
+	EthereumNativeCrossChainID = "eth-native"


hmm if this really needs to be in common, it should probably be in a separate file.
My impression is this is token-related, so it should be in some token-related package

Yes, can be moved there.

dlipicar · 2025-09-12T01:23:14Z

pkg/tokenlists/types.go

+	LogoURI      string             `json:"logoUri"`
+
+	CustomToken   bool           `json:"custom"`
+	CommunityData *CommunityData `json:"communityData,omitempty"`


yeah this definitely doesn't belong here. If Status wants community data for some specific token it should get it from some community service.

As said in the previous comment, was thinking which approach to go with, but then wallet-sdk is here mainly for our needs and is aware of custom tokens and community tokens and knows how to handle them if they are provided. If the client doesn't need them, there are default custom and community token storer that can be used. In that case the package behaves as a regular token manager.

but then wallet-sdk is here mainly for our needs

I think we should treat this as a general-purpose library (that’s what an SDK is for). Ideally it should stay flexible enough so it can also be used in the Market-Proxy or even in Logos core, without assumptions about Status-client specific needs.

as we claim in https://github.com/status-im/go-wallet-sdk/blob/master/docs/specs.md

Go Wallet SDK is a modular Go library intended to support the development of multi‑chain cryptocurrency wallets and blockchain applications. The SDK exposes self‑contained packages for common wallet tasks such as fetching account balances across many EVM chains and interacting with Ethereum JSON‑RPC.

saledjenic · 2025-09-15T07:22:00Z

I feel we could go one step further and create packages with self-contained single-responsibility functionality, that way these modules become reusable in a more granular level.

We can, ofc, if there is a need for that. That's how it is organized in stautsgo now.

saledjenic · 2025-09-15T17:39:05Z

@friofry @dlipicar as suggested, constants moved to the token package, community data type removed, examples added for the package.

For now, I kept it all in a single package and split the logic into related files. When it becomes a need, we can think of having a package for client, config, fetching, parsing and so on.

- Added new package `tokenlists` for managing token lists. - Implemented `tokensList` struct for thread-safe state management. - Introduced `refreshWorker` for background updates of token lists. - Created fetchers for remote token lists and individual token lists. - Added parsers for CoinGecko, Uniswap and Status list token formats. - Included configuration options for token lists management. - Implemented validation for token lists against JSON schemas. - Added comprehensive tests for all new functionalities.

friofry · 2025-09-18T09:51:51Z

examples/tokenlists-usage/main.go

+
+func printToken(token *tokenlists.Token) {
+	fmt.Printf("Token Key: %s:\n", token.Key())
+	fmt.Printf("  Name: %s (%s)\n", token.Name, token.Symbol)


I ran the example app

Shouldn't it display Arbitrum or Base Name instead of Ethereum (ETH)?

Token Key: 42161-0x707f635951193ddafbb40971a0fcaab8a6415160: Name: Status (SNT) Chain ID: 42161 Address: 0x707F635951193dDaFBB40971a0fCAAb8A6415160 Decimals: 18 Native: false Custom: false Logo: https://assets.coingecko.com/coins/images/779/thumb/status.png?1548610778 Token Key: 42161-0x0000000000000000000000000000000000000000: Name: Ethereum (ETH) Chain ID: 42161 Address: 0x0000000000000000000000000000000000000000 Decimals: 18 Native: true Custom: false Logo: https://raw.githubusercontent.com/trustwallet/assets/master/blockchains/ethereum/assets/0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2/logo.png Token Key: 8453-0x0000000000000000000000000000000000000000: Name: Ethereum (ETH) Chain ID: 8453 Address: 0x0000000000000000000000000000000000000000 Decimals: 18 Native: true Custom: false Logo: https://raw.githubusercontent.com/trustwallet/assets/master/blockchains/ethereum/assets/0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2/logo.png

You mean for Token Key: 42161-0x0000000000000000000000000000000000000000 a name should be Arbitrum Ethereum?
That's not how I did it. The name and symbol for ETH token are the same accross chains. That's also the case for tokens with the same cross-chain id.

friofry · 2025-09-18T09:55:14Z

pkg/tokenlists/tokenslist.go

+	// process last fetched main list
+	storedContent, err := tl.config.ContentStore.Get(tl.config.MainListID)
+	if err != nil {
+		tl.config.logger.Error("failed to get stored content for main list", zap.Error(err))


The error is printed when ContentStore.Get is run for the first time. This seems an expected situation.
Could you change it to a warning?

===================================== 2025-09-18T13:48:14.089+0400 ERROR tokenlists/tokenslist.go:308 failed to get stored content for main list {"error": "content not found"} github.com/status-im/go-wallet-sdk/pkg/tokenlists.(*tokensList).mergeMainList /Users/ricko/status/go-wallet-sdk/pkg/tokenlists/tokenslist.go:308 github.com/status-im/go-wallet-sdk/pkg/tokenlists.(*tokensList).buildState /Users/ricko/status/go-wallet-sdk/pkg/tokenlists/tokenslist.go:239 github.com/status-im/go-wallet-sdk/pkg/tokenlists.(*tokensList).Start /Users/ricko/status/go-wallet-sdk/pkg/tokenlists/tokenslist.go:53 main.main /Users/ricko/status/go-wallet-sdk/examples/tokenlists-usage/main.go:69 runtime.main /opt/homebrew/Cellar/go@1.24/1.24.7/libexec/src/runtime/proc.go:283

friofry · 2025-09-18T09:57:57Z

pkg/tokenlists/tokenslist.go

+	symbol := EthereumNativeSymbol
+	name := EthereumNativeName
+	logoURI := "https://raw.githubusercontent.com/trustwallet/assets/master/blockchains/ethereum/assets/0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2/logo.png"
+	if chainID == common.BSCMainnet || chainID == common.BSCTestnet {


Is it possible to retrieve the name programmatically instead of hardcoding it?

'Optimism', 'Arbitrum' and 'Base' are not supported here.

Not really, toke lists usually don't contain entries for native tokens.

friofry · 2025-09-18T10:01:47Z

pkg/tokenlists/constants.go

+	CustomTokenListID = "custom"
+
+	// #nosec G101
+	listOfTokenListsSchema = `{


It would be better if the schema was in a separate file

friofry · 2025-09-18T10:09:24Z

pkg/tokenlists/fetcher.go

+
+	for _, tokenList := range remoteListOfTokenLists.TokenLists {
+		wg.Add(1)
+		go func(ctx context.Context) {


Maybe pass both params as args?

go func(ctx context.Context, tl tokenList)

currently ctx - param, tl - captured

friofry · 2025-09-18T10:55:47Z

pkg/tokenlists/errors.go

+)
+
+var (
+	ErrConfigNotProvided                           = errors.New("config not provided")


There are lots of errors in tokenlists.go that are not listed here.

Should we declare here the errors that are part of the module interface and are returned to the developer using it?

friofry · 2025-09-18T11:00:47Z

pkg/tokenlists/tokenslist.go

+	currentState.tokenLists[tokenListID] = tokenList
+	for _, token := range tokenList.Tokens {
+		if _, exists := currentState.tokens[token.Key()]; !exists {
+			currentState.tokens[token.Key()] = token


This is where we can log duplicate tokens. I think this information might be useful

I wonder if we can have a priority of the tokenlists when we merging duplicates?

Token key is a unique thing, we process lists in a specific order which is defined in buildState(). Different lists may contain the same tokens, that's ok, but since we have a strict order of processing them when the token appears for the first time we store it, all the same tokens (same chain+address) that may appear later will be skipped and no need to log them.

friofry · 2025-09-18T11:19:06Z

pkg/tokenlists/types.go

+}
+
+// PrivacyGuard interface for checking privacy mode.
+type PrivacyGuard interface {


It's violation of the single responsibility principle: The token library should not be aware of the application's privacy policies.

Tight coupling: It creates a tight coupling between the application's business logic and the library.

Restricts reuse: Other applications may have different approaches to privacy.

I think there should be a general permissions mechanism for the entire go-wallet-sdk

after discussing in the direct messages:

I suggest remove the PrivacyGuard interface and add autoRefresh property to the manager. (it's part of the state, not an external dependency)

RefreshNow is misleading name because it works asynchronously. And it should be renamed to TriggerRefresh

Yes, that's what we discussed, naming is definitely misleading and having an interface for this simple thing is maybe too much.

friofry · 2025-09-18T11:24:09Z

pkg/tokenlists/httpclient.go

+	}
+	defer func() {
+		if err := resp.Body.Close(); err != nil {
+			log.Println(err.Error())


inconsistent logging

friofry · 2025-09-18T11:25:43Z

pkg/tokenlists/README.md

+        WithLogger(zap.NewNop())
+
+    // Create token list manager
+    tl, err := tokenlists.NewTokensList(config)


shouldnt it match the package name TokenLists?

saledjenic · 2025-09-19T14:45:22Z

@friofry I am reorganizing all this, splitting into simple packages.

saledjenic · 2025-09-23T15:55:17Z

Closing this one in favour of this one #20
It's what we have here, but split into standalone packages, which is more aligned with wallet-sdk policy.

saledjenic requested review from alaibe, friofry and dlipicar September 1, 2025 16:31

saledjenic force-pushed the feat/token-list branch from f3313c0 to b07c664 Compare September 1, 2025 16:34

saledjenic force-pushed the feat/token-list branch 6 times, most recently from 7c0e25d to 15d894a Compare September 8, 2025 09:25

saledjenic force-pushed the feat/token-list branch 2 times, most recently from 8753428 to a572056 Compare September 11, 2025 08:52

dlipicar reviewed Sep 12, 2025

View reviewed changes

saledjenic force-pushed the feat/token-list branch from a572056 to 52bb6d6 Compare September 12, 2025 06:51

saledjenic requested a review from a team as a code owner September 12, 2025 06:51

saledjenic mentioned this pull request Sep 12, 2025

feat!: a new token list approach - wallet sdk token lists integration status-im/status-go#6912

Open

saledjenic force-pushed the feat/token-list branch from 52bb6d6 to 79b8fc9 Compare September 15, 2025 17:33

saledjenic force-pushed the feat/token-list branch from 79b8fc9 to bd11449 Compare September 16, 2025 07:29

friofry requested changes Sep 18, 2025

View reviewed changes

saledjenic closed this Sep 23, 2025

feat: implement token lists management with fetcher and parser support #10

feat: implement token lists management with fetcher and parser support #10

Uh oh!

Conversation

saledjenic commented Sep 1, 2025

Uh oh!

codecov-commenter commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dlipicar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saledjenic commented Sep 15, 2025

Uh oh!

saledjenic commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saledjenic commented Sep 19, 2025

Uh oh!

saledjenic commented Sep 23, 2025

Uh oh!

Uh oh!

codecov-commenter commented Sep 1, 2025 •

edited

Loading