[Atlas] Never give up trying to replicate attachments

[rafaelcr]

We've recently had an issue reported in the Hiro API where a subdomain which was correctly registered as a name-update to BNS does not appear in our endpoint responses, redirecting the user to the Stacks registrar when querying for it.

Upon further investigation, I discovered that we did not receive the attachment message (/attachments/new RPC) that would've contained the updated zonefile with the new subdomain only in some of our API instances. In those cases, the API only registered the update transaction but never the subdomain.

Fortunately, the API instance that is currently live did receive that attachment and is displaying the name correctly, but we'd like to make sure we're covering all of our bases for processing attachments.

This brings me to my question:

Are attachments guaranteed to be transmitted by nodes? Or should we assume they are 'best effort' messages similar to mempool updates?

Thanks

[jcnelson]

Hey @rafaelcr,

Attachment replication is best-effort, because attachment data is not consensus-critical.

The node can be configured to track different contracts for attachment events, which are generated through (print ...) statements. By default, it tracks attachment events from .bns. Once these attachment events are logged, a separate subsystem -- the Atlas network (src/net/atlas) -- attempts to contact the peer's neighbors to see if any of them have the associated attachment data. Eventually, all nodes tracking attachments for the same contracts should receive the attachment. But, the node will only make ~50 attempts for an attachment until it gives up on the attachment altogether. So, periods of network unavailability or a dearth of neighbors with the attachment could prevent your node from receiving it.

The Atlas system could be made more reliable (e.g. the one in Stacks 1.0 never gave up, and we could make it so that this one never gives up either), but to answer your immediate question, the attachment replication system is very much best-effort, and there is no guarantee of replication like there is for blocks and microblocks.

[rafaelcr]

Thanks for the quick response @jcnelson 🤝 . That is very informative (cc @CharlieC3 )

Follow up: is there currently a way for us to request an attachment (or a zonefile) if we did not see it for some time after we receive a name-update?

[jcnelson]

Sure -- if you know the node that received it, you can query it with /v2/attachments/<attachment-20-byte-hash>

[rafaelcr]

Appreciate it @jcnelson , closing this as my question was answered

[jcnelson]

Going to reopen this. I think it's important to make sure that the Atlas state-machine always re-tries attachments (even if infrequently) in order to handle these kinds of corner cases.

A good acceptance test would be to see if we can instantiate a Stacks node, blow away its Atlas attachments, and verify that it's able to eventually re-download all of them.

[CharlieC3]

@jcnelson Thank you! Albeit not consensus-critical, missing attachments definitely impact the developer experience.

[pradel]

Hey all, as this is creating issues for a Sigle user, I was wondering if you have any idea of an ETA for this one?

[jcnelson]

Sometime after Stacks 2.1 ships, at the earliest.

[pradel]

Okay, thanks

[pradel]

It seems that some zonefile where lost in the past few days. We had many Sigle users that had working blog and are now unable to access them as the API is returning an empty zonefile entry.
Example: https://stacks-node-api.mainnet.stacks.co/v1/names/curator.btc

[zone117x]

More reports from Sigle related to this:

Hey all we have an increasing number of users at Sigle that seems to have issues with the API and zonefile attachements, they go to btc.us add the Gaia url in the input to link it to their name submit the tx and the zonefile field is still empty. Some affected users:

• https://stacks-node-api.mainnet.stacks.co/v1/names/sam-tlt.btc
• https://stacks-node-api.mainnet.stacks.co/v1/names/mendelmire.btc
  Some people managed to get it work but just resubmitting the same transaction again and again
  but for some that's not working

I know @lgalabru and @jbencin have been working on improving Atlas. Can this issue be moved out of the icebox and into development? For example these PRs are at least related:

• feat: Add Atlas configuration via Stacks.toml #3618
• chore: update atlas config #3612

[jbencin]

I've been looking over the Atlas code and I think this behavoir is controlled by unresolved_attachment_instances_expire_after in AtlasConfig. If you use the code from PR #3618, you can set this value in Stacks.toml up to a limit of u32::MAX, which is about 136 years. This should work as a temporary fix, at least for the next 135 years or so.

A better solution might be to attempt to download an attachment frequently at first, and retry with exponentially increasing delay after each failure, up to some maximum limit (like 24 hours), but never stop trying.

@CharlieC3 and @zone117x, can you try the proposed workaround and let me know if it works?

[pradel]

Hey all, just wanted to see if any progress has been made on that one as this is still affecting users on Sigle side.

[jbencin]

@pradel See my comment above. PR #3618 has added config options that should fix this, and has been approved and merged into develop, but has not yet made it into master or an official release. If you want to fix this now, you could try building it yourself by checking out the 2.4.0.0.0 tag and manually changing some default values:

Clone the project

git clone git@github.com:stacks-network/stacks-blockchain.git
cd stacks-blockchain
git checkout 2.4.0.0.0

Edit the default config

Open src/net/atlas/mod.rs and update the values in AtlasConfig to something like this:

        AtlasConfig {
            contracts,
            attachments_max_size: 1_048_576,
            max_uninstantiated_attachments: 10_000,
            uninstantiated_attachments_expire_after: 3_600,
            unresolved_attachment_instances_expire_after: u32::MAX,
            genesis_attachments: None,
        }

unresolved_attachment_instances_expire_after controls how long to look for attachments which have a corresponding hash on the blockchain. By setting it to the max value, we'll continue looking for over 100 years.

Build the stacks-node binary

cd testnet/stacks-node
cargo build --features monitoring_prom,slog_json --release

[314159265359879]

Is there any reason why users are currently experiencing this issue much more frequently? (not just related to subdomains but regular names too)

Bilal from btc.us found that out of the 28 successful transactions (between the 15th and 20th of Feb.) name-update only 6 show zone files on the API. Users use to have a chance at getting an update to stick but now they can send dozens without any change.

Example
Transaction 20 days ago to add lightning address to zone file was fine and worked.
Subsequent transactions had no effect
https://explorer.hiro.so/address/SP3WE8GKTG0MVP2NRV8C7CM3ZFFB3QRXZXMWHBJ1F?chain=mainnet

https://stacks-node-api.mainnet.stacks.co/v1/names/franek.btc/zonefile results in
{ "zonefile": "" }

Another one like it here https://stacks-node-api.mainnet.stacks.co/v1/names/cryptostonks.btc/zonefile several attempts from the user to get a zonefile added.

@jcnelson can this fix be included in the next version?

[314159265359879]

It doesn't get picked up by the api, 4 attempts in 2 days:
https://api.mainnet.hiro.so/v1/names/nebo.btc/zonefile
{ "zonefile": "" }

name-register
https://explorer.hiro.so/txid/0x274fdc78ac8210ee4a2488f0fbfe181fd2522096f81d70d5c29ae17d38863bb8?chain=mainnet
transaction 0 name-update
https://explorer.hiro.so/txid/0x4f1594fe0a842361f3025335cf42308b5771f75e403cb26e00a3e74a26fed4a4?chain=mainnet
transaction 1 name-update
https://explorer.hiro.so/txid/0x1a6f2058d40bad45ad4219ea6c837b306bd63e474c1738361ea988abcd0c8ca0?chain=mainnet
transaction 2 name-update
https://explorer.hiro.so/txid/0x470d21789e221164cc3eade7b114ab4a36e4ece1b383c6417fcb7bad69cb77b7?chain=mainnet

[nin-j]

i've been trying for days to update my domain at btc.us and the changes are never reflected. specifically, i'm trying to add a bitcoin address and redirect url
https://explorer.hiro.so/txid/0x71e9083c8dbe35b9793a1fc62108a62fbe9e7c06143da74aee53d2433d1ad20b?chain=mainnet
https://explorer.hiro.so/txid/0xef000472da40219e7d3e23631db1ab1aa2a4ec3d60b39c07e054920e9724c236?chain=mainnet

i've tried on my other domain names to do same update (add btc address and redirect url) and i don't know what i'm doing wrong. please help. thank you.

[slofslb]

domain error

https://explorer.hiro.so/txid/0x0b90a7afaf5107dd9fcfd72473b8d30f9e5c9eee144665536a4ec982827ed05b?chain=mainnet
https://explorer.hiro.so/txid/0x68c96edf6a800f9a579eae507812ce0b699fd3aeb1cfd032a41c5bbaccf7806d?chain=mainnet

[colinpower]

Hi, I experienced an error as well.

Tried with 2 txns to update my lightning wallet and btc wallet connected to my .btc address (colinpower.btc), but it did not work either time.

Appreciate your help looking into this! I'm very excited to use my .btc address.

[cryptocracy]

per tx 0xdccb0d81eb790cc07698ba2fcfe288cdf0dd0888e5febbf1a8f16e2bdc75dc8b
via btc.us I called name-update to add a profile uri record for citycoinstacker.btc to add
https://gaia.hiro.so/hub/1Q9PqNXToxSjZnHayN6oUxmdUKY5w1xWhe/profile.json

however after that tx confirmed,
I get https://api.mainnet.hiro.so/v1/names/citycoinstacker.btc/zonefile
and it returns:

{
  "zonefile": ""
}

As requested by the banner on btc.us, I'm adding my situation comment.

[sunnykinger]

per tx 0xdccb0d81eb790cc07698ba2fcfe288cdf0dd0888e5febbf1a8f16e2bdc75dc8b via btc.us I called name-update to add a profile uri record for citycoinstacker.btc to add https://gaia.hiro.so/hub/1Q9PqNXToxSjZnHayN6oUxmdUKY5w1xWhe/profile.json

however after that tx confirmed, I get https://api.mainnet.hiro.so/v1/names/citycoinstacker.btc/zonefile and it returns:

{
  "zonefile": ""
}

As requested by the banner on btc.us, I'm adding my situation comment.

is there any solution to this? I am facing similar issue

[dantrevino]

Some clarity on the future of BNS would be helpful. Does BNS have a Product Owner? If not, why not? Who is driving development? How is orange domains going to fit in? Where's the roadmap? I get that there are other priorities, but having no communication on the roadmap and ongoing open issues is really detrimental to a potential key part of the ecosystem.

[GinaAbrams]

Hey! Appreciate the flagging @dantrevino - all on the same team with the same desire to see BNS have more functionality. OD is absolutely a stakeholder in BNS and an update will be shared shortly in terms of the ongoing efforts there; nothing is happening without community participation. I'd also love to learn more about what you're building on BNS (and others on this thread too), would be great to have a public call or spaces where we can all meet soon; I'll take that as an action item on my end over the next few weeks! 🙂

[rafaelcr]

@314159265359879 do we know which Stacks node btc.us is using to broadcast its BNS transactions?

I'm in the process of getting familiar with the Atlas code to understand how the attachment flow works and I suspect there are other issues that could be getting in the way of attachment replication. For example, I believe AtlasConfig values for max_uninstantiated_attachments (defaults to 50,000) and uninstantiated_attachments_expire_after (defaults to 1 day) should also be set to higher values in the node that is receiving the name registration transactions (cc @jbencin ). There are no debug logs printed when attachments are evicted based on these configs so right now it's also difficult to know if this is the underlying issue.

Could they try this on the relevant Stacks node to see if UX improves? If they use Hiro nodes I can coordinate this change on our end ASAP to see if that works.

[314159265359879]

@rafaelcr I will forward this to someone who knows more.

[bilalanees98]

@rafaelcr btc.us uses @stacks/connect to broadcast all its transactions, so I'm not exactly sure what stacks node is in use.

Additionally we do not provide a coreNode property to our UserSession object in @stacks/connect (prompting a fallback to the default value). Which seems to be 'https://api.mainnet.hiro.so' according to this and this

Happy to coordinate on any experiments that need to be run.

[dantrevino]

@gina Abrams ***@***.***> my user case is related to wallet functionality. Ultimately I'm interested in how/where profile data will be found. Typically profile and bns are two different things, but we're *setting* some metadata related to nostr and lightning in BTC.us. So I would like to understand how/if profiles will be managed in the future? StackerDB? Gaia? Somewhere else? Are all update functions in the @stacks/profile lib? Or are updates required? Dan

…

On Thu, Apr 18, 2024, 12:03 PM Gina Abrams ***@***.***> wrote: Hey! Appreciate the flagging @dantrevino <https://github.com/dantrevino> - all on the same team with the same desire to see BNS have more functionality. OD is absolutely a stakeholder in BNS and an update will be shared shortly in terms of the ongoing efforts there; nothing is happening without community participation. I'd also love to learn more about what you're building on BNS (and others on this thread too), would be great to have a public call or spaces where we can all meet soon; I'll take that as an action item on my end over the next few weeks! 🙂 — Reply to this email directly, view it on GitHub <#3270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAB4OUBQL2MSZ3NVXZ6BIDY574HFAVCNFSM6AAAAAAQBPE57GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRUGU3TONZQGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[paradigma-cl]

Hey! Appreciate the flagging @dantrevino - all on the same team with the same desire to see BNS have more functionality. OD is absolutely a stakeholder in BNS and an update will be shared shortly in terms of the ongoing efforts there; nothing is happening without community participation. I'd also love to learn more about what you're building on BNS (and others on this thread too), would be great to have a public call or spaces where we can all meet soon; I'll take that as an action item on my end over the next few weeks! 🙂

At Paradigma we have launched https://ubid.app that empowers web applications with BNS names. It let's you use your BNS name as user name, and defining a password to login to a web2 app. It uses the OpenID Connect protocol defined as base standard for web apps in many countries and continents of the world for interoperability. The defined password never leaves the user browser making it one safest's solution on the world!

[paradigma-cl]

We use intensively the zonefile with the url definition that points to the profile.json file. Phillip.xck.app

[xxxJay123]

I have the problem too

My addy: SP3SMGGY6KE51A6JCTZ5DTVEJ6H154MKHDE9H8H8T

[jcnelson]

I think the fundamental issue here is that Atlas attachment replication itself is pretty anemic as built. Attachments are only ever posted to the node by the /v2/transactions upload interface. From there, nodes only pull attachments from one another; they don't push them. So, if you want your attachment to be widely replicated, you must not only push your attachment to a public node, but also (if you can), push it to multiple public nodes. If you push it to your local NAT'ed node, it'll never replicate.

Perhaps after Nakamoto ships, I'll have time to build Atlas v2.

[Blingman99]

Can someone here give a manual workaround to this problem until the Devs can work out a solution. Appreciate if anyone can drill this down so even a grandmother can do this.

COPILOT AI JUST SUGGESTED THIS SOLUTION:

Yes, you can manually update the zone file for your BNS domain and bypass the registrar by directly interacting with the blockchain. Here’s how you can do it:

Steps to Manually Update the Zone File:
Access Your Wallet: Open the cryptocurrency wallet that supports BNS and Stacks (STX) transactions.
Prepare the Zone File: Create or edit your zone file with the necessary records. Here’s an example of a basic zone file:
$ORIGIN yourdomain.btc.
_url IN TXT "https://targetwebsite.com"
_btc._addr IN TXT "1BAHK1Esvn6L1icZREB8SFqTy7bBSRDwaS"
_stx._addr IN TXT "SP000000000000000000002Q6VF78"

Sign the Transaction: Use your wallet to sign a transaction that includes the updated zone file. This typically involves creating a name-update transaction with the new zone file data.
Broadcast the Transaction: Submit the signed transaction to the Stacks network. This can be done through your wallet or a blockchain explorer that supports transaction broadcasting.
Tools and Resources:
Stacks Wallet: Ensure your wallet supports signing and broadcasting transactions for BNS domains.
Blockchain Explorer: Use a blockchain explorer like Stacks Explorer to broadcast your transaction if your wallet doesn’t support it directly.
Example Using Stacks CLI:
If you prefer using command-line tools, you can use the Stacks CLI to manually update the zone file:

Install Stacks CLI: Follow the instructions to install the Stacks CLI from the Stacks documentation.
Create Transaction: Use the CLI to create a name-update transaction with your updated zone file.
Sign and Broadcast: Sign the transaction with your private key and broadcast it to the network.
Important Considerations:
Ensure Accuracy: Double-check your zone file for any errors before broadcasting the transaction.
Transaction Fees: Make sure you have enough STX in your wallet to cover the transaction fees.
Would you like more detailed instructions on any of these steps or need help with something else related to BNS domains?

[hstove]

So, if you want your attachment to be widely replicated, you must not only push your attachment to a public node, but also (if you can), push it to multiple public nodes

While building Dots, I ran into this issue, and I was able to get pretty solidly consistent zonefile propagation by doing this. Instead of just broadcasting the tx to one URL, I basically did:

• Fetch https://api.hiro.so/v2/neighbors
• For each neighbor, broadcast the tx and the attachment to that /v2/broadcast endpoint (guessing that they're exposing RPC on 20443)

This is kinda slow but you can add some heuristics to cache and only send to some nodes. Some requests will fail (ie maybe that node doesn't have a public RPC port), which is fine. This is just a "take a bunch of shots and enough of them will hit" approach.

cc @Blingman99 for asking about a workaround