Unstable sort when ordering by genesis_block_height

[jack-linden]

Reproduction:
https://api.hiro.so/ordinals/v1/inscriptions?address=bc1patm6f65h7kvppjss2p0fh4na67um30z7dd6txtqxjzmrwv0tz9ss47mdlc&limit=60&offset=0

Result set is not always sorted consistently - leading to pagination issues.

[chresko]

How should I debug this @rafaelcr?

[rafaelcr]

@jack-linden when I add the genesis_block_height sorting it looks correct to me. Can you specify the mistake you're seeing? https://api.hiro.so/ordinals/v1/inscriptions?address=bc1patm6f65h7kvppjss2p0fh4na67um30z7dd6txtqxjzmrwv0tz9ss47mdlc&limit=60&offset=0&order_by=genesis_block_height

[jack-linden]

@rafaelcr I think its actually a caching issue and not related to pagination/sorting. See the below TS playground

This code paginates through the result set and pops each inscription_id into a set. The expected response set size should be 265 but instead varies (run several times in new incognito browsers or clear browser cache before each run). If you flip the "bypass_cache" flag to true (passes a Date.now() query param to the URL) it will more often give the accurate 265 response set size.

EDIT - even with the bypass_cache flag it will sometimes give incorrect result set sizes.

https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAjgB2AfShAzmgexFxQG8AoZZAG2AFtgwAuZEAVzoCNoBuS5AjBglmrDtyh8qYAmDjUW7Lr345cbamFwt0WNRrABtALp8AvuXKhIsRCl3Y8B5BSrAAJi1xgooAOZSYspQiuIqVHDu7mrayN6+IAHkFuQIRN7InACeaHC4uJgIiAAWKAC8yD5sEHzkeVkgCMgwbI1gwETNEGAIxZjucHIAFILC3aHBAJQsAApQBAwkADwOaoTEEAB8LvxpxGDIa+nlqADucIxdPcVDAAbFYGBo2gD0L7oAdMXA8x+4BC8CFB3KB5LgXgA3ACML1AuAQvjQ7XSAH5ItE8LgypwEFDcmA6AA2GCEgCsxQA7ABrCFoNAAK3yACY0AAGGDFAAsIDghIpHAAzKyAF4UqKEsAADzAAEdJfThXQoKcIaywMKAJz5TkUujuagIABktAYYDKhNZhtGIjKABJSNbumYrcDoJhsmU-BAQHhgAVONQCAgqZhSsA-A97dlcvlCiUUCjkLdDe06BA7aQACKDCAfEAEU5DSZmW7IFgAcnLJcmgT2GQGch0GEc+GOyAqcHOlyOGw+jKIRcCODAbCgIGQDbg5ks9UazVaCGR49yflBkEwuiLs3miwgSwhBA821cyDrBxXEEwuGAwogEwk7eQFsC1G6AiEInv0EfrJfb5kcjUC2BixEoD4VL+uzpAccIIsASIdMQj4+qcyAAMrdEs8T+Jsg6WFQ7gEDsVBUGeE6DHATZ6Hg6wkI+nYXAcMDdL0-QUSMH7dDW-CkdBhxOJosSrAJRjGI+k4fPogm1CRzRAsgQxkVJByCPx6iCZMxGycgsGIkuuAfOiQzKR8HjcbJKSyQB8jAYJ4kUR81nUIEVCOgcADUFQXleN41PwZjIKc3yvgpbnIEslSyDZym4NxUHEAQr4fIGfhDOWECSmgECLhA7ingQrSiOWAA0kWAbZWileWBCcCQUAQrl+WFRWpW6fB+l-L5kzJJYK5rpem41kAA

[rafaelcr]

@jack-linden that behavior makes sense. The API is designed with "eventual consistency" in mind, which means that whenever we receive new data from a Bitcoin block we process it, store it in the DB, and immediately after run other background tasks that cache some results, stats, counts, etc. so we can respond to API calls faster. It's entirely possible that while you run the TS playground code a new block comes by or we finish one of these background tasks and that alters pagination results.

Also, the Bitcoin chain does re-org from time to time which means we occasionally need to delete and re-index some blocks to go back to the canonical chain tip. This would also alter pagination results.

[jack-linden]

I would understand if the user is actively receiving or sending inscriptions (i.e., new block anchors -> indexer running) while these queries execute but this address's holdings have not changed in over a week and therefore there is no reason why the result set should ever be inconsistent in this immediate scenario.

I retract that this is only related to caching.
The result set is always correct when sorting by "ordinal" so this further suggests that there is something unintended happening when sorting by "genesis_block_height" and paginating the result set.

Please see the latest example below - I've mapped the result set by inscription_id -> number of times returned while paginating the entire result set and printed each instance of the count > 1. If the sort is stable across subsequent query pages then all counts should == 1 (and no new blocks that would interact with the query).

https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAjgB2AfShAzmgexFxQG8AoZZAG2AFtgwAuZEAVzoCNoBuS5AjBglmrDtyh8qYAmDjUW7Lr345cbamFwt0WNRrABtALp8AvuXKhIsRCl3Y8B5BSrAAJi1xgooAOZSYspQiuIqVHDu7mrayN6+IAHkFuQIRN7InACeaHC4uJgIiAAWKAC8yD5sEHzkeVkgCMgwbI1gwETNEGAIxZjucHIAFILC3aHBAJQsAApQBAwkADwOaoTEEAB8LvxpxGDIa+nlqADucIxdPcVDAAbFYGBo2gD0L7oAdMXA8x+4BC8CFB3KB5LgXgA3ACML1AuAQvjQ7XSAH5ItE8LgypwEFDcmA6AA2GCEgCsxQA7ABrCFoNAAK3yACY0AAGGDFAAsIDghIpHAAzKyAF4UqKEsAADzAAEdJfThXQoKcIaywMKAJz5TkUujuagIABktAYYDKhNZhtGIjKABJSNbumYrcDoJhsmU-BAQHhgAVONQCAgqZhSsA-A97dlcvlCiUUCjkLdDe06BA7aQACKDCAfEAEU5DSZmW7IFgAcnLJcmgT2GQGch0GEc+GOyAqcHOlyOGw+jKIRcCODAbCgIGQDbg5ks9UazVaCGR49yflBkEwuiLs3miwgSwhBA821cyDrBxXEEwuGAwogEwk7eQFsC1G6AiEInv0EfrJfb5kcjUC2BixEoD4VL+uzpAccIIsASIdMQmB0Ogj4+qcyAALLoEs8T+AANEEEibIOlhUO4BA7FQVBnhOgxwE2eh4OsJCPp2FwHDA3S9P09EjB+3Q1vwNHQYcTiaLEqziUYxiPpOHz6BJtTUc0QLIEMtGKQcghieoEmTFRKnIMAMBDLBiJLgUKFoF8eRDFpHweJMBnmfBlnIegfzdPZ0mOe4hFmcQcEIekHk2V6YA+XpYB+QZAA+cXIKyBkANTIFCkzCdREDUKxrkhUh1leZFDkeIRGVZSkKkAfIwESXJ9EfDV1CBFQjoHClFQXleN41PwZjIKc3yvup7XIEslSyLVWm4EJUHEAQr4fIGfhDIYHwbfl7lFd6PjAHgRbGB8MDAJo0BDLtUBZO22yXVkhhQrJ2wZZlKQrmul6bjWQA

[rafaelcr]

Ok thanks, this is very helpful. I'll take a closer look at this sorting and get back to you @jack-linden

[chresko]

@rafaelcr Can I close this one? Thanks!

[rafaelcr]

@jack-linden this will be fixed on the next production deployment