Posts not being bridged to bluesky

[TomCasavant]

Not a huge deal, just noticed that a couple of my posts the past few days haven't been bridged and I'm wondering if there's a reason that's happening that I'm missing

https://tomkahe.com/@tom/113268743282701901 this one from last night doesn't seem to be on my bluesky profile (https://bsky.app/profile/tom.tomkahe.com.ap.brid.gy), it has the green checkmark next to it on my bridgy profile (https://fed.brid.gy/ap/@tom@tomkahe.com)
Here's the log link to that one https://fed.brid.gy/log?path=%2Fqueue%2Fwebmention%2C%2Finbox&start_time=1728343863&key=https%3A%2F%2Ftomkahe.com%2Fusers%2Ftom%2Fstatuses%2F113268743282701901%2Factivity

And then this one from Saturday didn't bridge:
https://tomkahe.com/@tom/113257256370193417, also has the checkmark and here's the log link to that one https://fed.brid.gy/log?path=%2Fqueue%2Fwebmention%2C%2Finbox&start_time=1728169652&key=https%3A%2F%2Ftomkahe.com%2Fusers%2Ftom%2Fstatuses%2F113257256370193417%2Factivity

[snarfed]

Hey Tom, sorry for the trouble! That's odd indeed. The green checks mean they were successfully bridged as far as we can follow them, which I checked more logs and confirmed. Here are the ATProto records we committed and sent to the relay for those posts, below. Not sure why the relay or AppView might have rejected them; I'll keep looking.

https://tomkahe.com/@tom/113268743282701901
at://did:plc:glvkfxk3igeeybszoue3apxi/app.bsky.feed.post/3l5xhp5dmxgx2
bafyreigyeyoyee2tofbteaytu32woiqrqzhdvgo7kqptnzjh3jgbcary5a
seq 2420578
sent to the bsky.network relay at 2024-10-07 23:31:07.166 UTC

{
  "$type": "app.bsky.feed.post",
  "bridgyOriginalText": "<p>Wrote a quick blog post about making a python wrapper for the Ohio Department of Transportation's OHGO API (and built a bot that posts random images from traffic cams around Ohio, <span class=\"h-card\" translate=\"no\"><a href=\"https://tomkahe.com/@ohgo\" class=\"u-url mention\">@<span>ohgo</span></a></span>) </p><p><a href=\"https://tomcasavant.com/ohgo-wrapper/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\" translate=\"no\"><span class=\"invisible\">https://</span><span class=\"\">tomcasavant.com/ohgo-wrapper/</span><span class=\"invisible\"></span></a></p>",
  "bridgyOriginalUrl": "https://tomkahe.com/@tom/113268743282701901",
  "createdAt": "2024-10-07T23:31:01.000Z",
  "facets": [{
    "$type": "app.bsky.richtext.facet",
    "features": [
      {
        "$type": "app.bsky.richtext.facet#link",
        "uri": "https://tomcasavant.com/ohgo-wrapper/"
      }
    ],
    "index": {
      "byteEnd": 225,
      "byteStart": 188
    }
  }],
  "langs": ["en"],
  "text": "Wrote a quick blog post about making a python wrapper for the Ohio Department of Transportation's OHGO API (and built a bot that posts random images from traffic cams around Ohio, @ohgo)\n\nhttps://tomcasavant.com/ohgo-wrapper/"
}

https://tomkahe.com/@tom/113257256370193417
at://did:plc:glvkfxk3igeeybszoue3apxi/app.bsky.feed.post/3l5sftypenzx2
bafyreif3xbpbquke7l4ncxdpv5uujh5afw5rscjutwscl5ataywjyo2bwq
seq 2374958

{
  "$type": "app.bsky.feed.post",
  "bridgyOriginalText": "<p><a href=\"https://tomkahe.com/tags/CFB\" class=\"mention hashtag\" rel=\"tag\">#<span>CFB</span></a></p>",
  "bridgyOriginalUrl": "https://tomkahe.com/@tom/113257256370193417",
  "createdAt": "2024-10-05T22:49:44.000Z",
  "embed": {
    "$type": "app.bsky.embed.images",
    "images": [{
      "$type": "app.bsky.embed.images#image",
      "alt": "Image of Thanos with Vanderbilt logo overlayed on his head\n\n\"Dread it run from it. Destiny arrives all the same",
      "image": {
        "$type": "blob",
        "mimeType": "image/jpeg",
        "ref": {"/": "bafkreidpjhweqp64254kxprdef3nnxs2h22mqtke2paacqp72giibwqcrq"},
        "size": 64737
      }
    }]
  },
  "facets": [{
    "$type": "app.bsky.richtext.facet",
    "features": [
      {
        "$type": "app.bsky.richtext.facet#tag",
        "tag": "cfb"
      }
    ],
    "index": {
      "byteEnd": 4,
      "byteStart": 0
    }
  }],
  "langs": ["en"],
  "text": "#CFB"
}

[snarfed]

I checked both records in https://atproto.tools/lexicons , and they both validate against the app.bsky.feed.post lexicon.

[bruvellu]

Hey @snarfed

I have a similar problem, most of my posts are not being bridged to Bluesky. But in my case, these posts didn't get the green checkmark. Only three out of several made it through (these got checkmarks).

https://fed.brid.gy/ap/@bruvellu@biologists.social

It seems random, I'm wondering why this is happening. Do you have an idea?

Since it's different than what @TomCasavant describes, I'm happy to open a separate issue for it.

Cheers
Bruno

[snarfed]

@bruvellu looks like the missing posts are all reposts and replies? Bridgy Fed doesn't bridge reposts of or replies to users who aren't bridged themselves.

[bruvellu]

@snarfed Ah, I see! The two reposts that went through were indeed from bridged accounts. Now it makes sense, thanks!

[TomCasavant]

Here's several more from today on a different account (same thing with a checkmark but the post not visible from bluesky)

https://tomkahe.com/@surrender_index/113308811760580418

https://tomkahe.com/@surrender_index/113308734458340992

https://tomkahe.com/@surrender_index/113308692813345154

[snarfed]

Current theory: this is https://console.cloud.google.com/errors/detail/CO2g4eLG_tOkZg;service=atproto-hub;time=P1D;locations=global?project=bridgy-federated , ie the second assert here that we hit occasionally.

Really difficult to trace before, and I have no clue how to reproduce it yet, but it's my best guess so far.

[snarfed]

Oops, didn't mean to close this!

[snarfed]

Not so sure that the assert above is the cause here after all. When we hit it, we close the websocket connection, before we've emitted that sequence number's blocks.

Example: we asserted on at://did:plc:wfa35ubq3begiqqmny7wgnby/app.bsky.feed.post/3l6l6muo544q2, seq 2665497, at 2024-10-15 12:42:06.9 UTC. The relay reconnected at 19:42:07 with cursor 2665496, one behind the asserted seq. We emitted that previous commit again, then emitted the crashed one ok, and then continued on. The asserted commit got ingested fine, it's visible at https://bsky.app/profile/did:plc:wfa35ubq3begiqqmny7wgnby/post/3l6l6muo544q2

[snarfed]

Hit the assert again earlier this morning. I'm a bit surprised that snarfed/arroba@396016e didn't fix it. Maybe it fixed some of the causes, occurence seems maybe down, but it's intermittent, so it's hard to be sure.

Failing event was seq 2712825, at://did:plc:lekoi5r6xyt4ryvppjcyrfxe/app.bsky.feed.post/3l6pgpzqdiyx2, commit bafyreidt5kmspdlki44mryys2r6kabnni4tofpyxtsty4vxigbpb2hjzsa. Same thing happened as above, relay reconnected with cursor one before (2712824), we emitted this failing event fine, and we continued on. Hrm.

[snarfed]

Failed record is https://bsky.app/profile/did:plc:lekoi5r6xyt4ryvppjcyrfxe/post/3l6pgpzqdiyx2 , looks like a pretty normal post.

{
  "$type": "app.bsky.feed.post",
  "createdAt": "2024-10-17T12:17:36.110Z",
  "text": "\u7b2c\u4e8c\u6b21\uff1fTwitter\u306e\u4e71"
  "bridgyOriginalText": "<p><span>\u7b2c\u4e8c\u6b21\uff1fTwitter\u306e\u4e71</span></p>",
  "bridgyOriginalUrl": "https://novelskey.tarbin.net/notes/9zgtyxkey2",
}

[snarfed]

Aha, got it, we're losing our ndb context in the middle of reading blocks for a given commit: https://github.com/snarfed/arroba/blob/396016e1bf1f15cf7da6143418000ea010fd5a49/arroba/datastore_storage.py#L533

            except ContextError as e:
                logging.warning(f'lost ndb context! client may have disconnected? "{e}"')
                return

Should be straightforward to fix. Still probably not the root cause of the missing posts here though. 😕

[snarfed]

Oops, didn't mean to close this!

[snarfed]

Another example from yesterday, https://shelter.moe/@Yamakyu/113341674036160854 , reported here. Created in the local ATProto repo as at://did:plc:dfim3tpwsrnugnje3v443z6a/app.bsky.feed.post/3l6xu4g23ohu2, emitted it to the relay at 2024-10-20T20:38:39Z with seq 2850793, and https://atproto.tools/lexicons says the record is valid, so it should be visible at https://bsky.app/profile/Yamakyu.shelter.moe.ap.brid.gy/post/3l6xu4g23ohu2 , but it's not.

[snarfed]

Some of these, we're definitely emitting to the relay, but I suspect some we're not. And I suspect at least some of those are caused by snarfed/arroba#34.

[snarfed]

Made progress on snarfed/arroba#34, should hopefully stop us from missing posts and other records.

[sveken]

Unsure if this is related?.
However the account i replied to has federated, i can see the like went through however my reply did not.
https://social.sveken.au/@siris.bsky.social@bsky.brid.gy/113435390546051888
https://bsky.app/profile/siris.bsky.social/post/3labhvcftct2n

[Tamschi]

@sveken A link to your reply would be helpful, if possible.

[sveken]

@sveken A link to your reply would be helpful, if possible.
My bad I thought I had the right URL
https://social.sveken.au/@sveken/113437711096440184

[Tamschi]

@sveken A link to your reply would be helpful, if possible.

My bad I thought I had the right URL
https://social.sveken.au/@sveken/113437711096440184

This is a 'Quiet public' post. Bridgy Fed currently bridges only posts that are 'Public'.

We have some discussion about bridging 'Quiet public' replies here: #1036

[snarfed]

@TomCasavant have you seen this recur since Monday? If not, I'm tempted to close as fixed by snarfed/arroba#34 / snarfed/arroba@c6da148. 🤞🤞🤞

[TomCasavant]

From what I've checked the last few days it looks like everything is being bridged from all my accounts

[snarfed]

I'm not 100% sure, but I'm hopeful this is fixed. Tentatively closing. Feel free to reopen!

[TomCasavant]

Hm, everything from the last 20 minutes has this problem again https://fed.brid.gy/ap/@surrender_index@tomkahe.com

(https://tomkahe.com/@surrender_index/113499634496098429 and several others)

Edit: oh nevermind I assume that's this? snarfed/arroba#39

[snarfed]

Hey, yeah, sorry, had some downtime this morning! Should be catching up now hopefully.