[bug] Custom emojis in profiles returned with no URL to cached images

[amdavidson]

Describe the bug with a clear and concise description of what the bug is.

Custom emojis in profiles are returned with blank "url" and "static_url" properties which prevents clients from rendering them.

What's your GoToSocial Version?

0.17.3+git-6f4cb2f

GoToSocial Arch

x86 Docker

What happened?

Loading a post from a user with a custom emoji will return the emoji with missing data for "url" and "static_url" preventing the emoji from being rendered properly.

...
  "account": {
...
    "emojis": [
          {
            "shortcode": "ablobcatmaracasevil",
            "url": "",
            "static_url": "",
            "visible_in_picker": true
          },
...

In some clients we see a graceful fallback to presenting the shortcode text, but in Ice Cubes it prevents timelines or profiles from loading at all (Dimillian/IceCubesApp#2202)

What you expected to happen?

"url" and "static_url" should be populated with urls that reference cached copies of the emoji.

How to reproduce it?

Querying the account of or any post by @catsalad@infosec.exchange on my instance will present this issue with their :ablobcatmaracasevil: profile emoji.

curl -H 'Content-Type: application/json' -H 'Authorization: Bearer XYZ' 'https://gts.amd.im/api/v1/statuses/01JEHQ79VG9ZRPVQ3ECR7W1ZA3'

Anything else we need to know?

The :ablobcatmaracasevil: emoji does have a record in the database but it is missing urls to a locally cached version. Dumped SQL record:

CREATE TABLE IF NOT EXISTS "emojis" ("id" CHAR(26) NOT NULL, "created_at" timestamptz NOT NULL DEFAULT current_timestamp, "updated_at" timestamptz NOT NULL DEFAULT current_timestamp, "shortcode" VARCHAR NOT NULL, "domain" VARCHAR, "image_remote_url" VARCHAR, "image_static_remote_url" VARCHAR, "image_url" VARCHAR, "image_static_url" VARCHAR, "image_path" VARCHAR NOT NULL, "image_static_path" VARCHAR NOT NULL, "image_content_type" VARCHAR NOT NULL, "image_static_content_type" VARCHAR NOT NULL, "image_file_size" INTEGER NOT NULL, "image_static_file_size" INTEGER NOT NULL, "image_updated_at" timestamptz NOT NULL DEFAULT current_timestamp, "disabled" BOOLEAN NOT NULL DEFAULT false, "uri" VARCHAR NOT NULL, "visible_in_picker" BOOLEAN NOT NULL DEFAULT true, "category_id" CHAR(26), cached BOOLEAN DEFAULT false, PRIMARY KEY ("id"), UNIQUE ("id"), UNIQUE ("uri"), CONSTRAINT "domainshortcode" UNIQUE ("shortcode", "domain"));
INSERT INTO emojis VALUES('01JCYVS07NA0AK9D203YHW639K','2024-11-18 05:26:24.245122+00:00','2024-12-07 22:54:56.03471+00:00','ablobcatmaracasevil','infosec.exchange','https://media.infosec.exchange/infosec.exchange/custom_emojis/images/000/170/718/original/8119c991add30c0f.gif',NULL,NULL,NULL,'','','','',0,0,'2024-11-18 05:26:24',0,'https://infosec.exchange/emojis/170718',1,NULL,0);

350 out of the 8367 emojis in my database are missing image_url and image_static_url. Their created dates vary and cover every day that my personal instance has been operational.

I am using a local folder for media storage in my docker-compose.yml and have gigabytes of free space so I do not believe there was a time when storage prevented downloading a cached copy.

volumes:
      - ./data:/gotosocial/storage

[amdavidson]

Alright, I ran delete from emojis where image_url is null and waited for new emojis to show up so I could catch the logs:

gotosocial  | timestamp="07/12/2024 21:21:38.106" func=dereferencing.(*Dereferencer).fetchEmojis level=WARN msg="partially loaded emoji: processEmojiSafely: error loading emoji ablobcatmaracasevil@infosec.exchange: store: error executing data function: DereferenceMedia: media body exceeds max size 100kiB"

We've got a big emoji problem. In these situations, should we just use the original emoji URL instead of caching? or should a record not be stored at all so that only the shortcode is rendered?

Emitting an unusable emoji object with blank URLs seems like a less desirable option.

[tsmethurst]

Oh wow thanks for doing this detailed investigation! <3 Indeed we should fix this.

I think there's a few different possible options for emojis we can't (or won't) cache locally, due to size or network outages or other issues:

1. Provide the url we would have used for the emoji had we cached it, and return 404 at that url if we still can't cache the emoji.
2. Provide the remote url, ie. hotlink the emoji.
3. Don't include emojis in the API response at all when we don't have a url for them.

I'm not sure yet which of the above is best. I'd steer away from number 2 since I don't really like hotlinking. That leaves 1 or 3.

The problem with 1 is that we don't necessarily know before decoding the emoji image what file extension and content type we'd be using for it, which makes estimating a probable path for it difficult.

The problem with 3 is that it precludes on-demand (re)cache attempts via attempting to access the image url (ie., what we do for media attachments), so instead of doing that we'd have to attempt to (re)cache all of a status's emojis in response to an API request for the status, rather than requests for each of the emoji image URLs. That's possible to do but requires a bit of jiggling.

I'm sort of leaning towards option 3 as a fix for this 🤔

@NyaaaWhatsUpDoc do you have any feelie weelies on the matter?

[tsmethurst]

Related to #3486 as it's basically the same issue/request.

[daenney]

I think we can treat media attachments and emojis differently.

In the case of a missing emoji, you still get the :text: in a client I believe, as long as we omit the emojis member? If that's the case, then no context is lost in that sense. Whereas for media that's potentially a problem since the media might be rather pertinent to what a post is trying to communicate.

I think trying to fetch the emoji and if it fails, we leave it at that until we for some reason refresh the status, ought to be good enough in practice? We can try fetching an emoji with the typical backoff on ingestion, but if we hit a size limitation we ought to give up entirely since that cannot be resolved without an admin changing the instance configuration.