Add the ability to fetch data from lists

[csolisr]

• Use the /api/v1/accounts/:id/lists backend to get the list of accounts followed on a given list (this requires the permission read:lists)
• Backfill the accounts included in these lists
• Expose the option, either as from-lists or max-lists in the configuration

[nanos]

Thanks for the suggestion @csolisr

I must admit I don't think I quite understand what you are trying to achieve here.

Say, I got the id of a user, e.g. yourself, this endpoint will return the list of lists that I have added you to.

What would I want to back fill here?

[csolisr]

Sorry for the delay, what I meant here is to do the following:

• For a given user, check all of the lists
• For each list, check all the accounts it includes
• Backfill each account listed on each of those lists

[nanos]

Ah, got it now. Actually, someone brought up a similar request in #19 (and I think elsewhere, but I can't find it now).

I'll look at doing two things:

1. Backfill users on your lists
2. fetch replies to posts on your lists

[p37307]

First, I like this. Unless I am misunderstanding what this does, this could be "dangerous" for me, personally. There are a few lists I would like to backfill, but I assign everyone to a list based on primary subject matter so, it would backfill like crazy. If there were a way to configure which list ID to fetch... that would be great.

I don't know how many people use lists, but I know back in the early days of all the influx of Twitter users after Musk bought it, lists were being pushed as a way to curate our non-algorithmic experience.

Say you have 500 users in different lists, what action does it do every time it runs? Backfill and fetch those 500 users each time?

[nanos]

Say you have 500 users in different lists, what action does it do every time it runs? Backfill and fetch those 500 users each time?

No: the script remembers users it has backfilled, and won't re-backfill them. Especially with the backoff implemented in #124, I would judge the risk very low. What it would do is:

1. Get a list of the user's lists (actually a max of 99 - an arbitrary hardcoded limit, which may change)
2. For each of those 99 lists:
3. Get a list of the last 100 (or whatever the user has configured) posts in this list, and fetch new context for each post in that list according to the rules specified in ‘Rate-Limit’ FediFetcher  #123.
4. Get a list of the last 10 (or whatever the user has configured) accounts added to the list, filter those it has already processed in a previous run, and for the remaining fetch the latest posts.

As such, after the first run the impact should be negligible.

If you don't want users on your server to be able to use this, then you could always block the FediFetcher user agent from the /api/v1/lists endpoint, using robots.txt, nginx or your firewall: that would prevent FediFetcher from getting past step 1.

[p37307]

This is good to know, esp. point 3. Thanks for explaining that to me. I am the only user on my self-hosted server and your script makes my experience so much better–no blocking in robots.txt. My favorite feature is backfilling users. It's indispensable. I've used it since you first released it. Thanks for the tool and your continued work to make it better.

[nanos]

Let me know how you get on with this as well, especially if you use lists extensively. As I don't personally use them at all, I find it hard to predict exactly how well it'll work.

If need be it shouldn't be hard to limit it to a specific list of lists to process.

[p37307]

I will let you know. I am going to upgrade to this version and do the first run manually later today.