HTTPSubscriber: check with If-None-Match or If-Modified-Since

[lidel]

There are two ways HTTP servers (incl. HTTP gateway responses for /ipns paths) may support update checks: via Etag or Last-Modified headers. We could improve HTTPSubscriber to maximize cache hits, and if Etag or Last-Modified are present in the response, store them, and use in follow-up update checks:

• if Etag was present, check for update with If-None-Match
• if Etag was not present, but Last-Modified was present, check for update with If-Modified-Since
  • (we prefer If-None-Match, because Etags are more deterministic than time-based check)

This is a bit nicer because if the content did not change since the last check, the response will be HTTP 304 Not Modified without any payload, rather than HTTP 5XX error, and will be produced by edge cache/cdn, rather than hitting the backend every time.

[hsanjuan]

In an append-only world, if we are requesting bytes from last-known-size and file has not changed, we will get a 0-byte payload anyways. I don't see the advantage vs. the complexity of dealing with it with more code.

[lidel]

I agree that append-only world is simpler, but it is not how things work today.

Right now, the only list ecosystem knows is NOT append only:
the list at https://badbits.dwebops.pub is sorted.

Everyone and their uncle is using rainbow with broken list.

Should we at very least:

1. add CI job to https://github.com/protocol/badbits.dwebops.pub to publish append-only version there (next to the sorted one)?
2. swap link on https://badbits.dwebops.pub to point at append-only version (we dont want to touch current file, because it is hardcoded in badbits-ingester and we really dont want to touch that legacy)

OR even less

1. Keep things as-is, and just add note to https://badbits.dwebops.pub or Rainbow docs to use append-only version instead?

?

[hsanjuan]

I think denyli.st is the easiest thing that people should use until someone worries enough about the official badbits to fix it in some way.

I don't want to support non-append lists that require trashing the db and reparsing the whole thing on every update. It is not sustainable, nor elegant apart from being a huge pain (how does that interact with other lists for example?).