Add 2 prefixes, and define them all

[mikedilger]

Here I'm proposing 2 new prefixes:

deleted - for EVENT, when the relay rejects the event because it has been deleted
redacted - for CLOSED, when some matching events were not served due to the requester not being authenticated or authorized.

I'm also defining what the prefixes mean, and what contexts they are used in (either EVENT or CLOSED or both).

[vitorpamplona]

I like this

[Semisol]

redacted has the side effect of closing the subscription, meaning new events aren't received if the client wishes to do nothing.

If the client acts on redacted, it should reopen the REQ.

[staab]

redacted has the side effect of closing the subscription, meaning new events aren't received if the client wishes to do nothing.

In this case I think a notice would make more sense. Redacted is going to be very common, it would be annoying to have subscriptions start flapping. I like deleted, since clients can follow up by requesting the delete to verify, remove the deleted event from their local database, and maybe pass the delete event along to the relay that served the deleted event.

[Semisol]

In this case I think a notice would make more sense. Redacted is going to be very common, it would be annoying to have subscriptions start flapping.

We may need better ways of communicating information to clients in general. NOTICE is too general.

I like deleted, since clients can follow up by requesting the delete to verify, remove the deleted event from their local database, and maybe pass the delete event along to the relay that served the deleted event.

Deleted is nice, agreed.

[vitorpamplona]

NOTICE is too general.

Kill notice. It is useless (only good for debugging purposes)

[Semisol]

Kill notice. It is useless (only good for debugging purposes)

And that is exactly why it is useful. If a relay has issues, or a client is hitting issues with a relay (malformed query, specific rate-limits, etc), additional details can be sent in a NOTICE.

[mikedilger]

redacted has the side effect of closing the subscription, meaning new events aren't received if the client wishes to do nothing.

If the client acts on redacted, it should reopen the REQ.

I was considering adding a <message> to EOSE just for this case.

[mikedilger]

redacted has the side effect of closing the subscription, meaning new events aren't received if the client wishes to do nothing.

In this case I think a notice would make more sense. Redacted is going to be very common, it would be annoying to have subscriptions start flapping. I like deleted, since clients can follow up by requesting the delete to verify, remove the deleted event from their local database, and maybe pass the delete event along to the relay that served the deleted event.

I'm a bit worried that clients who haven't updated to this PR (it always takes time for that to happen) won't understand redacted and may enter error handling code, either because they think the subscription errored or because they can't parse the machine-readable prefix. And what of it? Would they try again without comprehension? Or can we assume they will ignore what they don't understand?

I think redacted gives the client useful information. I've had multiple occasions where I was baffled that the event wasn't showing up... I was sure it was on the relay.. and it turned out to be redacted for one reason or another - my debugging effort would have been better directed if the relay gave me that hint.

[staab]

Yeah, I agree redacted is useful. But overloading EOSE or CLOSED will cause problems (overloading EOSE would prevent using redacted for ongoing subscriptions). A new REDACTED verb seems like overkill, but I can't think of where else to fit it in if we don't want to use NOTICE.

[fiatjaf]

["AUTH", "<challenge>", "<some message goes here letting the client or user know that performing auth is not mandatory but will affect the results of their queries>"]

[fiatjaf]

Also I think having clients send more granular subscriptions instead of trying to mash a bunch of unrelated stuff together in the same subscription because of arbitrary relay limits will go a long way, you can do CLOSED more freely for some subscriptions and fulfill others fully.

[Semisol]

["AUTH", "<challenge>", "<some message goes here letting the client or user know that performing auth is not mandatory but will affect the results of their queries>"]

This works.

[mleku]

NOTICE is too general.

Kill notice. It is useless (only good for debugging purposes)

you can never un-declare a thing in an open protocol, you can only deprecate it

but i'm of the opinion like @Semisol that notice definitely has debugging use case and extremely important especially for people developing relays to be able to enable this kind of information and for it to not be rejected

and i also think that clients should show a notice history in the relay information page, anyway, this would also help with relay debugging

[mleku]

Also I think having clients send more granular subscriptions instead of trying to mash a bunch of unrelated stuff together in the same subscription because of arbitrary relay limits will go a long way, you can do CLOSED more freely for some subscriptions and fulfill others fully.

i think if you split apart the filter structure and give them different names it would help

• fetching IDs is exclusive to all the rest, so why allow the whole mess at all
• search field is mostly not implemented except a few DVMs and such, mostly because laziness, but as a result most clients don't use it anyway... it should be a separate message
• what remains, pubkeys, tags and timestamps, is a filter

because these things are jumbled together in the spec it is possible to write some really retarded queries, and like, what is the semantics anyway when you have a search term and filter fields? this has to be an AND operation, surely

and the last point i'm gonna make is that results from filters and search queries should not return the whole event but only a list of matching event IDs, this way the client has to handle query state and they can selectively fetch those matches in whatever way suits them, and this eliminates the complexity of specifying a pagination scheme, you can do whatever fits your client's requirements, it could even be based on number of lines that render on a display, for example, which is a different iteration than asking for segments of 10

the entire REQ protocol should be split into these three parts and results as lists of event IDs be available, the easiest way to do this is start a new naming scheme that provides context as well so encoders don't have to be prompted, they just unpack it according to reading the message header - first string field.

[SilberWitch]

I second the ability to get a list of eventIDs as a result. Would make my life a lot easier. Important for OtherStuff implementations.

[staab]

I believe that can be accomplished today with negentropy.

[Semisol]

I believe that can be accomplished today with negentropy.

Negentropy is computationally expensive, complicated to implement compared to an IDs-only filter on both the relay and client, and amplifies the effects of network latency.

[Semisol]

You also cannot limit the amount of events you scan, which allows DoS, otherwise the client will think you only have those events.

[mikedilger]

["AUTH", "<challenge>", "<some message goes here letting the client or user know that performing auth is not mandatory but will affect the results of their queries>"]

This is unnecessary and doesn't distinguish the case when something was actually redacted versus when nothing was.

[Semisol]

["AUTH", "<challenge>", "<some message goes here letting the client or user know that performing auth is not mandatory but will affect the results of their queries>"]

This is unnecessary and doesn't distinguish the case when something was actually redacted versus when nothing was.

If any form of redaction is possible, the client should AUTH immediately, unless it is trying to access public content. Also, redacted gives you a side channel on REQs that had filtered events, and allowing you to narrow it down with more filters.

Public content in this context is any content that is trying to be read from a user's write relays in an outbox context, and things relating to that, such as reactions on read relays.

Unless it is explicitly signaled that certain content is expected to be public (such as listed as a write relay in outbox), it should be treated as possibly-AUTH-gated.

Also, I do not understand why AUTH is such a big concern:

• For your own read and write relays, you already lack any privacy, and should AUTH immediately.
• For outbox, you should be AUTHing immediately if you are trying to send DM events or similar restricted event types. Also rate-limits for gift wraps.
• For most outbox reads, you do not need to AUTH.
• If you get a rate-limit, you could prompt for AUTH.

There is only one case where you may want to prompt for it.

[mleku]

Also I think having clients send more granular subscriptions instead of trying to mash a bunch of unrelated stuff together in the same subscription because of arbitrary relay limits will go a long way, you can do CLOSED more freely for some subscriptions and fulfill others fully.

separating subscription from query would be a good start...

[mikedilger]

Today I'm not sure ["CLOSED", "redacted: ..."] is any more useful than ["CLOSED", "auth-required: ..."] and I'm leaning towards closing any subscription as soon as any event needs redacting, rather than giving partial results and silently or confusingly (in a new way that not all clients recognize) explaining such. Clients should all understand ["CLOSED", "auth-required: ..."] so that is safest.

The tangle of problems that arises going the other way is too much. Clients need to learn of 'redacted'. If a subscription stays open, EOSE needs a 'redacted' message too. Etc.

I'll push PR updates after discussion (if any).

[mikedilger]

I'm not going to use "redacted". I went ahead and used "auth-required" (after serving the partial results).

I still like "deleted" but not enough to push this PR forward.

[fiatjaf]

I still think we could use a way to notify clients they're missing events, but I'm not convinced "redacted" is the perfect solution. Or maybe this is too subjective and should be done more manually anyway.

[mikedilger]

Ok I'm reopening. @yukibtc suggested something similar nostrability/nostrability#167 (comment)

[vitorpamplona]

Question, couldn't we keep the sub open even with redacted? It could mean that new events might also be redacted.

[fiatjaf]

What about just adding a new message? ["REDACTED", "<subid>", "by not being authenticated you're missing some events"]

[mikedilger]

What about just adding a new message? ["REDACTED", "<subid>", "by not being authenticated you're missing some events"]

I think that would be more compatible than adding a prefix to EOSE.

[vitorpamplona]

Frankly, I think redacted should be the default relay behavior. So much that the notice doesn't need to exist. For clients, everything is redacted. We are always at the mercy of the relay. We make do with whatever comes down. The message helps to debug, but that's it.

It would be nice to understand how to up our user's permissions in the relay, if available, like:

["REDACTED", "<subid>", "Follow this key to get everything: npub.. "]
["REDACTED", "<subid>", "Auth to get everything: <challenge> "]
["REDACTED", "<subid>", "Pay this invoice to get everything: lnbc1.. "]

But maybe these types of "upgrades" should be on #901 instead. Because, in the end, it will be the user's action, not the client's.

[mikedilger]

Yes, what I care about is knowing if I can get more results by AUTHing, but also if there is some other way I'd want to know that too.