Expire relations outside of a transaction

[doug-galvanick]

The question here is if there is a way within ddlog to expire data (delete relations) outside of a "window" (e.g. id < latest_id - 10000) or if that needs to be handled by an external application. Scenario follows.

We have some events being streamed. We alert on a simple detection. (This is cool, I love this.)

input stream EventA(
	id: bit<64>, // monotonically increasing across events
	value: bit<16>)

output stream AlertA(id: bit<64>)

AlertA(id) :- 
	EventA(id, value),
	(value > 255).

Next we have a more complex rule that relies on EventA and EventB whose occurrences might fall outside a single transaction. For this we create an intermediary relation DependencyA. Then if we see DependencyA and EventB matching some conditions, we alert.

relation DependencyA(
	id: bit<64>, 
	value: bit<16>)

DependencyA(id, value) :-
	EventA'(id, value),
	(value == 26).

input stream EventB(
	id: bit<64>, 
	value: bit<16>)

output stream AlertAB(
	idA: bit<64>, 
	idB: bit<64>)

AlertAB(idA, idB) :- 
	IntermediaryA(idA, _),
	EventB'(idB, valueB),
	(valueB == 10).

My interest is in deleting DependencyA when the latest event id grows large enough beyond it (think of a timestamp, and EventA and EventB must occur withing N seconds). I can handle this by adding a primary key and using delete_key in the Go code I am using for ingesting events and handling alerts. However, I wanted to check if there was a way to handle this within ddlog. Specifically -

1. Is it possible to query_index for a range? Something like query_index DependencyAIndex((id) (id < 1000)) with a lambda.
2. Is it possible to query_index or dump_index from the Go API?
3. Is it possible to delete from a relation?
4. Is it possible to delete or delete_key on a range of values?

Thanks so much!

[mihaibudiu]

One way to do it would be to create a view on DependencyA which only has recent events and use that instead.
But this may accumulate some garbage.
However, you can also use this view to tell you what to delete from the input relation: the negative changes in this view will be the keys that you have to delete.

[doug-galvanick]

Ah, I like this very much. Is a view a separate concept, or is this another name for an output relation?

[mihaibudiu]

Yes, an output relation is a really a view.
Let the system do the work for you as much as possible.

[doug-galvanick]

This is working as I expect it to.

input stream EventA(
	id: s64,     // monotonically increasing across events
	value: u16)

output relation AlertA(id: s64)

AlertA(id) :- 
	EventA'(id, value),
	(value > 255).

stream IDFold(
	event: string,
	id: s64
)

IDFold(event, id) :- IDFold-1(event, id).
IDFold(event, id) :- 
	EventA(id, _),
	var event = "A".

output relation MaxID(
	event: string,
	max: s64)

MaxID(event, max) :- 
	IDFold'(event, id),
	var max = id.group_by(event).max().

output relation DependencyA(
	id: s64, 
	value: u16)

DependencyA(id, value) :-
	EventA'(id, value),
	MaxID(event, max),
	(value == 26),
	(event == "A"),
	(id > (max-10)).

DependencyA(id, value) :-
	DependencyA-1(id, value),
	MaxID(event, max),
	(event == "A"),
	(id > (max-10)).

input stream EventB(
	id: s64, 
	value: u16)

output relation AlertAB(
	idA: s64, 
	idB: s64)

AlertAB(idA, idB) :- 
	DependencyA(idA, _),
	EventB'(idB, valueB),
	(idA < idB),
	(valueB == 10).

# AlertA fires
>> start;
>> insert EventA(0, 256);
>> commit dump_changes;
AlertA:
AlertA{.id = 0}: +1
MaxID:
MaxID{.event = "A", .max = 0}: +1

# AlertA drops
# DependencyA is created
# Max is updated
>> start;
>> insert EventA(1, 26);
>> commit dump_changes;
AlertA:
AlertA{.id = 0}: -1
DependencyA:
DependencyA{.id = 1, .value = 26}: +1
MaxID:
MaxID{.event = "A", .max = 0}: -1
MaxID{.event = "A", .max = 1}: +1

# No computation
>> start;
>> insert EventB(2, 100);
>> commit dump_changes;

# AlertAB fires
>> start;
>> insert EventB(3, 10);
>> commit dump_changes;
AlertAB:
AlertAB{.idA = 1, .idB = 3}: +1

# AlertAB drops
>> start;
>> insert EventA(4, 20);
>> commit dump_changes;
AlertAB:
AlertAB{.idA = 1, .idB = 3}: -1
MaxID:
MaxID{.event = "A", .max = 1}: -1
MaxID{.event = "A", .max = 4}: +1

# DependencyA drops
>> start;
>> insert EventA(14, 200);
>> commit dump_changes;
DependencyA:
DependencyA{.id = 1, .value = 26}: -1
MaxID:
MaxID{.event = "A", .max = 4}: -1
MaxID{.event = "A", .max = 14}: +1

# State looks good
>> dump;
AlertA:

AlertAB:

DependencyA:

MaxID:
MaxID{.event = "A", .max = 14}