Discuss migration to Crux 1.18 integration points

[refset]

Having spoken with @rschmukler previously, I understand crux-geo was developed by imitating the integration defined in the crux-lucene module, as it worked prior to the 1.18.0 release. The new version of crux-lucene brings the following changes, which will probably need to be factored into crux-geo at some point as you look to migrate:

1. Transaction processing - synchronous bus listening is now superseded by tx/register-index!
2. Eviction - the internal exclusive-avs API has been removed and crux-lucene no longer stores AVs independently. I don't know how to reconcile this with crux-geo's storage, but I think in the worst case users could simply avoid including geometry directly within any documents/entities that might be subject to eviction later (or users could rebuild their indexes to process evictions)
3. fsync-loop & checkpointing - these ensure consistency and the ability spin up nodes without having to replay every time
4. Index version check - this is a nice-to-have, and is, on very recent reflection (since the release!), something we are likely to incorporate into the protocol alongside index! and evict! in the near future

Separately, @rschmukler and I have also previously discussed the idea of secondary indexes being able to return sorted keys directly to the query engine to allow for lazy predicate processing and avoiding any potential re-hashing of keys. I don't have any news to report on this front unfortunately, but the crux-lucene integration has gone in the opposite direction and we now hope that users there can build the predicates they need without hooking into query internals at all (e.g. https://github.com/juxt/crux/blob/3e9f48b5ea9fa6815a5ef216ec65059db9219359/crux-lucene/test/crux/lucene/extension_test.clj#L56-L72). That's not to say something couldn't still be done in this space and we'd be happy to discuss it again sometime :)

[rschmukler]

@refset Thanks for creating this issue. I haven't yet had time to get up to speed with the new 1.18.0 APIs but it's been on my list for a while now. I'm currently coordinating a move but will likely take a go at migrating things sometime in September.

Edit: Also, you made me realize that I hadn't enabled Discussions for this repo. Moving it over to one!

[qwwwpp]

Hi, is there any progress on the migration?

[rschmukler]

Hey! I have made some progress but ran into an interesting bug that caused me to postpone. I believe the bug exists in the existing version as well so perhaps I'll just try and ship an updated version that is only the migration.