Generalize/deduplicate async batch verification services / create a derive-able proc macro

[dconnolly]

We now have three async batch verifiers in zebra-consensus::primitives, and they won't be the last (Orchard will bring halo2::Verifier and redpallas::Verifier eventually). There is a lot of boilerplate being copy-pasted around, which may be ok for easy deletion, but when it's duplicated amongst ~5 variants, it's a signal to revisit where we can DRY this up or make a more generic async batch Verifier, which can then be instantiated with different Items/ItemWrappers and some call method/function.

Or we could create a derive-able proc macro: https://github.com/imbolc/rust-derive-macro-guide

[jvff]

I ended up looking at the code as part of working on #2371 (and investigating what became #2390), and I stumbled upon an idea. I'll post it as a comment here before I forget it and just in case it's useful in the future.

I thought about creating a trait that resembles something like:

pub trait BatchVerifier {
    type Item;
    type Error;

    const MAX_BATCH_SIZE: usize;
    const MAX_BATCH_LATENCY: usize;

    fn enqueue(&mut self, item: Self::Item) -> Result<(), Self::Error>;
    fn verify(&mut self) -> Result<(), Self::Error>;

    fn verify_single(item: Self::Item) -> Result<(), Self::Error>;
}

and also a wrapper type to have the actual implementation of the service:

pub struct Verifier<V>(V);

impl<V: BatchVerifier> Service<BatchControl<V::Item>> for Verifier<V> {
    // ...
}

The remaining piece would be to have the actual batch and fallback wrapping code. This could be done by having a single constructor exposed for the Verifier<V> wrapper type:

impl<V: BatchVerifier> Verifier<V> {
    pub fn new() -> impl Service<V::Item, Response = (), Error = BoxError> {
        Fallback::new(
            Batch::new(
                Verifier(V::default()),
                V::MAX_BATCH_SIZE,
                V::MAX_BATCH_LATENCY,
            ),
            tower::service_fn(|item| future::ready(V::verify_single(item))),
        )
    }
}

With this trait and this wrapper type, it's possible to only implement the BatchVerifier trait for each verifier, which should be simpler than reimplementing the service for every verifier.

Not actually sure any of this works, so more research will be needed in the future.

[teor2345]

const MAX_BATCH_SIZE: usize;
const MAX_BATCH_LATENCY: usize;

I think trait constants might have to be const generics, but they actually work in recent rust versions. Or they could just be functions.

[jvff]

Associated constants should work since Rust 1.20. Or do you mean it should be configurable when instantiated?

[teor2345]

Ah you're right, I just haven't used them with traits before:
https://doc.rust-lang.org/edition-guide/rust-2018/trait-system/associated-constants.html

[dconnolly]

Nice to have, but I think redundant now