perf: reduce memory use

[joonazan]

What ❔

Enables improving peak memory use by computing circuits incrementally.

The API between this crate and the witness generator changes significantly. Instead of returning circuits, this crate now takes callbacks which it calls whenever a new circuit is ready. It could directly take a channel instead but I feel that is maybe not generic enough.

This PR incrementalizes just one type of circuit to make sure that it is possible.

Also removes unnecessary datastructures, clones etc. which improves memory use as well.

Why ❔

We have to improve memory use to be able to process larger batches on reasonable computers.

The new API allows emitting circuits as soon as possible instead of keeping around circuit inputs till the end and then building all of them. I measured that the vast majority of memory goes into storing the inputs of RAM circuits and MainVM circuits. Fixing those could solve all our memory trouble.

Even with every circuit type incrementalized, an array of all ClosedFormInputCompactFormWitnesses is collected. I don't know how hard it is to get rid of that or if it is big enough to be problematic.

[joonazan]

I get the same failures and timeouts when running the tests before and after this PR, so I guess it passes...

[shamatar]

We should rollout it on staging, generate witness for the full block, and basically compare them (those are serializable structures, so just compare bytes)

[joonazan]

@shamatar A note about comparing witnesses: simple diffing will indicate they have changed because the circuits are in a valid but different order. I used the following script to diff circuits.

import os, os.path, filecmp

def sorted_files(dir):
    return map(lambda x: os.path.join(dir, x), sorted(os.listdir(dir), key=file_order))

def file_order(filename):
    [a, b, c, *r] = os.path.basename(filename).split("_")
    return (int(a), int(c), int(b))

for a, b in zip(sorted_files("reference_artifacts/prover_jobs_fri"), sorted_files("artifacts/prover_jobs_fri")):
    if not filecmp.cmp(a, b):
        print("mismatch", a, b)

We'll do comprehensive testing next week. I'm unavailable tomorrow.

I am very confident that the output of this version is identical to 1.4.1 in all other respects.