continuation optimization #217

dajuguan · 2023-12-08T16:43:33Z

Presently, when zkwasm's instructions exceed 2 billion (as observed in zkGo), the generated trace table becomes too large to fit into memory. Moreover, the generation of the witness table consumes a considerable amount of time, taking, for instance, up to 7 hours for 2 billion instructions. This pull request aims to optimize several aspects:

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice, as in 1rd commit.
Introducing a tracer callback mechanism to dump tables periodically per compute_slice_capability function's output, as in the 2rd commit. Note that there is also a related pr in wasmi repo. Notably，wasm's maximum memory has been hard-coded to 64MB via LINEAR_MEMORY_MAX_PAGES , otherwise, the current implementation will incur OOM due to push_init_memory pushes all the memory into imtable.
Adding support for binary files as private inputs for scenarios involving large input sizes. The new arg is --private <filename>:file, as in the 3rd commit.
Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling, as in this commit
[WIF] reconstruct code

xgaozoyoe

small adjustments are needed.

crates/specs/src/lib.rs

crates/specs/src/state.rs

crates/zkwasm/src/test/test_rlp_slice.rs

xgaozoyoe · 2023-12-10T01:12:24Z

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

dajuguan · 2023-12-10T01:56:39Z

Presently, when zkwasm's instructions exceed 2 billion (as observed in zkGo), the generated trace table becomes too large to fit into memory. Moreover, the generation of the witness table consumes a considerable amount of time, taking, for instance, up to 7 hours for 2 billion instructions. This pull request aims to optimize several aspects:

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice, as in the current commit.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions. This approach resembles the methodology outlined in this commit.

Adding support for binary files as private inputs for scenarios involving large input sizes, mirroring the implementation in this commit.

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling, similar to the enhancements introduced in this commit

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

Marked the as WIPs.

dajuguan · 2023-12-10T02:09:33Z

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

Regarding comparing test_rlp_from_file with test_rlp_slice, test_rlp_from_file operates by extracting tables from Slice, reconstructing the circuit using the dumped file, and subsequently validating each slice's integrity via its respective mock_test, mirroring the process of test_rlp_slice.

Others are marked as WIP.

… writing witness table

…ss & fmt

junyu0312 · 2023-12-25T03:34:14Z

crates/cli/src/app_builder.rs

+                    context_output.clone(),
+                )?;
+
+                write_context_output(&context_out.lock().unwrap(), context_out_path)?;


replace context_out with context_output and remove context_out variable?

junyu0312 · 2023-12-25T03:43:41Z

crates/specs/src/imtable.rs

@@ -41,3 +76,931 @@ impl InitMemoryTable {
        self.0.get(&(ltype, offset))
    }
 }
+
+pub fn memory_event_of_step(event: &EventTableEntry) -> Vec<MemoryTableEntry> {


If it’s necessary, could you move the function to specs/src/mtable.rs? I’m curious why it was moved to the specs crate.

… writing witness table

…ss & fmt

chore: cont opt - dump slice to file and load

db8554a

xgaozoyoe reviewed Dec 10, 2023

View reviewed changes

dajuguan mentioned this pull request Dec 12, 2023

EthStorage Devs Meeting #60 Agenda ethstorage/pm#65

Closed

chore: cont-opt[2] enable dumping witness table periodically

2af9e83

dajuguan mentioned this pull request Dec 13, 2023

continuation optimization DelphinusLab/wasmi#46

Open

dajuguan added 3 commits December 14, 2023 16:37

chore: cont-op[3] add file type for private inputs

c45f18e

chore: cont_opt[4] caching lookup for wasmi, flexbuffer & pooling for…

cbd3cab

… writing witness table

chore: cont-op -- replace callback with trait for dumping slice witne…

a60c749

…ss & fmt

dajuguan mentioned this pull request Dec 19, 2023

EthStorage Devs Meeting #61 Agenda ethstorage/pm#66

Closed

junyu0312 reviewed Dec 25, 2023

View reviewed changes

junyu0312 added 16 commits January 8, 2024 07:10

update comment

c77cd1d

fix: use F instead of AssignedCell to avoid unwrap

d46090b

fix: fix termination state and imtable

bd1c40d

fix etable assignment

c095762

fix: fix constraints of etable

f99839a

refine image table

a6053b2

refine image table

dd9b812

fix hardcode

5205f31

add transpose for InitializationState

ae734ed

chore: refine image table assignment

3168fb0

fmt code

66b16d7

set the number of frame table entry always 2

5970057

fix code style

945c29d

fix comment

664ab04

add name for post image table col

4f8abe6

fix image table assigner

cbc68c5

junyu0312 and others added 13 commits January 8, 2024 07:16

fix circuit_without_witness

37beae0

fix rlp_slice test

829a66b

fix compiling error

57adf94

fix code style

0748504

fix: fix a bug of post image table

231db4b

refine etable assignment

aaedece

decrease the degree of post init memory lookup

31c8ed3

chore: cont opt - dump slice to file and load

a22fffb

chore: cont-opt[2] enable dumping witness table periodically

fdafec0

chore: cont_opt[4] caching lookup for wasmi, flexbuffer & pooling for…

5434ced

… writing witness table

chore: cont-op -- replace callback with trait for dumping slice witne…

6addb1c

…ss & fmt

rebase cont_dev

6fbdd47

feat: add proof-from-trace cli

b0cacad

junyu0312 deleted the branch DelphinusLab:cont_dev April 26, 2024 04:04

junyu0312 closed this Apr 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

continuation optimization #217

continuation optimization #217

dajuguan commented Dec 8, 2023 •

edited

Loading

xgaozoyoe left a comment

xgaozoyoe commented Dec 10, 2023

dajuguan commented Dec 10, 2023

dajuguan commented Dec 10, 2023

junyu0312 Dec 25, 2023

junyu0312 Dec 25, 2023

continuation optimization #217

continuation optimization #217

Conversation

dajuguan commented Dec 8, 2023 • edited Loading

xgaozoyoe left a comment

Choose a reason for hiding this comment

xgaozoyoe commented Dec 10, 2023

dajuguan commented Dec 10, 2023

dajuguan commented Dec 10, 2023

junyu0312 Dec 25, 2023

Choose a reason for hiding this comment

junyu0312 Dec 25, 2023

Choose a reason for hiding this comment

dajuguan commented Dec 8, 2023 •

edited

Loading