New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modules/zstd: Add Repacketizer #1314

Closed

lpawelcz wants to merge 34 commits into google:main from antmicro:52954-repacketizer

Contributor

lpawelcz commented Feb 21, 2024

This PR adds Repacketizer proc. It is used at the end of the processing flow in the ZSTD decoder. It gathers the output of SequenceExecutor proc, removes the bits that are not marked as valid and creates new packets with payloads having only valid bits. Such packets are then send out to the output of the whole ZSTD decoder.

NOTE: this is based on #1295 , please ignore commits from that branch when reviewing.
This is part of #1211.

rw1nkler and others added 30 commits

January 15, 2024 15:43


          modules/zstd: Add buffer library

fef83f7

This commit adds a DSLX Buffer library that provides the Buffer struct,
and helper functions that can be used to operate on it. The Buffer
is meant to be a storage for data coming from the channel. It acts like
a FIFO, allowing data of any length to be put in or popped out of it.
Provided DSLX tests verify the correct behaviour of the library.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add Buffer use-case example

01f2dcd

This commit adds a simple test that shows, how one can use the Buffer
struct inside a Proc.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add library for parsing magic number

614f650

This commit adds the library with functions for parsing a magic number and
tests that verify its correctness.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add library for parsing frame header

9838e81

This commit adds the library with functions for parsing a frame header.
The provided tests verify the correcness of the library.

Internal-tag: [#49967]
Co-authored-by: Roman Dobrodii <rdobrodii@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/frame_header: Add benchmarking rules

09a7eba

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          dependency_support/libzstd: Make zstd_errors.h public

4fbe593

Required for expected_status inference in C++ tests for ZSTD decoder
components

Internal-tag: [#53465]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          dependency_support: Add decodecorpus binary

f85661c

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add data generator library

5941efe

This commit adds a binary that calls decoding to generate data and loads
it into a vector of bytes.

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add zstd frame header tests

070089f

Internal-tag: [#50967]
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add common zstd definitions

05e4bf7

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add raw block decoder

80a9841

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd/raw_block_decoder: Add benchmarking rules

98cd055

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add rle block decoder

Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/rle_block_dec: Specify fifo depths for internal channels

c72e115

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/rle_block_decoder: Add benchmarking rules

80b5bc7

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block header parsing library

5ef1e1d

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add SequenceExecutorPacket to common definitions

a89ca51

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block data muxer library

3d15529

This commit adds DecoderMux Proc, which collects data from specialized
Raw, RLE, and Compressed Block decoders and re-sends them in the correct order.

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd/dec_mux: Add benchmarking rules

4552b37

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block demuxer library

327b2c1

This DSLX proc responsibility is to dispatch encoded blocks to a correct
decoder: RAW, RLE, COMPRESSED.

It tracks and assigns block IDs.
The ID counter is reset on the frame's last block on the last data packet.

Internal-tag: [#51736]
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Dudek <mdudek@antmicro.com>


          modules/zstd/dec_demux: Specify fifo depths for internal channels

a9d9717

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/dec_demux: Add benchmarking rules

34c711d

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block decoder module

43f3e6b

This adds a decoder of block data. It decodes block header and
demuxes remaining input data into one of specific block decoders
depending on the type of the parsed block. Then it muxes outputs
from those decoders into single output channel.

Internal-tag: [#51873]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/block_dec: Specify fifo depths for internal channels

8ee1391

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/block_dec: Add benchmarking rules

3b1da80

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          xls/moduels/common: Specify decoder output format

86c5d9a

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          examples/ram: Export internal RAM API to other modules

24f1ee2

This commit marks SimultaneousReadWriteBehavior enum and
num_partitions function as public to allow for creating
simpler tests that interact with RAM models.

Internal-tag: [#53241]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add Offset type to common zstd definitions

2486fae

Internal-tag: [#54705]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add RamPrinter Proc

0fc2f48

This commit adds RAM printer block usefull for debugging
HistoryBuffer inside SequenceExecutor.

Internal-tag: [#54705]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add SequenceExecutor Proc

bd07cc1

Add Proc responsible for handling ZSTD Sequence Execution step,
which is described in:
https://datatracker.ietf.org/doc/html/rfc8878#name-sequence-execution

Internal-tag: [#54705]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

rw1nkler and others added 4 commits

February 14, 2024 15:13


          modules/zstd: Add verilog generation and benchmarking rules for Seque…

a0bffc4

…nceExecutor

Internal-tag: [#54705]


          xls/modules/zstd: Add repacketizer

1c8035f

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          xls/modules/zstd/BUILD: Add repacketizer build rules

122edc5

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          xls/modules/zstd/BUILD: benchmark repacketizer

ba47228

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

lpawelcz added a commit to antmicro/xls that referenced this pull request


          [TEMP] Add Repacketizer

2071d9d

google#1314

xls/modules/zstd: Add repacketizer

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/BUILD: Add repacketizer build rules

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/BUILD: benchmark repacketizer

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/repacketizer: fix the ordering of data in reorganized packets

Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

lpawelcz mentioned this pull request

modules/zstd: Add ZstdDecoder top level proc #1315

Merged

lpawelcz added a commit to antmicro/xls that referenced this pull request


          [TEMP] Add Repacketizer

f2fade7

google#1314

xls/modules/zstd: Add repacketizer

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/BUILD: Add repacketizer build rules

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/BUILD: benchmark repacketizer

Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/repacketizer: fix the ordering of data in reorganized packets

Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

cdleary added the app label

Member

proppy commented Mar 29, 2024

should we close this and focus on reviewing #1315 ?

Contributor Author

lpawelcz commented Mar 29, 2024

Closing. The review will take place in #1315

lpawelcz closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app