feat(model): introduce ArchivalBinaryData #1321
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There are cases where we know we have binary data in input and we
want the output to be binary data using the dictionary encoding like
`{"format":"base64","data":"..."}`.
Such cases are, for example, DNS messages bodies.
There is no need for us to pass through the MaybeArchivalBinaryData
in such cases. Additionally, this makes MaybeArchivalBinaryData a bit
more complex than it probably needs to be.
What's more, ArchivalBinaryData is for when you do not require to
scrub the data. I want to introduce new data types that automatically
perform scrubbing when they're used.
But this puts even more pressure on MaybeArchivalBinaryData, and
hence this commit, to start differentiating between:
1. always binary data vs maybe binary data
2. no scrubbing requires vs scrubbing required
The rationale for doing this set of changes has been explained in
#1319.
The reference issue is ooni/probe#2531.
For now, this commit just adds the new type and tests for it
without using the type, which we'll do later once we have added
better marshal/unmarshal testing for the interested types.
bassosimone
commented
Sep 28, 2023
bassosimone
commented
Sep 28, 2023
bassosimone
commented
Sep 28, 2023
Murphy-OrangeMud
pushed a commit
to Murphy-OrangeMud/probe-cli
that referenced
this pull request
Feb 13, 2024
There are cases where we know we have binary data in input and we want
the output to be binary data using the dictionary encoding like
`{"format":"base64","data":"..."}`.
Such cases are, for example, DNS messages bodies.
There is no need for us to pass through the MaybeArchivalBinaryData in
such cases. Additionally, this makes MaybeArchivalBinaryData a bit more
complex than it probably needs to be.
What's more, ArchivalBinaryData is for when you do not require to scrub
the data. I want to introduce new data types that automatically perform
scrubbing when they're used.
But this puts even more pressure on MaybeArchivalBinaryData, and hence
this commit, to start differentiating between:
1. always binary data vs maybe binary data
2. no scrubbing required vs scrubbing required
The rationale for doing this set of changes has been explained in
ooni#1319.
The reference issue is ooni/probe#2531.
For now, this commit just adds the new type and tests for it without
using the type, which we'll do later once we have added better
marshal/unmarshal testing for the interested types.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are cases where we know we have binary data in input and we want the output to be binary data using the dictionary encoding like
{"format":"base64","data":"..."}.Such cases are, for example, DNS messages bodies.
There is no need for us to pass through the MaybeArchivalBinaryData in such cases. Additionally, this makes MaybeArchivalBinaryData a bit more complex than it probably needs to be.
What's more, ArchivalBinaryData is for when you do not require to scrub the data. I want to introduce new data types that automatically perform scrubbing when they're used.
But this puts even more pressure on MaybeArchivalBinaryData, and hence this commit, to start differentiating between:
always binary data vs maybe binary data
no scrubbing required vs scrubbing required
The rationale for doing this set of changes has been explained in #1319.
The reference issue is ooni/probe#2531.
For now, this commit just adds the new type and tests for it without using the type, which we'll do later once we have added better marshal/unmarshal testing for the interested types.