-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for files up to 500MB #224
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…default. Added ArrayBuffer Dish type.
…the much smaller and actively maintained utf8 library.
…card is displayed.
…quare' and 'Extract EXIF' operations.
…nd padEnd. 'To Hex' now supports ArrayBuffers.
BRAVO68WEB
pushed a commit
to BRAVO68WEB/CyberChef
that referenced
this pull request
May 29, 2022
[FEATURE] Multi-Search with Custom Bangs Fixes gchq#206
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces support for files up to around 500MB in size (depending on your browser).
ArrayBuffer
data type has been added to the Dish so that large files can be held in their native format without the need to cast to abyteArray
orstring
unless required by a specific operation.These changes vastly improve the stability of the web app when dealing with large amounts of data. The main issues previously revolved around having to render huge amounts of text in the DOM, which browsers aren't particularly good at. This solution does not render the actual content unless it is specifically asked for.
The introduction of
ArrayBuffer
as a Dish type allows for operations to treat the input as a typed array. Over time, we will move several operations over to supportArrayBuffers
instead ofbyteArrays
where this makes sense. In this PR, only the 'Detect File Type' and 'Scan for Embedded Files' operations support theArrayBuffer
Dish type. This means they can run over files without having to cast the data at all. During testing, 'Detect File Type' was run over a 500MB file in less than a millisecond. Other operations may take a longer time to run over large files, although the MD5 operation executes in a reasonable time on files up to about 100MB.A 500MB file loaded, ingested and ready to be downloaded again
Running the 'Detect File Type' operation over a 500MB file full of random data (hence no detected magic bytes). Note the processing time of 0ms.
Running the 'Detect File Type' operation on a JPG file
Viewing the first 1024 bytes of a JPG file. The icon in the top right of the output area allows you to pull up the file overlay again and select a different slice of the file for viewing.
In addition to these changes, this PR also removes
CryptoJS
as a dependency ofUtils.js
. The much smaller and better maintainedutf8
library is used instead.CryptoJS
is still used for the cipher operations, but removing it fromUtils.js
means it does not have to be loaded unless the Cipher module is requested.