Use file utility to check if files are binary
#2615
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jurre
changed the title
file utility to check if files are binary
|
I will add some tests to demonstrate pathological cases tomorrow, but would love other feedback in the meanwhile. |
jasonrudolph
approved these changes
Oct 8, 2020
jurre
force-pushed
the
jurre/add-extensions
branch
2 times, most recently
from
67d9529 to
61963a4
Compare
|
These cargo failures seem unrelated and I'm also getting them on |
Noticed a bunch of extensions that were incorrectly being submitted as binary to the GitHub API. Alternatively we could also invert this and list a few known binary extensions, for example: ```ruby BINARY_EXTS = %w( .gem .zip .tar .gz .tgz .wav .rar .mp3 ) ``` This has the downside that any binary files without an extension are not picked up. However, we have that same problem currently with text files. I already spotted some `bin/whatever` files that should not be submitted as binary to the GH API.
The `.bundlecache` file is an empty file that bundler puts in a git dependencies `vendor/cache/<gem>` folder, to indicate that the file does not need to be re-downloaded, see: https://github.com/rubygems/bundler/blob/6394536271a0326efd4ed3544b78e528da7e90b1/lib/bundler/plugin/api/source.rb#L175-L192 Empty files without an extension are treated as binary files by the `file` util. This is not problematic for submitting to the GitHub API, since it ends up simply base64 encoding (and subsequently decoding in GitHub) an empty string.
jurre
force-pushed
the
jurre/add-extensions
branch
from
3db52f4 to
5282864
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Noticed a bunch of extensions that were incorrectly being submitted as
binary to the GitHub API.
Detecting if files are binary or not is hard, and lists of extensions are a pain to maintain, plus there can be any number of files that don't have an extension. This is especially problematic for
gitdependencies.Currently, when a file is not marked correctly as binary, this causes issues with PR creation, as we need to encode and upload those files differently
This 🩹 should at least make sure that we do the right thing.
Longer term, we want to investigate creating compressed git packs, and uploading those packs using the API.