fix: strip BOM when reading JSON files

[rmg]

fs.readFile* don't strip the BOM header, even when specifying 'utf8' as
the encoding, and JSON.parse() doesn't handle it either. There are
technically a bunch of BOM indicators, but this is the one seen most
commonly and actually appears in a number of package.json files in the
wild.

See nodejs/node#6924, nodejs/node#3040 for background.

[rmg]

Rebased after #6 was merged.

[rmg]

I don't have a large sample size, but it looks like among my local cache, the problem file is a singular occurrence:

$ find ~/.npm/_cacache/content-v2/sha1 -type file | while read f; do \
   tar -C ~/deleteme/bom --strip-components 1 -xUzf $f package/package.json \
    && file ~/deleteme/bom/package.json; \
  done | cut -d: -f2 | sort | uniq -c
2810  ASCII text
  56  ASCII text, with CRLF line terminators
   7  ASCII text, with very long lines
   1  ASCII text, with very long lines, with no line terminators
   1  HTML document text, ASCII text
   1  UTF-8 Unicode (with BOM) text
 170  UTF-8 Unicode text
   1  UTF-8 Unicode text, with CRLF line terminators
   2  UTF-8 Unicode text, with very long lines

[rmg]

Went hunting to see how npm does this, in case it has a clever way of handling it.. Turns out it is basically doing the same thing: https://github.com/npm/npm/blob/latest/lib/utils/parse-json.js

if (content.charCodeAt(0) === 0xFEFF) {
  content = content.slice(1)
}

[rmg]

@zkat if this feels like deja vu.. npm/npm#8724

[zkat]

This LGTM.

Can we get a test for it, too?

[zkat]

Looks good, but needs a test (for all of npm-shrinkwrap.json, package-lock.json, and package.json, imo)

[rmg]

Can we get a test for it, too?

Incoming. I skipped it on the first pass because every time I looked at failing file with vim it "helpfully" stripped the BOM for me.

Looks good, but needs a test (for all of npm-shrinkwrap.json, package-lock.json, and package.json, imo)

Is it really necessary to test 3 different filenames in a unit test for readJson?

[rmg]

@zkat I've rebased and added a unit test for for readJson.