Decoder not present in Node.

[mbostock]

I am not able to reproduce this error myself, but I have received several issue reports to my shapefile parser of an error generated by text-encoding: 1 2 3. The error is:

Decoder not present. Did you forget to include encoding-indexes.js first?

But since the Node API is simply require("text-encoding"), the error is internal, because the text-encoding library should be loading the indexes itself in Node.

In an attempt to workaround this error, I reverted shapefile to use text-encoding@0.6.1 rather than 0.6.3, as it appears that changes 8c43765 b8503e7 may be related. Sorry I don’t have any more information to go on. If users report any more details I will relay them here. Thank you!

[nickrttn]

Replying to mbostock/shapefile/issues/30 here as requested.

1. Did you install shp2json locally (npm install shapefile) or globally (npm install -g shapefile)?

I'm installing shp2json globally.

2. If you installed it locally, what is the output of npm ls? If globally, npm ls -g?

The output of npm ls -g is below. Worth to note I tried to run it from a clean install of node v7.4.0.

/Users/nickrutten/.nvm/versions/node/v7.4.0/lib
├─┬ npm@4.0.5
│ ├── abbrev@1.0.9
│ ├── ansi-regex@2.0.0
│ ├── ansicolors@0.3.2
│ ├── ansistyles@0.1.3
│ ├── aproba@1.0.4
│ ├── archy@1.0.0
│ ├── asap@2.0.5
│ ├── chownr@1.0.1
│ ├── cmd-shim@2.0.2
│ ├─┬ columnify@1.5.4
│ │ └─┬ wcwidth@1.0.0
│ │   └─┬ defaults@1.0.3
│ │     └── clone@1.0.2
│ ├─┬ config-chain@1.1.11
│ │ └── proto-list@1.2.4
│ ├── debuglog@1.0.1
│ ├── dezalgo@1.0.3
│ ├── editor@1.0.0
│ ├── fs-vacuum@1.2.9
│ ├── fs-write-stream-atomic@1.0.8
│ ├── fstream@1.0.10
│ ├─┬ fstream-npm@1.2.0
│ │ └─┬ fstream-ignore@1.0.5
│ │   └─┬ minimatch@3.0.3
│ │     └─┬ brace-expansion@1.1.6
│ │       ├── balanced-match@0.4.2
│ │       └── concat-map@0.0.1
│ ├─┬ glob@7.1.1
│ │ ├── fs.realpath@1.0.0
│ │ ├─┬ minimatch@3.0.3
│ │ │ └─┬ brace-expansion@1.1.6
│ │ │   ├── balanced-match@0.4.2
│ │ │   └── concat-map@0.0.1
│ │ └── path-is-absolute@1.0.1
│ ├── graceful-fs@4.1.11
│ ├── has-unicode@2.0.1
│ ├── hosted-git-info@2.1.5
│ ├── iferr@0.1.5
│ ├── imurmurhash@0.1.4
│ ├── inflight@1.0.6
│ ├── inherits@2.0.3
│ ├── ini@1.3.4
│ ├─┬ init-package-json@1.9.4
│ │ ├─┬ glob@6.0.4
│ │ │ ├─┬ minimatch@3.0.3
│ │ │ │ └─┬ brace-expansion@1.1.6
│ │ │ │   ├── balanced-match@0.4.2
│ │ │ │   └── concat-map@0.0.1
│ │ │ └── path-is-absolute@1.0.0
│ │ └── promzard@0.3.0
│ ├─┬ JSONStream@1.2.1
│ │ ├── jsonparse@1.2.0
│ │ └── through@2.3.8
│ ├── lockfile@1.0.2
│ ├── lodash._baseindexof@3.1.0
│ ├─┬ lodash._baseuniq@4.6.0
│ │ ├── lodash._createset@4.0.3
│ │ └── lodash._root@3.0.1
│ ├── lodash._bindcallback@3.0.1
│ ├── lodash._cacheindexof@3.0.2
│ ├── lodash._createcache@3.1.2
│ ├── lodash._getnative@3.9.1
│ ├── lodash.clonedeep@4.5.0
│ ├── lodash.restparam@3.6.1
│ ├── lodash.union@4.6.0
│ ├── lodash.uniq@4.5.0
│ ├── lodash.without@4.4.0
│ ├─┬ mississippi@1.2.0
│ │ ├─┬ concat-stream@1.5.2
│ │ │ ├─┬ readable-stream@2.0.6
│ │ │ │ ├── core-util-is@1.0.2
│ │ │ │ ├── isarray@1.0.0
│ │ │ │ ├── process-nextick-args@1.0.7
│ │ │ │ ├── string_decoder@0.10.31
│ │ │ │ └── util-deprecate@1.0.2
│ │ │ └── typedarray@0.0.6
│ │ ├─┬ duplexify@3.4.5
│ │ │ ├─┬ end-of-stream@1.0.0
│ │ │ │ └── once@1.3.3
│ │ │ └── stream-shift@1.0.0
│ │ ├─┬ end-of-stream@1.1.0
│ │ │ └── once@1.3.3
│ │ ├── flush-write-stream@1.0.0
│ │ ├── from2@2.3.0
│ │ ├── pump@1.0.1
│ │ ├── pumpify@1.3.5
│ │ ├── stream-each@1.1.2
│ │ └─┬ through2@2.0.1
│ │   ├─┬ readable-stream@2.0.6
│ │   │ ├── core-util-is@1.0.2
│ │   │ ├── isarray@1.0.0
│ │   │ ├── process-nextick-args@1.0.7
│ │   │ ├── string_decoder@0.10.31
│ │   │ └── util-deprecate@1.0.2
│ │   └── xtend@4.0.1
│ ├─┬ mkdirp@0.5.1
│ │ └── minimist@0.0.8
│ ├─┬ node-gyp@3.4.0
│ │ ├─┬ minimatch@3.0.3
│ │ │ └─┬ brace-expansion@1.1.6
│ │ │   ├── balanced-match@0.4.2
│ │ │   └── concat-map@0.0.1
│ │ ├─┬ npmlog@3.1.2
│ │ │ ├─┬ are-we-there-yet@1.1.2
│ │ │ │ └── delegates@1.0.0
│ │ │ ├── console-control-strings@1.1.0
│ │ │ ├─┬ gauge@2.6.0
│ │ │ │ ├── has-color@0.1.7
│ │ │ │ ├── object-assign@4.1.0
│ │ │ │ ├── signal-exit@3.0.1
│ │ │ │ ├─┬ string-width@1.0.2
│ │ │ │ │ ├─┬ code-point-at@1.0.0
│ │ │ │ │ │ └── number-is-nan@1.0.0
│ │ │ │ │ └─┬ is-fullwidth-code-point@1.0.0
│ │ │ │ │   └── number-is-nan@1.0.0
│ │ │ │ └── wide-align@1.1.0
│ │ │ └── set-blocking@2.0.0
│ │ └─┬ path-array@1.0.1
│ │   └─┬ array-index@1.0.0
│ │     ├─┬ debug@2.2.0
│ │     │ └── ms@0.7.1
│ │     └─┬ es6-symbol@3.1.0
│ │       ├── d@0.1.1
│ │       └─┬ es5-ext@0.10.12
│ │         └── es6-iterator@2.0.0
│ ├── nopt@3.0.6
│ ├── normalize-git-url@3.0.2
│ ├─┬ normalize-package-data@2.3.5
│ │ └─┬ is-builtin-module@1.0.0
│ │   └── builtin-modules@1.1.1
│ ├── npm-cache-filename@1.0.2
│ ├── npm-install-checks@3.0.0
│ ├── npm-package-arg@4.2.0
│ ├─┬ npm-registry-client@7.3.0
│ │ ├─┬ concat-stream@1.5.2
│ │ │ ├─┬ readable-stream@2.0.6
│ │ │ │ ├── core-util-is@1.0.2
│ │ │ │ ├── isarray@1.0.0
│ │ │ │ ├── process-nextick-args@1.0.7
│ │ │ │ ├── string_decoder@0.10.31
│ │ │ │ └── util-deprecate@1.0.2
│ │ │ └── typedarray@0.0.6
│ │ └─┬ npmlog@3.1.2
│ │   ├─┬ are-we-there-yet@1.1.2
│ │   │ └── delegates@1.0.0
│ │   ├── console-control-strings@1.1.0
│ │   ├─┬ gauge@2.6.0
│ │   │ ├── has-color@0.1.7
│ │   │ ├── object-assign@4.1.0
│ │   │ ├── signal-exit@3.0.1
│ │   │ ├─┬ string-width@1.0.2
│ │   │ │ ├─┬ code-point-at@1.0.1
│ │   │ │ │ └── number-is-nan@1.0.1
│ │   │ │ └─┬ is-fullwidth-code-point@1.0.0
│ │   │ │   └── number-is-nan@1.0.1
│ │   │ └── wide-align@1.1.0
│ │   └── set-blocking@2.0.0
│ ├── npm-user-validate@0.1.5
│ ├─┬ npmlog@4.0.1
│ │ ├─┬ are-we-there-yet@1.1.2
│ │ │ └── delegates@1.0.0
│ │ ├── console-control-strings@1.1.0
│ │ ├─┬ gauge@2.7.1
│ │ │ ├── has-color@0.1.7
│ │ │ ├── object-assign@4.1.0
│ │ │ ├── signal-exit@3.0.1
│ │ │ ├─┬ string-width@1.0.2
│ │ │ │ ├── code-point-at@1.1.0
│ │ │ │ └─┬ is-fullwidth-code-point@1.0.0
│ │ │ │   └── number-is-nan@1.0.1
│ │ │ └── wide-align@1.1.0
│ │ └── set-blocking@2.0.0
│ ├── once@1.4.0
│ ├── opener@1.4.2
│ ├─┬ osenv@0.1.3
│ │ ├── os-homedir@1.0.1
│ │ └── os-tmpdir@1.0.1
│ ├── path-is-inside@1.0.2
│ ├─┬ read@1.0.7
│ │ └── mute-stream@0.0.5
│ ├── read-cmd-shim@1.0.1
│ ├─┬ read-installed@4.0.3
│ │ └── util-extend@1.0.3
│ ├─┬ read-package-json@2.0.4
│ │ ├─┬ glob@6.0.4
│ │ │ ├─┬ minimatch@3.0.3
│ │ │ │ └─┬ brace-expansion@1.1.6
│ │ │ │   ├── balanced-match@0.4.2
│ │ │ │   └── concat-map@0.0.1
│ │ │ └── path-is-absolute@1.0.0
│ │ └─┬ json-parse-helpfulerror@1.0.3
│ │   └── jju@1.3.0
│ ├── read-package-tree@5.1.5
│ ├─┬ readable-stream@2.2.2
│ │ ├── buffer-shims@1.0.0
│ │ ├── core-util-is@1.0.2
│ │ ├── isarray@1.0.0
│ │ ├── process-nextick-args@1.0.7
│ │ ├── string_decoder@0.10.31
│ │ └── util-deprecate@1.0.2
│ ├── readdir-scoped-modules@1.0.2
│ ├── realize-package-specifier@3.0.3
│ ├─┬ request@2.79.0
│ │ ├── aws-sign2@0.6.0
│ │ ├── aws4@1.5.0
│ │ ├── caseless@0.11.0
│ │ ├─┬ combined-stream@1.0.5
│ │ │ └── delayed-stream@1.0.0
│ │ ├── extend@3.0.0
│ │ ├── forever-agent@0.6.1
│ │ ├─┬ form-data@2.1.2
│ │ │ └── asynckit@0.4.0
│ │ ├─┬ har-validator@2.0.6
│ │ │ ├─┬ chalk@1.1.3
│ │ │ │ ├── ansi-styles@2.2.1
│ │ │ │ ├── escape-string-regexp@1.0.5
│ │ │ │ ├── has-ansi@2.0.0
│ │ │ │ └── supports-color@2.0.0
│ │ │ ├─┬ commander@2.9.0
│ │ │ │ └── graceful-readlink@1.0.1
│ │ │ ├─┬ is-my-json-valid@2.15.0
│ │ │ │ ├── generate-function@2.0.0
│ │ │ │ ├─┬ generate-object-property@1.2.0
│ │ │ │ │ └── is-property@1.0.2
│ │ │ │ ├── jsonpointer@4.0.0
│ │ │ │ └── xtend@4.0.1
│ │ │ └─┬ pinkie-promise@2.0.1
│ │ │   └── pinkie@2.0.4
│ │ ├─┬ hawk@3.1.3
│ │ │ ├── boom@2.10.1
│ │ │ ├── cryptiles@2.0.5
│ │ │ ├── hoek@2.16.3
│ │ │ └── sntp@1.0.9
│ │ ├─┬ http-signature@1.1.1
│ │ │ ├── assert-plus@0.2.0
│ │ │ ├─┬ jsprim@1.3.1
│ │ │ │ ├── extsprintf@1.0.2
│ │ │ │ ├── json-schema@0.2.3
│ │ │ │ └── verror@1.3.6
│ │ │ └─┬ sshpk@1.10.1
│ │ │   ├── asn1@0.2.3
│ │ │   ├── assert-plus@1.0.0
│ │ │   ├── bcrypt-pbkdf@1.0.0
│ │ │   ├── dashdash@1.14.1
│ │ │   ├── ecc-jsbn@0.1.1
│ │ │   ├── getpass@0.1.6
│ │ │   ├── jodid25519@1.0.2
│ │ │   ├── jsbn@0.1.0
│ │ │   └── tweetnacl@0.14.3
│ │ ├── is-typedarray@1.0.0
│ │ ├── isstream@0.1.2
│ │ ├── json-stringify-safe@5.0.1
│ │ ├─┬ mime-types@2.1.13
│ │ │ └── mime-db@1.25.0
│ │ ├── oauth-sign@0.8.2
│ │ ├── qs@6.3.0
│ │ ├── stringstream@0.0.5
│ │ ├─┬ tough-cookie@2.3.2
│ │ │ └── punycode@1.4.1
│ │ ├── tunnel-agent@0.4.3
│ │ └── uuid@3.0.1
│ ├── retry@0.10.0
│ ├── rimraf@2.5.4
│ ├── semver@5.3.0
│ ├── sha@2.0.1
│ ├── slide@1.1.6
│ ├── sorted-object@2.0.1
│ ├─┬ sorted-union-stream@2.1.3
│ │ ├─┬ from2@1.3.0
│ │ │ └─┬ readable-stream@1.1.14
│ │ │   ├── core-util-is@1.0.2
│ │ │   ├── isarray@0.0.1
│ │ │   └── string_decoder@0.10.31
│ │ └── stream-iterate@1.1.1
│ ├── strip-ansi@3.0.1
│ ├─┬ tar@2.2.1
│ │ └── block-stream@0.0.8
│ ├── text-table@0.2.0
│ ├── uid-number@0.0.6
│ ├── umask@1.1.0
│ ├─┬ unique-filename@1.1.0
│ │ └── unique-slug@2.0.0
│ ├── unpipe@1.0.0
│ ├─┬ validate-npm-package-license@3.0.1
│ │ ├─┬ spdx-correct@1.0.2
│ │ │ └── spdx-license-ids@1.2.0
│ │ └─┬ spdx-expression-parse@1.0.2
│ │   ├── spdx-exceptions@1.0.4
│ │   └── spdx-license-ids@1.2.0
│ ├─┬ validate-npm-package-name@2.2.2
│ │ └── builtins@0.0.7
│ ├─┬ which@1.2.12
│ │ └── isexe@1.1.2
│ ├── wrappy@1.0.2
│ └── write-file-atomic@1.2.0
└─┬ shapefile@0.5.9
  ├── array-source@0.0.3
  ├─┬ commander@2.9.0
  │ └── graceful-readlink@1.0.1
  ├─┬ path-source@0.1.2
  │ └── file-source@0.6.1
  ├── slice-source@0.4.1
  ├── stream-source@0.3.4
  └── text-encoding@0.6.2

3. How are you invoking shp2json? Are you specifying the --encoding argument? Are you passing a shapefile as the first argument, or are you piping a shapefile via stdin?

I'm invoking it as follows: shp2json -n shapefile.shp > shapefile.geojson. I've also tried without the pipe, which has the same effect.

4. Do you have anything unusual in your Node setup, such as an alias that would pass default arguments to Node (say --use-strict)?

I don't have anything unusual in my setup, but it might be useful to know I'm installing node through nvm. This has not given me issues with shp2json in any node version before though.

[inexorabletash]

@therufa - I suspect encoding-indexes.js's assignment to global["encoding-indexes"] is overwritten by the change in encodings.js (global['encoding-indexes'] = require("./encoding-indexes.js");). I.e. the change in b8503e7 was lost.

[inexorabletash]

(FYI, won't be able to modify and/or revert until later tonight.)

Hopefully this is not an incompatibility between the semantics of node and webpack for require(). I don't personally use either so I'm at the mercy of bug reports and apparently dueling PRs to try and sort this out. Wheee.

encoding-indexes probably just needs to detect that it's being used in a module-like system and export the indexes, falling back to assigning to the global namespace otherwise.

[inexorabletash]

I've unpublished 0.6.3 to stop this from tripping anyone else up while this is sorted out.

[therufa]

@inexorabletash you're onto something. First contribution starts with a disaster -.- nice!

Edit: So I have fixed the issue, it was that missing accessor from b8503e7. Shame on me.

@mbostock the tests on shapefile ran through after the fix.

[inexorabletash]

you're onto something. First contribution starts with a disaster -.- nice!

That's the attitude! :)

Anyone want to volunteer to help get Travis-CI set up?