fix: handle strings the same in cjs, esm, and deno

[isaacs]

This also ports some of the // istanbul ignore comments to their
associated /* c8 ignore start/stop */ equivalents, and
coverage-ignores some value fallbacks that are there for safety but can
never be hit in normal usage.

Fix: #138

[isaacs]

It seems a bit weird to double up the dependencies, but I guess if those modules were shipped as hybrid esm/cjs, it'd be the same amount of code anyway. The only hazard is that the esm versions will probably get bugfixes that aren't going to be backported, but all three of them seem extremely stable, so I think the risk is low.

[isaacs]

Is nyc used when testing this library? I left the Istanbul comments in, but maybe they should be removed?

[isaacs]

Failure is from the standardx linter, will fix shortly.

[shadowspawn]

Switched from nyc to c8 in #80, so I think the comments can go.

[isaacs]

Hm, the standardx complaint is kind of confusing. It seems like it doesn't like import() statements, but they're clearly valid and supported. I tried declaring it as a global, but doesn't seem to have any effect. Any ideas? I'm not very familiar with standard or eslint, been all in on prettier in my projects for a long time now.

[shadowspawn]

You have the file named for explicit CommonJS (.cjs): cjs-esm-compare.cjs

Given you want to get access to both implementations and using require+import, should that be .mjs?

Update: well, tried that and didn't help!

[shadowspawn]

Hacking about without a deep understanding yet, but got new test running. The esm tests are a bit loose as not covered by standardx yet.

Renamed to cjs-esm-compare.mjs and moved down into esm folder. Added to test:esm.

    "test:esm": "c8 mocha ./test/esm/*.mjs",

Reworked to use imports consistently:

'use strict'

/* global describe, it */
import cliuiESM from '../../index.mjs';
import cliuiCJS from '../../build/index.cjs'
import { expect } from 'chai';

const text = `usage: git tag [-a | -s | -u <key-id>] [-f] [-m <msg> | -F <file>] [-e]
               <tagname> [<commit> | <object>]
   or: git tag -d <tagname>...
   or: git tag [-n[<num>]] -l [--contains <commit>] [--no-contains <commit>]
               [--points-at <object>] [--column[=<options>] | --no-column]
               [--create-reflog] [--sort=<key>] [--format=<format>]
               [--merged <commit>] [--no-merged <commit>] [<pattern>...]
   or: git tag -v [--format=<format>] <tagname>...`

describe('consistent wrapping', () => {
  it('should produce matching output in cjs and esm', () => {
    const uiCJS = cliuiCJS({})
    const uiESM = cliuiESM({})
    uiCJS.div({
      padding: [0, 0, 0, 0],
      text,
    })
    uiESM.div({
      padding: [0, 0, 0, 0],
      text,
    })
    expect(uiCJS.toString()).to.equal(uiESM.toString())
  })
})

Then npm run test:esm runs the new test and it passes, but the old test fails (which might be expected since wrapping has changed). Running the test on its own:

% npx mocha test/esm/cjs-esm-compare.mjs


  consistent wrapping
    ✔ should produce matching output in cjs and esm


  1 passing (6ms)

[isaacs]

You have the file named for explicit CommonJS (.cjs): cjs-esm-compare.cjs

Yes, dynamic import() is now CommonJS node modules load ESM modules.

I suppose it could be refactored to be a .mjs test, and then load the cjs one with createRequire().

[isaacs]

Ah, didn't notice test:esm.

Updated the existing esm test to verify the correct behavior, instead of verifying the incorrect behavior. Should be passing now, everything looks good locally.

[isaacs]

Ah, needed to update deno test as well.

[shadowspawn]

For interest I had a look at the impact in package size. As might be expected, installed size about doubles since two of everything. 😅

Details

# OLD

% npm pack                                        # master

npm notice 📦  cliui@8.0.1
npm notice 
npm notice 📦  cliui@8.0.1
npm notice === Tarball Contents === 
npm notice 731B   LICENSE.txt              
npm notice 3.0kB  README.md                
npm notice 10.0kB build/index.cjs          
npm notice 1.1kB  build/index.d.cts        
npm notice 9.7kB  build/lib/index.js       
npm notice 1.0kB  build/lib/string-utils.js
npm notice 309B   index.mjs                
npm notice 2.0kB  package.json             
npm notice === Tarball Details === 
npm notice name:          cliui                                   
npm notice version:       8.0.1                                   
npm notice filename:      cliui-8.0.1.tgz                         
npm notice package size:  6.4 kB                                  
npm notice unpacked size: 27.7 kB                                 
npm notice shasum:        b1a4ebefef09e89647b03e97b38bb5ad51ad77ba
npm notice integrity:     sha512-54py6xQmmLGZW[...]wTZmWq1VmXvRw==
npm notice total files:   8                                       
npm notice 

% npm i /Users/john/Documents/Sandpits/yargs/cliui/my-fork/cliui-8.0.1.tgz
added 10 packages, and audited 11 packages in 1s

% du -d=1 -h node_modules 
312K	node_modules

% npm ls --all
old@1.0.0 /Users/john/Documents/Sandpits/play/try-cliui/old
└─┬ cliui@8.0.1
  ├─┬ string-width@4.2.3
  │ ├── emoji-regex@8.0.0
  │ ├── is-fullwidth-code-point@3.0.0
  │ └── strip-ansi@6.0.1 deduped
  ├─┬ strip-ansi@6.0.1
  │ └── ansi-regex@5.0.1
  └─┬ wrap-ansi@7.0.0
    ├─┬ ansi-styles@4.3.0
    │ └─┬ color-convert@2.0.1
    │   └── color-name@1.1.4
    ├── string-width@4.2.3 deduped
    └── strip-ansi@6.0.1 deduped

# NEW

% npm pack                                         # isaacs-isaacs/esm-cjs-consistency
npm notice 
npm notice 📦  cliui@8.0.1
npm notice === Tarball Contents === 
npm notice 731B   LICENSE.txt       
npm notice 3.0kB  README.md         
npm notice 10.4kB build/index.cjs   
npm notice 1.1kB  build/index.d.cts 
npm notice 10.1kB build/lib/index.js
npm notice 299B   index.mjs         
npm notice 2.2kB  package.json      
npm notice === Tarball Details === 
npm notice name:          cliui                                   
npm notice version:       8.0.1                                   
npm notice filename:      cliui-8.0.1.tgz                         
npm notice package size:  6.1 kB                                  
npm notice unpacked size: 27.7 kB                                 
npm notice shasum:        20504c591d90743add1b812cf439b9d845108468
npm notice integrity:     sha512-F9qOs1ByD8UAg[...]uVxRAh/gdtH3g==
npm notice total files:   7                                       
npm notice 

% npm i /Users/john/Documents/Sandpits/yargs/cliui/my-fork/cliui-8.0.1.tgz
added 23 packages, and audited 24 packages in 5s

% du -d=1 -h node_modules 
756K	node_modules

% npm ls --all
new@1.0.0 /Users/john/Documents/Sandpits/play/try-cliui/new
└─┬ cliui@8.0.1
  ├─┬ string-width-cjs@npm:string-width@4.2.3
  │ ├── emoji-regex@8.0.0
  │ ├── is-fullwidth-code-point@3.0.0
  │ └─┬ strip-ansi@6.0.1
  │   └── ansi-regex@5.0.1
  ├─┬ string-width@5.1.2
  │ ├── eastasianwidth@0.2.0
  │ ├── emoji-regex@9.2.2
  │ └── strip-ansi@7.0.1 deduped
  ├─┬ strip-ansi-cjs@npm:strip-ansi@6.0.1
  │ └── ansi-regex@5.0.1
  ├─┬ strip-ansi@7.0.1
  │ └── ansi-regex@6.0.1
  ├─┬ wrap-ansi-cjs@npm:wrap-ansi@7.0.0
  │ ├─┬ ansi-styles@4.3.0
  │ │ └─┬ color-convert@2.0.1
  │ │   └── color-name@1.1.4
  │ ├─┬ string-width@4.2.3
  │ │ ├── emoji-regex@8.0.0
  │ │ ├── is-fullwidth-code-point@3.0.0 deduped
  │ │ └── strip-ansi@6.0.1 deduped
  │ └─┬ strip-ansi@6.0.1
  │   └── ansi-regex@5.0.1
  └─┬ wrap-ansi@8.1.0
    ├── ansi-styles@6.2.1
    ├── string-width@5.1.2 deduped
    └── strip-ansi@7.0.1 deduped

[isaacs]

Anything else needing to be done to land this? Some folks getting upset at me using a git dep while waiting.

[isaacs]

(Apparently the build fails when installing learna for some reason.)

[shadowspawn]

Anything else needing to be done to land this?

Nothing else from you at this point from me anyway, thanks.

I'll review it as a supply-both approach to getting wrapping working in esm et al.

A high level question is whether we want to land a double-dependency update, and I'll want input from @bcoe on that. I know people have expressed concerns about number of dependencies and size of Yargs install in past so this is of some concern, which is why I checked install sizes for reference. (And Benjamin has also mentioned heading for ESM whether esm-first or esm-only at some point.)

Some folks getting upset at me using a git dep while waiting.

I don't have a timeline for you. Could you work around the limitation in the meantime by using require from esm and getting the CommonJs implementation? (I am not sure whether this makes any sense for your runtime setup.)

[isaacs]

I'll just publish my fork as @isaacs/cliui for now. Was just being lazy. I don't mind littering my npm account, it's a garbage pile of throwaway junk after all these years anyway lol

[OpportunityLiu]

The "string-width-cjs": "npm:string-width@^4.2.0" part breaks in yarn v1. While other packages still deps "string-width@^4", yarn install only one string-width-cjs in root of node_modules.

"string-width-cjs@npm:string-width@^4.2.0", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.3:
  version "4.2.3"
  resolved "https://registry.npmmirror.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010"
  integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==
  dependencies:
    emoji-regex "^8.0.0"
    is-fullwidth-code-point "^3.0.0"
    strip-ansi "^6.0.1"

> yarn why string-width    
yarn why v1.22.19
[1/4] Why do we have the module "string-width"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "string-width@5.1.2"
info Reasons this module exists
   - "web-ext#update-notifier#boxen" depends on it
   - Hoisted from "web-ext#update-notifier#boxen#string-width"
   - Hoisted from "web-ext#update-notifier#boxen#widest-line#string-width"
   - Hoisted from "web-ext#update-notifier#boxen#wrap-ansi#string-width"
   - Hoisted from "rimraf#glob#jackspeak#@isaacs#cliui#string-width"
info Disk size without dependencies: "200KB"
info Disk size with unique dependencies: "280KB"
info Disk size with transitive dependencies: "300KB"
info Number of shared dependencies: 4
Done in 0.54s.
> yarn why string-width-cjs
yarn why v1.22.19
[1/4] Why do we have the module "string-width-cjs"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "string-width-cjs@4.2.3"
info Reasons this module exists
   - "yargs" depends on it
   - Hoisted from "yargs#string-width-cjs"
   - Hoisted from "yargs#cliui#string-width-cjs"
   - Hoisted from "web-ext#yargs#string-width-cjs"
   - Hoisted from "yargs#cliui#wrap-ansi-cjs#string-width-cjs"
   - Hoisted from "web-ext#addons-linter#yargs#string-width-cjs"
   - Hoisted from "rimraf#glob#jackspeak#@isaacs#cliui#string-width-cjs"
   - Hoisted from "web-ext#update-notifier#boxen#ansi-align#string-width-cjs"
info Disk size without dependencies: "84KB"
info Disk size with unique dependencies: "164KB"
info Disk size with transitive dependencies: "184KB"
info Number of shared dependencies: 4
Done in 0.55s.

[shadowspawn]

@isaacs Does using a CommonJS implementation in ESM work for clients of jackspeak?

You mentioned createRequire in #138 (comment) so might be ok, but you may have decided since then that really want "true ESM"!

Related: In older discussion, Deno and Yargs browser support were the scenarios of concern for "true ESM": #89 (comment)

[isaacs]

"True ESM" is definitely the goal in principle, as eventually I would like tap to work in deno and browsers, so I've been shipping full hybrid builds for everything. That said, the stuff using cliui won't ever work in browsers per se, and most of the cli runner is pretty node specific, so getting to deno support will be a big lift as well, so just shipping CJS is fine, too. The main thing for me here is, import * from cliui should be functionally equivalent to require('cliui'). However you wanna accomplish that is fine.

[shadowspawn]

Good info, thanks @isaacs

The reason I am considering different approaches is that the underling lack of ansi support in ESM flavour of cliui generated a few issues across yargs in a couple of years. Using the npm aliases via jackspeak generated a couple of reports within days (due to breaking builds). Which we only know now because you tried an approach @isaacs , full credit for that!

[naugtur]

When yarn3 attempts to run audit on a project with @isaacs/cliui in dependencies, it gets a HTTP400 from the audit API because of the *-cjs references:

https://github.com/isaacs/cliui/blame/main/package.json#L53

Repro with details:
danjm/yarn-audit-bug#1

[shadowspawn]

I wasn't able to reproduce this. I cloned out your repo, and ran (on Mac):

% yarn --version
3.2.4
% yarn install
...
% yarn npm audit
➤ YN0001: No audit suggestions

[shadowspawn]

I pulled from correct repo, but still can not reproduce. What commands are you running that are showing an issue?

[naugtur]

Sorry!
You need to add --recursive because otherwise yarn won't send transitive dependencies. I'll update the description there.

[isaacs]

...?!

Yarn doesn't audit transitive deps by default??

[naugtur]

🤷‍♂️ c'est la vie

[shadowspawn]

Ok, reproduced now.

I have an alternative PR for cliui prepared, but don't have any yarns knowledge to try plugging it in to your repo and see if it performs better. If you can easily try this repo as a replacement for the code from this PR, I would love to know if avoids the problem. No worries if you are not interested!

• feat: add proper wrapping for esm and Deno by updating dependencies #143
• https://github.com/shadowspawn/cliui/tree/feature/esm-wrapping

[Bonnie-dot]

I meet the same issue with yarn. I went through all comments, but I didn't find any solution except using npm. And I used npm, it can works. Hopefully it can be solved in the future.

[G-Rath]

@shadowspawn I've tried your PR and confirmed it doesn't have the same problem as @isaacs/cliui 🎉

[LennonReid]

I encountered the same issue since yesterday. However, after switching to PNPM, the problem was resolved. Running the command 'yarn why cliui' pointed me to the @ngneat/transloco. Thanks for the guidance; you all made my day!

[shadowspawn]

For yarn 1 users hitting this page because of runtime failures with cliui and string-width, try upgrading to yarn v1.22.22 or higher.

• https://github.com/yarnpkg/yarn/releases/tag/v1.22.22

[shadowspawn]

With a yarn 1 update available, I think this is now ok.

I have an alternative PR open which uses rollup plugins rather than a package manager alias to support the cjs/esm implementations, different tradeoffs but similar outcome: #143.

I do prefer approach in #143 and still getting people reporting issues from the breakage in yarn 1

[slorber]

Not sure how helpful this is, but this change in @isaacs/cliui fork scared me.

I'm running this on the Docusaurus repo, trying to detect possible supply chain attacks, and got these aliases being reported:

npx lockfile-lint --path yarn.lock --type yarn --allowed-hosts yarn --validate-https --validate-package-names

detected resolved URL for package with a different name: string-width-cjs
    expected: string-width-cjs
    actual: string-width

detected resolved URL for package with a different name: strip-ansi-cjs
    expected: strip-ansi-cjs
    actual: strip-ansi

detected resolved URL for package with a different name: wrap-ansi-cjs
    expected: wrap-ansi-cjs
    actual: wrap-ansi

 ✖ Error: security issues detected!

And there's this "anonymous" guy that published empty packages on npm with the exact same name:
https://www.npmjs.com/package/string-width-cjs
https://www.npmjs.com/package/strip-ansi-cjs
https://www.npmjs.com/package/wrap-ansi-cjs

I don't know how harmful it could be, but this looks suspicious that those packages even get a few weekly downloads.

[isaacs]

This is not a security issue, it's just a dependency alias, which every js package manager should support by now. If yarn is complaining, it's likely a yarn bug.

Those packages likely get downloads because the registry is constantly being mirrored by many third-party registry instances. I don't know if they're malicious or just litter, but they're irrelevant here.

[shadowspawn]

That a might be a false positive from lockfile-lint, possibly it does not look for package aliases. The author of lockfile-lint may be interested.

If is interesting that there are some matching packages published. The optimistic view is perhaps someone was investigating working around the bug in earlier versions of yarn. The pessimistic view is someone was investigating exploiting the problem. The failures we have seen are a runtime failure rather than a download so not directly fixable/exploitable in this way. (But someone might possibly see the alias name in a message and think it was missing and try installing it.)

[isaacs]

Yes, it is definitely a false positive in whatever tool is generating this warning. I recommend reporting it to them.

[lirantal]

@isaacs considering that the npm registry itself is/was vulnerable to npm package aliases and allowed spoofing package names (the way you used them here in the package.json) is enough to warrant that this is a real-world concern and not theoretical, in my opinion. See here for evidence: https://snyk.io/blog/exploring-extensions-of-dependency-confusion-attacks-via-npm-package-aliasing/

@slorber lockfile-lint allows you to accept the risk of package aliases as long as you explicitly call them out, here's how to allow-list one of the packages:

npx lockfile-lint --path package-lock.json --allowed-hosts yarn npm --validate-https --validate-package-names --allowed-package-name-aliases string-width-cjs:string-width

[isaacs]

@lirantal That blog post is misleading. It's not a "supply chain security" issue, it's just a website display issue. I'm not saying it's not a bug, and yes it does potentially lead to falsely providing reputation in some way to the other package name, but it's imo quite a stretch to call it evidence of a supply chain security bug for package publishers or consumers using package aliases as they're intended. The website is not part of the supply chain, and the registry and all modern clients of it handle aliases just fine.

The failure of lockfile-lint to do so as well leads to false positives like this one creating make-work and wasting everyone's time. Please reconsider.