[@theia/application-package: 0.3.7] Building 0.3.7 based Theia application is failing

[jopit]

I'm trying to build our Theia application based on 0.3.7, but it's failing with the following error:

Uncaught Exception:  TypeError: application_package_1.ApplicationPackageManager is not a constructor
TypeError: application_package_1.ApplicationPackageManager is not a constructor
    at Object.<anonymous> (/home/theia/node_modules/@theia/cli/lib/theia.js:79:15)
    at Module._compile (module.js:643:30)
    at Object.Module._extensions..js (module.js:654:10)
    at Module.load (module.js:556:32)
    at tryModuleLoad (module.js:499:12)
    at Function.Module._load (module.js:491:3)
    at Module.require (module.js:587:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/theia/node_modules/@theia/cli/bin/theia:2:1)
    at Module._compile (module.js:643:30)

I'm guessing this has something to do with the release of 0.3.8 yesterday. Any help would be greatly appreciated.

[jopit]

Here's the package.json I'm using

{
    "private": true,
    "dependencies": {
        "@theia/typescript": "0.3.7",
        "@theia/navigator": "0.3.7",
        "@theia/terminal": "0.3.7",
        "@theia/outline-view": "0.3.7",
        "@theia/preferences": "0.3.7",
        "@theia/messages": "0.3.7",
        "@theia/git": "0.3.7",
        "@theia/file-search": "0.3.7",
        "@theia/markers": "0.3.7",
	"@theia/java": "0.3.7"
    },
    "devDependencies": {
        "@theia/cli": "0.3.7"
    }
}

[jopit]

Changing devDependencies @thei/cli version to "^0.3.7" gets me somewhat further. It then fails because some of the Theia dependencies of the explicitly specified Theia dependencies get pulled in at the 0.3.8 level, and they are not compatible. I saw errors like the following:

ERROR in ./node_modules/@theia/git/lib/browser/git-widget.js
Module not found: Error: Can't resolve '@theia/editor/lib/browser/diff-uris' in '/home/theia/node_modules/@theia/git/lib/browser'
 @ ./node_modules/@theia/git/lib/browser/git-widget.js 110:18-64
 @ ./node_modules/@theia/git/lib/browser/git-frontend-module.js
 @ ./src-gen/frontend/index.js

So I then explicitly specified all required Theia packages in our package.json, but now I'm running into an out of memory condition on the build

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

How do I increase the memory limit of the build?

[marcdumais-work]

@jopit Not sure why this is not working. I can reproduce the issue using your package.json above. In principle I think using version 0.3.7 of these Theia extensions should have continued to work after we release 0.3.8. But I am a bit over my head in this area.

It's not usually a good idea to mix extensions that are based on a different Theia release. Even if it compiles it will probably not run correctly. You might have better chance by updating all your dependencies to 0.3.8 (or ATM equivalent "latest").

When I do so, using your package.json, then executing the following commands, I get a Theia application stat start and appears to run correctly:

yarn
yarn theia build
yarn theia start

If you have home-made extensions in the mix, you will have to update them so they are compatible with Theia 0.3.8.

[kittaakos]

Theia application based on 0.3.7, but it's failing with the following error:

@jopit, what are you trying to build and on which platform? I hit a similar problem when I have tried to build an electron-based Theia application. Note, in my case the problem occurred at runtime. I am not sure whether our issues are related or not.

using version 0.3.7 of these Theia extensions should have continued to work

Yes.

[kittaakos]

I am not sure whether our issues are related or not.

Probably not. I was able to build Theia based on your package.json, @jopit. Please tell us more what did you do.

ls -al && cat package.json && node --version && npm --version && yarn --version && yarn
total 8
drwxr-xr-x   3 akos.kitta  staff   96 Mar 30 08:22 .
drwx------+ 12 akos.kitta  staff  384 Mar 30 08:19 ..
-rw-r--r--   1 akos.kitta  staff  422 Mar 30 08:20 package.json
{
  "private": true,
  "dependencies": {
    "@theia/typescript": "0.3.7",
    "@theia/navigator": "0.3.7",
    "@theia/terminal": "0.3.7",
    "@theia/outline-view": "0.3.7",
    "@theia/preferences": "0.3.7",
    "@theia/messages": "0.3.7",
    "@theia/git": "0.3.7",
    "@theia/file-search": "0.3.7",
    "@theia/markers": "0.3.7",
    "@theia/java": "0.3.7"
  },
  "devDependencies": {
    "@theia/cli": "0.3.7"
  }
}v8.11.1
5.6.0
1.5.1
yarn install v1.5.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
warning @theia/typescript > @theia/core > @types/perfect-scrollbar@1.3.0: This is a stub types definition for perfect-scrollbar (https://github.com/noraesae/perfect-scrollbar). perfect-scrollbar provides its own type definitions, so you don't need @types/perfect-scrollbar installed!
warning @theia/typescript > @theia/core > electron > electron-download > nugget > progress-stream > through2 > xtend > object-keys@0.4.0: 
[2/4] 🚚  Fetching packages...
warning vscode-base-languageclient@0.0.1-alpha.3: The engine "vscode" appears to be invalid.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 34.09s.

[marcdumais-work]

@kittaakos On Linux I get the same result as you (with slightly different versions of tools and minus the nifty icons):

ls -al && cat package.json && node --version && npm --version && yarn --version && yarn && yarn theia build
total 28
drwxrwxr-x  2 marc marc  4096 Mar 30 06:01 .
drwxrwxrwt 23 root root 20480 Mar 30 06:01 ..
-rw-rw-r--  1 marc marc   423 Mar 30 05:47 package.json
{
  "private": true,
  "dependencies": {
    "@theia/typescript": "0.3.7",
    "@theia/navigator": "0.3.7",
    "@theia/terminal": "0.3.7",
    "@theia/outline-view": "0.3.7",
    "@theia/preferences": "0.3.7",
    "@theia/messages": "0.3.7",
    "@theia/git": "0.3.7",
    "@theia/file-search": "0.3.7",
    "@theia/markers": "0.3.7",
    "@theia/java": "0.3.7"
  },
  "devDependencies": {
    "@theia/cli": "0.3.7"
  }
}
v8.9.0
5.7.1
1.5.1
yarn install v1.5.1
info No lockfile found.
[1/4] Resolving packages...
warning @theia/typescript > @theia/core > @types/perfect-scrollbar@1.3.0: This is a stub types definition for perfect-scrollbar (https://github.com/noraesae/perfect-scrollbar). perfect-scrollbar provides its own type definitions, so you don't need @types/perfect-scrollbar installed!
warning @theia/typescript > @theia/core > electron > electron-download > nugget > progress-stream > through2 > xtend > object-keys@0.4.0: 
[2/4] Fetching packages...
warning vscode-base-languageclient@0.0.1-alpha.3: The engine "vscode" appears to be invalid.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 29.66s.

At that point it looks like the dependencies have been revolved, but no actual application has been built?

ls -al
total 128
drwxrwxr-x   3 marc marc  4096 Mar 30 06:01 ./
drwxrwxrwt  23 root root 20480 Mar 30 06:05 ../
drwxrwxr-x 326 marc marc 12288 Mar 30 06:01 node_modules/
-rw-rw-r--   1 marc marc   423 Mar 30 05:47 package.json
-rw-rw-r--   1 marc marc 89266 Mar 30 06:01 yarn.lock

Taking inspiration from how we build in our docker images, I then tried yarn theia build, and got the same error as @jopit :

yarn theia build
yarn run v1.5.1
$ /tmp/a/node_modules/.bin/theia build
Uncaught Exception:  TypeError: application_package_1.ApplicationPackageManager is not a constructor
TypeError: application_package_1.ApplicationPackageManager is not a constructor
    at Object.<anonymous> (/tmp/a/node_modules/@theia/cli/lib/theia.js:79:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/tmp/a/node_modules/@theia/cli/bin/theia:2:1)
    at Module._compile (module.js:635:30)
Done in 0.20s.

Is that the correct way to build? Do you see the same?

[jopit]

We are building in a docker image, using yarn theia build

[marcdumais-work]

I noticed that I get the same error calling directly the theia script from @theia/cli , for version 0.3.7. So maybe not even build related as such?

$ /tmp/a/node_modules/.bin/theia 
Uncaught Exception:  TypeError: application_package_1.ApplicationPackageManager is not a constructor
TypeError: application_package_1.ApplicationPackageManager is not a constructor
    at Object.<anonymous> (/tmp/a/node_modules/@theia/cli/lib/theia.js:79:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/tmp/a/node_modules/@theia/cli/bin/theia:2:1)
    at Module._compile (module.js:635:30)

Trying the same with @theia/cli version 0.3.8 works as expected:

$ ./node_modules/.bin/theia
Commands:
  start             start the browser backend
  clean             clean for the browser target
  build             webpack the browser frontend
  rebuild           rebuild native node modules for the browser
  rebuild:browser   rebuild native node modules for the browser
  rebuild:electron  rebuild native node modules for the electron

Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]

Not enough non-option arguments: got 0, need at least 1

[kittaakos]

Confirmed.

[marcdumais-work]

I have checked-out the commit made for release 0.3.7 (07907c6), cleaned and rebuilt.

When we invoke yarn theia, theia.js from @theia/cli is executed in the end. The version of that file I built is identical to the one published on npm for 0.3.7.

However there is a difference in the @theia/application-package dependency:
In the version I pulled from npm, /lib/application-package-manager.js is missing (so is the corresponding .map and .d.ts), vs what's built in the source tree. Makes sense considering the error message.

A couple of other .js files are also missing from npm: application-process.js, rebuild.js . What is present in npm seems identical to what I built from source (at least judging by file sizes).

diff:

[marcdumais-work]

hummm... I was wrong about assuming the "missing" files were not part of the published npm package:

wget https://registry.npmjs.org/@theia/application-package/-/application-package-0.3.7.tgz
--2018-03-30 09:50:48--  https://registry.npmjs.org/@theia/application-package/-/application-package-0.3.7.tgz
Resolving registry.npmjs.org (registry.npmjs.org)... 151.101.84.162
Connecting to registry.npmjs.org (registry.npmjs.org)|151.101.84.162|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 38731 (38K) [application/octet-stream]
Saving to: ‘application-package-0.3.7.tgz’

application-package-0.3.7.tgz   100%[======================================================>]  37.82K   101KB/s    in 0.4s    

2018-03-30 09:50:49 (101 KB/s) - ‘application-package-0.3.7.tgz’ saved [38731/38731]

tar xzvf application-package-0.3.7.tgz
$ find . -name application-package-manager.js
./package/lib/application-package-manager.js
$ find . -name application-process.js
./package/lib/application-process.js

On a hunch I added an explicit dev dependency for "@theia/application-package": "0.3.7" to @jopit ;s package.json from above. This seems to take of the missing files, which then are present. But very puzzling to me, I still get the exact same error when I try yarn theia build.

???

[marcdumais-work]

TL;DR: see at the bottom of this post for a work-around to make the package.json reported in this issue work.

I think I understand a bit more now. Using npm ls, I see that despite all Theia extensions specified in package.json being version 0.3.7, a bunch of 0.3.8 extensions (transitive dependencies) are installed by yarn.

I think this is because our extensions define their dependencies to other Theia extensions using the "current version or higher" (i.e. ^0.3.7 for release 0.3.7). yarn prefers pulling the latest possible version of package dependencies specified that way. After the recent release, that means version 0.3.8 for Theia package dependencies.

After running yarn on @jopit 's package.json, I have the following Theia packages in node_modules:

$ npm ls | grep theia | grep -v 'deduped'
├─┬ @theia/cli@0.3.7
│ └─┬ @theia/application-package@0.3.8
├─┬ @theia/file-search@0.3.7
│ ├─┬ @theia/core@0.3.8
│ ├─┬ @theia/filesystem@0.3.8
│ ├─┬ @theia/workspace@0.3.8
├─┬ @theia/git@0.3.7
│ ├─┬ @theia/editor@0.3.8
├─┬ @theia/java@0.3.7
│ ├─┬ @theia/languages@0.3.8
│ │ ├─┬ @theia/output@0.3.8
│ ├─┬ @theia/monaco@0.3.8
│ │ ├─┬ @theia/markers@0.3.8
│ │ │ ├─┬ @theia/navigator@0.3.8
│ │ ├─┬ @theia/outline-view@0.3.8
├─┬ @theia/markers@0.3.7
├─┬ @theia/messages@0.3.7
├─┬ @theia/navigator@0.3.7
├─┬ @theia/outline-view@0.3.7
├─┬ @theia/preferences@0.3.7
│ ├─┬ @theia/userstorage@0.3.8
├─┬ @theia/terminal@0.3.7
│ ├─┬ @theia/process@0.3.8
└─┬ @theia/typescript@0.3.7
  ├─┬ @theia/callhierarchy@0.3.8

That's not good and it explains application-manager being missing in the error messages above; it used to be bundled together with application-package but was split-out after the 0.3.7 release. Looking above, yarn installs @theia/cli@0.3.7 as specified in package.json but then installs the latest available version of its dependency: @theia/application-package@0.3.8. These two versions are not compatible together, since we end-up missing @theia/application-manager@0.3.8, which is essential for yarn theia to run.

Even if we made the compilation work (e.g. using "@theia/cli": "0.3.8" in package.json), the mix-and-match of two different Theia versions in the extensions would almost for sure not work.

Given that, I wonder if it will ever reliably work to use anything other than the current version or symbolic versions latest or next?

Even the following minimalist example fails at runtime, I think because it pulls different, incompatible, versions of dependent Theia packages:

{
    "private": true,
    "dependencies": {
        "@theia/terminal": "0.3.7"    
    },
    "devDependencies": {
        "@theia/cli": "0.3.8"
    }
}

$ npm ls | grep theia | grep -v 'deduped'
├─┬ @theia/cli@0.3.8
│ └─┬ @theia/application-manager@0.3.8
│   ├─┬ @theia/application-package@0.3.8
└─┬ @theia/terminal@0.3.7
  ├─┬ @theia/core@0.3.8
  ├─┬ @theia/filesystem@0.3.8
  ├─┬ @theia/process@0.3.8
  ├─┬ @theia/workspace@0.3.8

There might be a way I am not seeing now - I am only beginning to understand that stuff...

---

Work around: Looking at the documentation for yarn , I found a way to nail down dependencies to force all extensions being the same version, using a resolutions block. It's not ideal, but it seems to work. Here's the updated package.json. I just added all internal extensions to that block, that were published as part of the 0.3.7 Theia release, as well as @theia/application-package. Not all were required for this application, but this way it should still work if other extensions from the main Theia repo are added in dependencies:

{
    "private": true,
    "dependencies": {
        "@theia/typescript": "0.3.7",
        "@theia/navigator": "0.3.7",
        "@theia/terminal": "0.3.7",
        "@theia/outline-view": "0.3.7",
        "@theia/preferences": "0.3.7",
        "@theia/messages": "0.3.7",
        "@theia/git": "0.3.7",
        "@theia/file-search": "0.3.7",
        "@theia/markers": "0.3.7",
        "@theia/java": "0.3.7"
    },
    "devDependencies": {
        "@theia/cli": "0.3.7"
    },
    "resolutions": {
        "@theia/application-package": "0.3.7",
        "@theia/callhierarchy": "0.3.7",
        "@theia/core": "0.3.7",
        "@theia/cpp": "0.3.7",
        "@theia/editor": "0.3.7",
        "@theia/extension-manager": "0.3.7",
        "@theia/file-search": "0.3.7",
        "@theia/filesystem": "0.3.7",
        "@theia/git": "0.3.7",
        "@theia/java": "0.3.7",
        "@theia/keymaps": "0.3.7",
        "@theia/languages": "0.3.7",
        "@theia/markers": "0.3.7",
        "@theia/merge-conflicts": "0.3.7",
        "@theia/messages": "0.3.7",
        "@theia/metrics": "0.3.7",
        "@theia/monaco": "0.3.7",
        "@theia/navigator": "0.3.7",
        "@theia/outline-view": "0.3.7",
        "@theia/output": "0.3.7",
        "@theia/preferences": "0.3.7",
        "@theia/preview": "0.3.7",
        "@theia/process": "0.3.7",
        "@theia/python": "0.3.7",
        "@theia/search-in-workspace": "0.3.7",
        "@theia/task": "0.3.7",
        "@theia/terminal": "0.3.7",
        "@theia/typescript": "0.3.7",
        "@theia/userstorage": "0.3.7",
        "@theia/workspace": "0.3.7"
    }
}

[jopit]

Thanks for the workaround, it's now working.