rework build system #9710

paul-marechal · 2021-07-07T21:15:13Z

What it does

Fixes: #7275.

Simplify scripts/compile-references.js. TypeScript Server now properly
understand TypeScript references for navigation, so we can simplify our
tsconfig setup.

Rewrite some npm/yarn scripts to do less on install (yarn or yarn install). It will only run the required prepare scripts across our
packages, wire tsconfig.json files and build the TypeScript sources.

To build the example applications you need to run one of yarn browser build, yarn electron build or yarn build:examples.

Linting is done as part of CI, but it won't automatically run locally
anymore. You can run yarn lint to lint your code. For the best
experience you should also enable the ESLint extension in your IDE.

How to test

Installation:

yarn alone should not build the example applications anymore, nor should it lint.

Building:

yarn build should build TypeScript and both apps.
yarn compile should build TypeScript.
yarn browser build should build both TypeScript and the browser app.
yarn electron build should build both TypeScript and the electron app.
yarn build:examples should build both apps.

Watching:

yarn watch should watch TypeScript and both example apps.
yarn watch:compile should watch only TypeScript.
yarn browser watch should watch both TypeScript and the browser app.
yarn electron watch should watch both TypeScript and the electron app.

Linting:

yarn lint should lint TypeScript sources.

Review checklist

as an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

as a reviewer, I agree to behave in accordance with the review guidelines

paul-marechal · 2021-07-07T21:19:30Z

This is especially useful when using the main repo as a submodule for a different repository such as a Theia extension:

My approach was to clone Theia as a submodule in the extension repo, then add submodules/theia/packages/* and submodules/theia/dev-packages/* to the parent repo yarn workspaces. Finally, I would run the TypeScript compilation in Theia to simply build the TypeScript packages, as this is all that is required for dependents to consume.

yarn link doesn't play nicely with our Theia building tools for some reason, hence why this alternative approach is interesting.

This PR also makes developing a bit more comfortable: only the minimal amount of work required is done, if you need more you need to explicitly trigger it. E.g. building a given app, linting, etc...

doc/Developing.md

colin-grant-work · 2021-07-09T16:29:15Z

This is especially useful when using the main repo as a submodule for a different repository such as a Theia extension

This sounds like it goes some way to addressing some of the pain points described in #9153, as well.

tsmaeder · 2021-07-12T07:34:14Z

yarn link doesn't play nicely with our Theia building tools for some reason, hence why this alternative approach is interesting.

Could you detail in what way it does not play well with yarn link? it seems to work for me, but I fear I'm missing something.

tsmaeder · 2021-07-12T07:40:30Z

doc/Developing.md


 ## Watching

+### Watch the TypeScript packages


I'm not sure about the term "typescript packages". What would a non-typescript package be, for example?

This mostly refers to TypeScript compilation which happens before bundling, so if we have a pure JS extension then there will be nothing to watch.

I understand TypeScript package as a package written using TypeScript and requiring transpilation.

So it's really "typescript files", because some packages may container both *.js and *.ts files, right? Also, not sure how that works, exactly, but don't we have to reload the code when *.js files change, as well?

In Theia the watch scripts are related to code transformations. There's basically two steps: TypeScript compilation and Webpack bundling. This documentation refers to the former. JavaScript sources don't require any change during the first step, but bundling still depends on them indeed.

examples/browser/package.json

scripts/compile-references.js

package.json

packages/callhierarchy/tsconfig.json

scripts/compile-references.js

package.json

paul-marechal · 2021-07-12T16:40:10Z

Could you detail in what way it does not play well with yarn link? it seems to work for me, but I fear I'm missing something.

All the times I remember doing yarn link in each Theia package for consumption in third-party Theia extension projects, it ended up with Webpack getting confused and duplicating packages in the output bundles for some reason, leading to runtime errors.

Since we migrated to Webpack 5 maybe this won't be an issue anymore...

So independently from that, this PR still simplifies the compile-references.js script, allows one to manage dependencies without triggering a full build, and cleans up scripts.

tsmaeder · 2021-07-13T07:21:53Z

doc/Developing.md


 ### Watch the core and extension packages

-To rebuild each time a change is detected run:
+To rebuild _everything_ each time a change is detected run:


"Everything" being both the regular packages and the example applications?

Yes, both steps: TypeScript compilation and the bundling of both examples.

msujew · 2021-08-18T22:35:05Z

@paul-marechal I'll review this, if you're still interested in getting this merged. Would you mind rebasing and fixing merge conflicts?

paul-marechal · 2021-08-20T03:58:29Z

@msujew thanks for proposing! I just rebased. ~~Note that one of the test lifecycle keeps failing and I don't know why...~~

edit: I forgot to download the plugins, fixed.

msujew

I really like the proposed changes, especially the newer watch options and the removal of yarn lint from every build.

I also have an addition to propose: In my usual workflow when working on Theia I needed to start the server/electron app and watch both typescript and webpack. The latter two are now merged using yarn browser watch and yarn electron watch. I would like to see the former in there as well. Something like this:

"scripts": {
    "start:watch": "concurrently --kill-others -n tsc,bundle,run -c red,yellow,green \"tsc -b -w --preserveWatchOutput\" \"yarn watch:bundle\" \"yarn start\""
}

What's your opinion on that?

doc/Developing.md

Co-authored-by: Mark Sujew <mark.sujew@typefox.io>

Sort scripts by name for root, browser and electron packages. Download plugins as part of `build:examples`. Cleanups.

msujew

@paul-marechal Still fine with merging, everything looks good 👍

paul-marechal · 2021-09-21T14:17:03Z

I'll just wait on CI to turn green...

Follow up from #9710 When doing refactorings, one might end up renaming/deleting TypeScript source files. But if a build was run previously, the outputted JavaScript files will still be in the build output folder. Then extensions that depend on the refactored package will not fail to build because the old path still resolves to the lingering JavaScript files. Add `ts-clean` script that deletes all generated files without a corresponding source file. Add the missing compile script to most packages.

thegecko · 2021-10-01T19:22:27Z

Really miss the easy yarn start:browser command in the root to ensure native modules are built correctly and start the browser example.

msujew · 2021-10-01T19:51:38Z

@thegecko I think you should be able to run yarn browser start to do exactly the same as the old yarn start:browser.

thegecko · 2021-10-10T12:43:06Z

I think you should be able to run yarn browser start

Confirmed, thanks!

paul-marechal requested review from tsmaeder, vince-fugnitto and colin-grant-work July 7, 2021 21:15

paul-marechal force-pushed the mp/build-clean branch 3 times, most recently from 618b1c2 to 0762ab1 Compare July 7, 2021 21:36

colin-grant-work reviewed Jul 9, 2021

View reviewed changes

doc/Developing.md Outdated Show resolved Hide resolved

tsmaeder reviewed Jul 12, 2021

View reviewed changes

paul-marechal force-pushed the mp/build-clean branch 3 times, most recently from 5abd78f to 4f80f9f Compare July 12, 2021 22:07

tsmaeder reviewed Jul 13, 2021

View reviewed changes

paul-marechal force-pushed the mp/build-clean branch from 146f79e to f899f7f Compare August 20, 2021 03:19

paul-marechal force-pushed the mp/build-clean branch from 30ff600 to 637ea3b Compare August 27, 2021 04:37

msujew reviewed Aug 27, 2021

View reviewed changes

doc/Developing.md Outdated Show resolved Hide resolved

paul-marechal force-pushed the mp/build-clean branch 9 times, most recently from fa34aff to 4d25a63 Compare August 28, 2021 01:46

paul-marechal and others added 12 commits September 20, 2021 17:56

fix baseline

b50de4e

doc fixup

9b1b846

add start:watch script in example apps

3b84de7

Update .gitpod.yml

b0ced2c

Co-authored-by: Mark Sujew <mark.sujew@typefox.io>

fix electron rebuild logic

db0b234

update license baseline

f155485

revert electron-rebuild version

44709d7

devfile: download plugins

7d5fc6a

fix api-test

79c52b1

revert changelog

a841dcd

fix doc

7812328

cleanup

02aa2e4

paul-marechal force-pushed the mp/build-clean branch from cbbb102 to 02aa2e4 Compare September 20, 2021 21:57

cleanup

a3cb5e0

Sort scripts by name for root, browser and electron packages. Download plugins as part of `build:examples`. Cleanups.

paul-marechal force-pushed the mp/build-clean branch from c55af33 to a3cb5e0 Compare September 20, 2021 22:30

fixup

fa89ff1

msujew approved these changes Sep 21, 2021

View reviewed changes

paul-marechal merged commit 23e09c2 into master Sep 21, 2021

github-actions bot added this to the 1.18.0 milestone Sep 21, 2021

paul-marechal mentioned this pull request Sep 21, 2021

New commands to build this repository #10145

Closed

azatsarynnyy mentioned this pull request Oct 5, 2021

Restore the naming of the published npm packages to be suffixed with a commit sha #10233

Closed

msujew mentioned this pull request Oct 8, 2021

Module did not self-register when command yarn start:electron #10246

Open

paul-marechal deleted the mp/build-clean branch October 10, 2021 15:10

tom-shan mentioned this pull request Nov 5, 2021

Remove redundant config files #10386

Closed

colin-grant-work mentioned this pull request Apr 14, 2022

Export page objects directly under lib #11023

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rework build system #9710

rework build system #9710

paul-marechal commented Jul 7, 2021 •

edited

Loading

paul-marechal commented Jul 7, 2021

colin-grant-work commented Jul 9, 2021 •

edited

Loading

tsmaeder commented Jul 12, 2021

tsmaeder Jul 12, 2021

paul-marechal Jul 12, 2021

paul-marechal Jul 12, 2021

tsmaeder Jul 13, 2021

paul-marechal Jul 13, 2021 •

edited

Loading

paul-marechal commented Jul 12, 2021

tsmaeder Jul 13, 2021

paul-marechal Jul 13, 2021

msujew commented Aug 18, 2021

paul-marechal commented Aug 20, 2021 •

edited

Loading

msujew left a comment

msujew left a comment

paul-marechal commented Sep 21, 2021

thegecko commented Oct 1, 2021

msujew commented Oct 1, 2021

thegecko commented Oct 10, 2021

rework build system #9710

rework build system #9710

Conversation

paul-marechal commented Jul 7, 2021 • edited Loading

What it does

How to test

Review checklist

Reminder for reviewers

paul-marechal commented Jul 7, 2021

colin-grant-work commented Jul 9, 2021 • edited Loading

tsmaeder commented Jul 12, 2021

tsmaeder Jul 12, 2021

Choose a reason for hiding this comment

paul-marechal Jul 12, 2021

Choose a reason for hiding this comment

paul-marechal Jul 12, 2021

Choose a reason for hiding this comment

tsmaeder Jul 13, 2021

Choose a reason for hiding this comment

paul-marechal Jul 13, 2021 • edited Loading

Choose a reason for hiding this comment

paul-marechal commented Jul 12, 2021

tsmaeder Jul 13, 2021

Choose a reason for hiding this comment

paul-marechal Jul 13, 2021

Choose a reason for hiding this comment

msujew commented Aug 18, 2021

paul-marechal commented Aug 20, 2021 • edited Loading

msujew left a comment

Choose a reason for hiding this comment

msujew left a comment

Choose a reason for hiding this comment

paul-marechal commented Sep 21, 2021

thegecko commented Oct 1, 2021

msujew commented Oct 1, 2021

thegecko commented Oct 10, 2021

paul-marechal commented Jul 7, 2021 •

edited

Loading

colin-grant-work commented Jul 9, 2021 •

edited

Loading

paul-marechal Jul 13, 2021 •

edited

Loading

paul-marechal commented Aug 20, 2021 •

edited

Loading