Support additional libs via compiler API #3863

kitsonk · 2020-02-03T01:22:48Z

~~Still very much a work in progress, but I got so excited when I got it working.~~ The following compiles without errors in Deno:

const [errors, program] = await Deno.compile(
  "main.ts",
  {
    "main.ts": `document.getElementById("foo");`
  },
  {
    lib: ["dom", "esnext"]
  }
);

~~It will lazily load the lib.dom.d.ts which matches the version of TypeScript bundled with Deno from github.com/microsoft/TypeScript and cache it (as it would any other remote module).~~

Instead of lazily loading them, we built them into the Deno binary.

Pandawan · 2020-02-03T22:59:21Z

Awesome!

If you wanted to include an extra library in addition to Deno's declarations, would you need to include lib: ["esnext", "deno"]? Or is Deno imported automatically in all cases?

Also, I think that's the case but just to make sure, this PR also provides support for /// <reference lib="dom" /> right?

kitsonk · 2020-02-04T00:36:41Z

If you wanted to include an extra library in addition to Deno's declarations, would you need to include lib: ["esnext", "deno"]? Or is Deno imported automatically in all cases?

Actually, lib: ["esnext", "deno.ns"]. I will update the manual with information about how to use these capabilities. I need to double check that the triple slash reference of the deno.ns works too, as that would be the easiest way to ensure that code work work seamlessly between the two.

Also, I think that's the case but just to make sure, this PR also provides support for /// right?

Yes, though if you tried to deno run/deno bundle a program that includes a /// <reference lib="dom" /> it would error, because lib.dom.d.ts conflicts with some of the built in global namespace of Deno. It essentially would only be useful with the runtime compiler APIs.

kitsonk · 2020-02-08T09:55:34Z

Hrm... the change in the thread pools I think has caused a bit of a problem. This has to resolve the remote module synchronously in TS land, though it is async in Rust. Need to figure out how to do that.

kitsonk · 2020-02-08T10:15:20Z

Hrmmmm... I suspect I am going to need some help, because the debug log is showing that it is hanging while connecting to the remote server to fetch the remote lib file. I suspect that because of 161cf7c something has changed while trying to do main -> worker -> async op which is essentially blocking infinitely because there isn't a "free" thread:

❯ deno -L debug --reload cli/tests/lib_ref.ts
DEBUG RS - deno::startup_data:26 - Deno isolate init with snapshots.
DEBUG RS - deno_core::shared_queue:67 - rust:shared_queue:reset
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=303
DEBUG JS - cwd /Users/kkelly/github/deno
DEBUG JS - args []
DEBUG RS - deno:383 - main_module file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts
DEBUG RS - deno::file_fetcher:183 - fetch_source_file_async specifier: file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts 
Compile file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts
DEBUG RS - deno::compilers::ts:613 - >>>>> compile_async START
DEBUG RS - deno::startup_data:52 - Deno isolate init with snapshots.
DEBUG RS - deno_core::shared_queue:67 - rust:shared_queue:reset
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=305
DEBUG TS - getMessage
DEBUG RS - deno::ops::web_worker:34 - op_worker_get_message
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=486
DEBUG RS - deno_core::shared_queue:177 - rust:shared_queue:pre-push: op=7, off=812, end=1300, len=488
DEBUG RS - deno_core::shared_queue:196 - rust:shared_queue:push: num_records=1, num_shifted_off=0, head=1300
DEBUG TS - >>> compile start { rootNames: [ "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts" ], type: "Compile" }
DEBUG TS - compiler_imports::resolveModules { specifiers: [ "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts" ], referrer: undefined }
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=81
DEBUG TS - compiler_imports::fetchSourceFiles { specifiers: [ "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts" ], referrer: undefined }
DEBUG RS - deno::file_fetcher:183 - fetch_source_file_async specifier: file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts 
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=487
DEBUG RS - deno_core::shared_queue:177 - rust:shared_queue:pre-push: op=4, off=812, end=1300, len=488
DEBUG RS - deno_core::shared_queue:196 - rust:shared_queue:push: num_records=1, num_shifted_off=0, head=1300
DEBUG TS - compiler::host.getCompilationSettings()
DEBUG TS - compiler::host.getDefaultLibFileName()
DEBUG TS - compiler::host.getSourceFile file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.deno.window.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.deno.ns.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.deno.shared_globals.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.esnext.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2020.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2019.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2016.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es5.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.core.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.collection.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.generator.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.iterable.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.symbol.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.promise.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.proxy.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.reflect.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2015.symbol.wellknown.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2016.array.include.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.object.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.sharedmemory.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.string.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.intl.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2017.typedarrays.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.asyncgenerator.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.asynciterable.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.promise.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.regexp.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2018.intl.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2019.array.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2019.object.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2019.string.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2019.symbol.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2020.string.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.es2020.symbol.wellknown.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.esnext.bigint.d.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.esnext.intl.d.ts
DEBUG TS - compiler::host.writeFile $deno$/lib_ref.js.map
DEBUG TS - compiler::host.getCompilationSettings()
DEBUG TS - compiler::cache { moduleId: "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts", emittedFileName: "$deno$/lib_ref.js.map", checkJs: false }
DEBUG RS - deno::fs:53 - set file perm to 438
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=26
DEBUG TS - compiler::host.writeFile $deno$/lib_ref.js
DEBUG TS - compiler::host.getCompilationSettings()
DEBUG TS - compiler::cache { moduleId: "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts", emittedFileName: "$deno$/lib_ref.js", checkJs: false }
DEBUG RS - deno::fs:53 - set file perm to 438
DEBUG RS - deno::fs:53 - set file perm to 438
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=26
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=26
DEBUG TS - <<< compile end { rootNames: [ "file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts" ], type: "Compile" }
DEBUG RS - deno::compilers::ts:630 - >>>>> compile_sync END
DEBUG RS - deno::compilers::ts:441 - compiled filename: "/Users/kkelly/Library/Caches/deno/gen/file/Users/kkelly/github/deno/cli/tests/lib_ref.ts.js"
DEBUG RS - deno_core::modules:314 - register_complete file:///Users/kkelly/github/deno/cli/tests/lib_ref.ts
DEBUG JS - Deno.compile { rootName: "main.ts", sources: true, options: { target: "es2018", lib: [ "es2018", "deno.ns" ] } }
DEBUG RS - deno::compilers::ts:613 - >>>>> compile_async START
DEBUG RS - deno::startup_data:52 - Deno isolate init with snapshots.
DEBUG RS - deno_core::shared_queue:67 - rust:shared_queue:reset
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=305
DEBUG TS - getMessage
DEBUG RS - deno::ops::web_worker:34 - op_worker_get_message
DEBUG RS - deno::ops::dispatch_json:36 - JSON response pre-align, len=862
DEBUG RS - deno_core::shared_queue:177 - rust:shared_queue:pre-push: op=7, off=812, end=1676, len=864
DEBUG RS - deno_core::shared_queue:196 - rust:shared_queue:push: num_records=1, num_shifted_off=0, head=1676
DEBUG TS - >>> runtime compile start { rootName: "main.ts", bundle: false, sources: [ "main.ts" ] }
DEBUG TS - compiler::host.getCompilationSettings()
DEBUG TS - compiler::host.getDefaultLibFileName()
DEBUG TS - compiler::host.getSourceFile main.ts
DEBUG TS - compiler::host.getSourceFile $asset$/lib.dom.d.ts
DEBUG TS - compiler_util::getAsset lib.dom.d.ts
DEBUG RS - deno::ops::compiler:174 - args.name: lib.dom.d.ts
DEBUG RS - deno::file_fetcher:171 - fetch_remote_asset_async name: lib.dom.d.ts 
DEBUG RS - deno::file_fetcher:183 - fetch_source_file_async specifier: https://raw.githubusercontent.com/microsoft/TypeScript/v3.7.2/lib/lib.dom.d.ts 
Download https://raw.githubusercontent.com/microsoft/TypeScript/v3.7.2/lib/lib.dom.d.ts
DEBUG RS - hyper::client::connect::dns:121 - resolving host="raw.githubusercontent.com"
DEBUG RS - hyper::client::connect::http:532 - connecting to 151.101.192.133:443

bartlomieju

@kitsonk I'm trying to debug the problem, I'll get back when I find something.

std/manual.md

bartlomieju · 2020-02-09T11:28:08Z

@kitsonk I'm afraid the simplest/quickest/dirtiest solution would be to use reqwest's sync Client to perform requests for type declarations. The other option is to put FileFetcher on a separate thread...

kitsonk · 2020-02-09T19:57:45Z

@bartlomieju well, it is wrapped up in the whole of the file_fetcher API to take advantage of the caching and the like, so using sync Client doesn't seem viable.

I will take a look at putting FileFetcher in a seperate thread, though my knowledge is very limited, any suggestions/pointers appreciated...

nayeemrmn · 2020-02-09T19:59:51Z

std/manual.md

@@ -174,6 +174,12 @@ command line:
 $ deno types
 ```

+The file is the concatenation of three library files that are built into Deno:


s/file/output, remove previous link.

The old link is still there.

bartlomieju · 2020-02-09T20:00:48Z

@kitsonk just use tokio_util::spawn_thread as a workaround for that. Please add a TODO with my handle. I'll be looking into better path handling in file fetcher/cache so I might tackle this problem as well.

kitsonk · 2020-02-10T00:38:15Z

Actually, once I rebased on master and the changes seem to have unblocked the thread. So hoping the CI works now, as it is working locally.

kitsonk · 2020-02-10T01:05:48Z

Crap, no, the file was in my cache, and I forgot --reload on my local machine... back to the drawing board.

kitsonk · 2020-02-10T05:33:15Z

@bartlomieju that effectively didn't work... I got something like this when using spawn_thread:

thread '<unnamed>' panicked at 'not currently running on the Tokio runtime.'

I also tried several variations of tokio::task::spawn and always blocked somewhere else...

I've cleaned up the code in the op a little bit, and it is a bit more clear, but still get stuck at the same point.

bartlomieju · 2020-02-11T09:59:22Z

@kitsonk here's a hacky patch that will get you forward:

diff --git a/cli/ops/compiler.rs b/cli/ops/compiler.rs
index 546e081f..a784d7d2 100644
--- a/cli/ops/compiler.rs
+++ b/cli/ops/compiler.rs
@@ -2,10 +2,12 @@
 use super::dispatch_json::Deserialize;
 use super::dispatch_json::JsonOp;
 use super::dispatch_json::Value;
+use crate::file_fetcher::SourceFile;
 use crate::futures::future::try_join_all;
 use crate::msg;
 use crate::ops::json_op;
 use crate::state::State;
+use crate::tokio_util;
 use deno_core::Loader;
 use deno_core::*;
 
@@ -173,14 +175,19 @@ fn op_fetch_remote_asset(
 ) -> Result<JsonOp, ErrBox> {
   let args: FetchRemoteAssetArgs = serde_json::from_value(args)?;
   debug!("args.name: {}", args.name);
-
+  let (sender, receiver) =
+    std::sync::mpsc::sync_channel::<Result<SourceFile, ErrBox>>(1);
   let global_state = state.borrow().global_state.clone();
-  let source_file = futures::executor::block_on(
-    global_state
+  // FIXME(bartlomieju): shouldn't spawn new thread
+  std::thread::spawn(move || {
+    let mut rt = tokio_util::create_basic_runtime();
+    let fut = global_state
       .file_fetcher
-      .fetch_remote_asset_async(&args.name),
-  )
-  .unwrap();
+      .fetch_remote_asset_async(&args.name);
+    let res = rt.block_on(fut);
+    sender.send(res).unwrap();
+  });
+  let source_file = receiver.recv().unwrap()?;
   let out = json!({
     "url": source_file.url.to_string(),
     "filename": source_file.filename.to_str().unwrap(),

It spawns new thread for each remote assets - but I guess we can get away with it for a while. Like I said earlier I'll be refactoring file fetcher for path handling so I can fix this problem as well.

kitsonk · 2020-02-14T05:41:44Z

Phew, finally this works! ping @ry

std/manual.md

kitsonk · 2020-02-17T21:14:31Z

Let's see what it look like if we build it in. If we require the full URL, then that is even more TypeScript code that will break under tsc, which doesn't do us any favours.

Fixes denoland#3726 This PR provides support for referencing other lib files that are not used by default in Deno via the compiler APIs.

ry · 2020-02-18T23:19:30Z

any idea what the delta on size of target/release/deno is?

kitsonk · 2020-02-19T00:41:59Z

This PR:

-rwxr-xr-x  2 kkelly  staff  56665972 19 Feb 11:40 target/release/deno

Master:

-rwxr-xr-x  2 kkelly  staff  54870092 19 Feb 11:34 target/release/deno

I tried the include-flate crate and it works, but the macro only accepts string literals and a lot of our files need concat!() of some of the env!() stuff and I couldn't easily figure out how to get it to work. Even then I was able to compress typescript.js and it didn't appear to save much if any in the debug build, and ultimately it eats up memory (because you have the compressed and uncompressed in memory at the same time).

Most of the assets (including typescript.js) are not needed at runtime, because they are built into the snapshot and never requested after the build. I don't know if there is an easy way to drop them from being inlined into the binary.

ry · 2020-02-19T00:57:33Z

This is painful, but I think it's better than downloading at runtime. I'll look into compressing stuff more later.

kitsonk · 2020-02-19T01:05:45Z

I tried looking at following the pattern in include-flate, I suspect we could embed the included bytes in a compressed format, and when requested, expand them on the fly, but it was becoming non-trivial.

ry

LGTM

Fixes denoland/deno#3726 This PR provides support for referencing other lib files (like lib.dom.d.ts that are not used by default in Deno.

kitsonk force-pushed the external-lib branch from 3c8c870 to fc7eb9f Compare February 4, 2020 09:53

kitsonk changed the title ~~[WIP] Support remote libs~~ Support remote libs Feb 4, 2020

kitsonk requested review from bartlomieju and ry February 4, 2020 09:55

kitsonk force-pushed the external-lib branch 3 times, most recently from b933e8b to 47b022b Compare February 8, 2020 01:48

bartlomieju reviewed Feb 8, 2020

View reviewed changes

std/manual.md Outdated Show resolved Hide resolved

kitsonk mentioned this pull request Feb 9, 2020

Outdated paths in std/manual.md #3939

Closed

nayeemrmn reviewed Feb 9, 2020

View reviewed changes

kitsonk force-pushed the external-lib branch from 47b022b to 1883dab Compare February 10, 2020 00:36

kitsonk force-pushed the external-lib branch 3 times, most recently from 2761553 to c9ed8d7 Compare February 14, 2020 04:19

kitsonk force-pushed the external-lib branch from d0c6134 to d905a6f Compare February 15, 2020 20:05

ry reviewed Feb 17, 2020

View reviewed changes

std/manual.md Outdated Show resolved Hide resolved

Support loading additional TS lib files

2b111e6

Fixes denoland#3726 This PR provides support for referencing other lib files that are not used by default in Deno via the compiler APIs.

kitsonk force-pushed the external-lib branch from d905a6f to 2b111e6 Compare February 18, 2020 22:46

kitsonk changed the title ~~Support remote libs~~ Support additional libs via compiler API Feb 18, 2020

ry approved these changes Feb 19, 2020

View reviewed changes

ry merged commit 046bbb2 into denoland:master Feb 19, 2020

kitsonk deleted the external-lib branch February 19, 2020 05:44

devvaneio mentioned this pull request May 19, 2020

Deno bundle doesn't include the dependencies properly #5610

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support additional libs via compiler API #3863

Support additional libs via compiler API #3863

kitsonk commented Feb 3, 2020 •

edited

Loading

Pandawan commented Feb 3, 2020

kitsonk commented Feb 4, 2020

kitsonk commented Feb 8, 2020

kitsonk commented Feb 8, 2020

bartlomieju left a comment

bartlomieju commented Feb 9, 2020

kitsonk commented Feb 9, 2020

nayeemrmn Feb 9, 2020

nayeemrmn Feb 18, 2020

bartlomieju commented Feb 9, 2020

kitsonk commented Feb 10, 2020

kitsonk commented Feb 10, 2020

kitsonk commented Feb 10, 2020

bartlomieju commented Feb 11, 2020

kitsonk commented Feb 14, 2020

kitsonk commented Feb 17, 2020

ry commented Feb 18, 2020

kitsonk commented Feb 19, 2020

ry commented Feb 19, 2020

kitsonk commented Feb 19, 2020

ry left a comment

Support additional libs via compiler API #3863

Support additional libs via compiler API #3863

Conversation

kitsonk commented Feb 3, 2020 • edited Loading

Pandawan commented Feb 3, 2020

kitsonk commented Feb 4, 2020

kitsonk commented Feb 8, 2020

kitsonk commented Feb 8, 2020

bartlomieju left a comment

Choose a reason for hiding this comment

bartlomieju commented Feb 9, 2020

kitsonk commented Feb 9, 2020

nayeemrmn Feb 9, 2020

Choose a reason for hiding this comment

nayeemrmn Feb 18, 2020

Choose a reason for hiding this comment

bartlomieju commented Feb 9, 2020

kitsonk commented Feb 10, 2020

kitsonk commented Feb 10, 2020

kitsonk commented Feb 10, 2020

bartlomieju commented Feb 11, 2020

kitsonk commented Feb 14, 2020

kitsonk commented Feb 17, 2020

ry commented Feb 18, 2020

kitsonk commented Feb 19, 2020

ry commented Feb 19, 2020

kitsonk commented Feb 19, 2020

ry left a comment

Choose a reason for hiding this comment

kitsonk commented Feb 3, 2020 •

edited

Loading