Explore how to implement the ergonomic dynamic import

[mununki]

Related #5593

• Emit the inlined js module path instead of import statement in the header

---

Thanks to JSX v4, now we can write the dynamic import for react component:

// LazyComp.res
@react.component
let make = (~x) => React.string(x)

// App.res
@val external import: 'a => Js.Promise.t<'a> = "import"
type option = {ssr?: bool}
@val external dynamic: (unit => Js.Promise.t<'a>, option) => 'a = "dynamic"

module L = {
  let make = dynamic(() => import(LazyComp.make), {ssr: false})
}

<L x="lazy" />

This example passes the type checker as we intended, but the import statement in js output should be emitted differently.

As-is:

import * as LazyComp from "./LazyComp.bs.js";

var make = dynamic((function (param) {
        return import(LazyComp.make);
      }), {
      ssr: false
    });

To-be:

// omit the import statement
var make = dynamic((function (param) {
        return import("./LazyComp.bs.js"); // <-- inlined
      }), {
      ssr: false
    });

[cristianoc]

Is this the (or a) idiomatic way to dynamically load a component?
return import("./LazyComp.bs.js")

[cristianoc]

Looking randomly at TS resources, here's an example:

async function renderWidget() {
  const container = document.getElementById("widget");
  if (container !== null) {
    const widget = await import("./widget");
    widget.render(container);
  }
}

renderWidget();

Looks like pretty much the same example should be expressible.
Which means, the magic would be in the treatment of import which would need to be provided by the compiler, e.g. Js.import or something.

[cristianoc]

I guess Js.import or however it's called, would take a value, under some restrictions, figure out where it is, and emit a dynamic import instruction.

[cristianoc]

@mattdamon108 just a high level brain dump here.
A couple of discrepancies so far:

• dynamic import is about modules/files, not values
• dynamic import conceptually only belongs in an async context

Making up random syntax etc, but this is how I would imagine the mechanism should instead behave at a high level:

let foo = async (x) => {
  switch x {
  | A =>
    module Lib = await LibA // dynamically load module LibA
    Lib.someFunction("aaa")
  | B =>
    module Lib = await LibB // dynamically load module LibB
    Lib.someFunction("bbb")
  }
}

[mununki]

dynamic import is about modules/files, not values

If we're going to implement import not only for react component but also for general purpose, yes it is correct that import is about modules and files not values.

dynamic import conceptually only belongs in an async context

Why should import only belong in an async context?

[mununki]

Is it valid?

let foo = () => {
  let m = LibA
  m
}

[mununki]

How about this without the constraint of being inside async context?

// in async context
let foo = async (x) => {
  switch x {
  | A =>
    module Lib = await @import LibA // dynamically load module LibA
    Lib.someFunction("aaa")
  | B =>
    module Lib = await @import LibB // dynamically load module LibB
    Lib.someFunction("bbb")
  }
}

// not in async
let foo = (x) => {
  switch x {
  | A =>
    let ma = @import LibA // dynamically load module LibA
    ma
  | B =>
    let mb = @import LibB // dynamically load module LibB
    mb
  }
}

[cristianoc]

First try to fix your initial example. The code does not extract make.
Then look at the result.

[mununki]

Yes, it doesn't extract make, because the module doesn't have an export default.

[mununki]

Do we have a feature to make an export default BTW?

@default let m = comp // something like?

[cristianoc]

Do we have a feature to make an export default BTW?

@default let m = comp // something like?

That's a separate question.
It's nice when you have it, but you can't assume.

It's "let default = ..."

[cristianoc]

Yes, it doesn't extract make, because the module doesn't have an export default.

Let's consider the case where make needs to be extracted, which is a common case at the moment.
Can you try to write a fix for the generated code for the initial example at the top of this issue, see how it looks like?

[cristianoc]

How about this without the constraint of being inside async context?

// in async context
let foo = async (x) => {
  switch x {
  | A =>
    module Lib = await @import LibA // dynamically load module LibA
    Lib.someFunction("aaa")
  | B =>
    module Lib = await @import LibB // dynamically load module LibB
    Lib.someFunction("bbb")
  }
}

// not in async
let foo = (x) => {
 switch x {
 | A =>
   let ma = @import LibA // dynamically load module LibA
   ma
 | B =>
   let mb = @import LibB // dynamically load module LibB
   mb
 }
}

The second example tries to assign a module to a value variable ma or mb. That's not part of the language. (One can look at first-class modules, though they are quite complex and need an explicit signature).
Also, not clear if it returns a promise or not. If it does not, then you just lost the ability to control when the module is loaded, from the point of view of the caller of foo.

[mununki]

Let's consider the case where make needs to be extracted, which is a common case at the moment. Can you try to write a fix for the generated code for the initial example at the top of this issue, see how it looks like?

Sure, I'll.
My intention was import(Foo.make) helps to infer the props type of react component. Now I realize that import(Foo.make) doesn't match the semantic against js import.

[cristianoc]

Let's consider the case where make needs to be extracted, which is a common case at the moment. Can you try to write a fix for the generated code for the initial example at the top of this issue, see how it looks like?

Sure, I'll. My intention was import(Foo.make) helps to infer the props type of react component. Now I realize that import(Foo.make) doesn't match the semantic against js import.

The point I'm trying to make is that if the generated code needs to extract make, then it needs to extract it from a promise, and presumably package it back into a promise.

[cristianoc]

The fact that you need to extract a value from a promise and package it back into a promise: that's what I mean when I say that conceptually dynamic loading belongs to an async context.

As an async context guides you do do exactly that without you even having to think about it.

[mununki]

As an async context guides you do do exactly that without you even having to think about it.

It makes sense.

[mununki]

Let me check quickly to see if it works.
If you allow using the dynamic import only in the async context, the generated output would be like this:
Correct?

var make = dynamic((function async (param) {
        let m = await import("./LazyComp.bs.js");
        return m.make
      }), {
      ssr: false
    });

[cristianoc]

Let me check quickly to see if it works. If you allow using the dynamic import only in the async context, the generated output would be like this: Correct?

var make = dynamic((function async (param) {
        let m = await import("./LazyComp.bs.js");
        return m.make
      }), {
      ssr: false
    });

That seems correct except, I don't know what the dynamic external is and what it expects.
Can you explain what it does and where it is coming from?

[mununki]

Side comment: this async import in ES6 is being used in many contexts.

React.lazy

const OtherComponent = React.lazy(() => import('./OtherComponent'));

Next.js

const DynamicHeader = dynamic(() => import('../components/header'), {
  suspense: true,
})

[mununki]

That seems correct except, I don't know what the dynamic external is and what it expects.
Can you explain what it does and where it is coming from?

dynamic from the Next.js https://nextjs.org/docs/advanced-features/dynamic-import#example

[mununki]

Therefore I had a bit frustration to make import only in the async context, but async/await is just a syntax sugar of Promise. So, that would be no problem, but just make sure if it works fine in runtime.

[cristianoc]

That seems correct except, I don't know what the dynamic external is and what it expects.
Can you explain what it does and where it is coming from?

dynamic from the Next.js https://nextjs.org/docs/advanced-features/dynamic-import#example

What's the actual type of that?

const DynamicHeader = dynamic(() => import('../components/header'), {
  suspense: true,
})

it seems to me it takes a function that returns a promise. Correct?
(Then somehow wraps the promise into the component and you don't see it again)

[cristianoc]

Therefore I had a bit frustration to make import only in the async context, but async/await is just a syntax sugar of Promise. So, that would be no problem, but just make sure if it works fine in runtime.

The examples I see in this answer do a lot of promise chaining:
https://stackoverflow.com/questions/54150819/using-react-lazy-with-typescript

So that seems to confirm that it's conceptually in an async context. When I say conceptually I don't mean necessarily that it will be used via async-await, just that async-await seems a natural way to express it.

The fact that even when you hide the promise inside React.lazy sometimes you need to do promise chaining (or at least people in that stack overflow answer think you do) seems interesting.

Basically I would like to explore the basic building blocks needed to support dynamic loading in the language, in general. Whether or not you're using some lazy UI framework.

[cristianoc]

Separately, there's the question of how to use Next and React with the least friction possible.

[cristianoc]

Essentially, the difficulty has to do with the language and its treatment of modules as separate entities from values.
To begin, let's take a module C which is a component, so it has a function C.make. We want to load it dynamically. Let's call this import.

The first question is: what is import(C)?
If it is a module, then we have the problem that you can't wrap a module into a promise, as promises in the language only take values.
Using async/await is one way to get out of the problem: by combining await import(C) you get back a module, and this fits in the language.

Without using async/await one needs to be a bit creative.
One possibility is returning a module with modified entries in it.
E.g. import(C).getMake could be a function that returns a promise. E.g. used as dynamic(() => import(C).getMake()).
The difficulty here is that if you want to turn import(C) back into a module that you can use just like C, then you need to extract make and then wrap it back into another module.

So I guess these are essentially the 2 kinds of ways of doing it, each with their own trade-offs.

[cristianoc]

@mattdamon108 thoughts about ^ ?

[mununki]

Sorry, I'm back now. (serial meetings..)

I think we are very closely standing on the same page now. I think we can start with the small step, that is import(C). If we can make a syntax of it, later we can figure out more context from usages. I agree not to constrain the import into only async context that's why.

The first question is: what is import(C)?

This is a very good question. import(C) is a Promise, which has a js module inside in case of being fulfilled. As you already investigated, the user might expect to use it inside the async context to extract something from the js module or use it without extracting.

In terms of language, how about this?

let p0 = import(C.make) // C.make is a value. The semantic seems too different from the js?

let p1 = await import(C.foo) // C.bar is a value too

transformed to

var p0 = import("path of C").then(m => m.make)

var c = await import("path of C")
var p1 = c.bar

[mununki]

I believe import("path of C").then(m => m.make) would work fine at least with React.lazy and next/dynamic.

[mununki]

The reason I don't suggest import(C) is that the type information would be too ambiguous and opaque for users. import(C) seems more likely to js semantic, but I think we can start with import(C.value) as a first step.

[zth]

In that case I guess import(Component.make) would be sugar for importing the file with Component, and then resolving with make? So the generated code might look something like:

import("../Component.bs.js").then(c => c.make)

From a cursory search, the runtime might need to generate different things depending on what module system is in use, but basically yes.

I really like that. I think that the JS way of importing an entire module is just a consequence of dynamic import working only on files, not because it's the best possible API. So if we can support both importing the module itself, and importing a single value, I think we'll have great ergonomics. I imagine it's probably at least as common to reach for a single value rather than the full module when importing. Having to refer to the full module in JS, just to then remap it to what you're really after, seems mostly like (unnecessary) manual labor.

[mununki]

Additionally, this syntax needs to work only in "package-specs": "module": "es6"

[cristianoc]

Additionally, this syntax needs to work only in "package-specs": "module": "es6"

If that's the assumption, it surely simplifies the back-end.
Not sure what's the current status, but if most people are using es6 and the percentage increases over time, then that's OK.

[mununki]

The import killer syntax will make users moving over to es6 😄

[cristianoc]

@mattdamon108 here's a skeleton to begin playing with, if you're interested in exploring further:
#5701

[mununki]

@mattdamon108 here's a skeleton to begin playing with, if you're interested in exploring further:
#5701

I've investigated how to implement this before by myself, I'll follow your work and learn from it. Thanks for letting me know!

[cristianoc]

@mattdamon108 here's a skeleton to begin playing with, if you're interested in exploring further:
#5701

I've investigated how to implement this before by myself, I'll follow your work and learn from it. Thanks for letting me know!

Great, take what you need from it, and leave what you don't need.

[cristianoc]

Another thought: it's pretty easy to load something dynamically, and also load it statically by mistake, by just writing other code that references the same module.
One could consider emitting a warning when that happens in the same file.
This would require listing what imports are emitted, and check that the module in the import is not one of those.

[mununki]

Do you mean this?

// in the same file
let lazyFoo = import(C.foo)

let lazyBar = import(C.bar)

If so, why would the warning need to be emitted?

[zth]

Probably more this:

// in the same file
let lazyFoo = import(C.foo)

let f = C.bar()

[mununki]

Probably more this:

// in the same file
let lazyFoo = import(C.foo)

let f = C.bar()

@zth what do you think? Do we need to emit an warning?

[zth]

Probably not for an initial release, but it highlights an interesting issue that's also magnified by the fact that using things in ReScript is easier thanks to no explicit imports. Also probably worth checking if such warnings exists for eslint/tslint etc.

[ah-yu]

Hi, I'm new here and don't know much about programming language design, I'm just expressing some of my thoughts here.

Dynamic import is quite natural in js because it is a nice addition to the static import without semantic changes.

import {foo} from "./foo"
const {foo} = await import("./foo")

So I'm wondering if can we introduce something like dynamically opening modules into the language? I think it's more natural than dynamic import.

open Foo
foo()

dopen Foo // compile to the dynamic import statement
foo()

[mununki]

@ah-yu Isn't it more natural to js expression you show?

let {foo} = import(module(C))

Can you elaborate the js expression corresponding to dopen Foo?

[zth]

I think we typically try to stay quite close to JS, so that the concepts map as well as possible to each other across the languages.

[ah-yu]

Forgot about dopen, it is absolutely not a good suggestion, just some random thoughts. Sorry for the bother.

Does close mean we should keep the same API with javascript? The module system of rescript is quite different from Js, so I thought there might have a more natural way to do a dynamic import.

another thought(just something came into mind, not thinking about it deeply):

let a = () => {
  module Foo = dynamic Foo
  Foo.foo()
}

[mununki]

Actually, there is no api dynamic in js. Your suggestion can be represented:

let a = async () => {
  let {foo} = await import(module(Foo))
  foo()
}

Does it make sense to you?

[ah-yu]

Thank you for your prompt reply. @mattdamon108

But the semantics are different. There is no import behavior in rescript. So the import API is really wired(just for me).

[mununki]

There wasn't async in ReScript either.
EDIT: nor dynamic neither.

[cristianoc]

Actually, there is no api dynamic in js. Your suggestion can be represented:

let a = async () => {
  let {foo} = await import(module(Foo))
  foo()
}

Does it make sense to you?

Filling in missing details: first-class modules are more painful to use than that. This is how it looks like:

module type FooT = module type of Foo

let a = async () => {
  let d = await import(module(Foo: FooT))
  module M = unpack(d)
  M.foo()
}

[cristianoc]

Might be better to have 2 different mechanisms after all, where the one for modules is this:

let a = async () => {
  module M = await import(Foo)
  M.foo()
}

Just to avoid pages of doc explaining what first-class modules are, and apologise about them.

[mununki]

First class module means that it can be taken as argument to function, right?

[ah-yu]

Unlike javascript, rescript can access modules without importing them. Why for dynamic modules we should import them first and then use them? It really doesn't make sense to me. Why not just mark them as dynamic

[mununki]

import * as foo from "foo.js" // 1

let m = import("./foo.js").then(m => m.default) // 2

@ah-yu IMHO, 1 and 2 are different in js. ReScript can access other modules without 1. The dynamic loading modules means 2.

[cristianoc]

Unlike javascript, rescript can access modules without importing them. Why for dynamic modules we should import them first and then use them? It really doesn't make sense to me. Why not just mark them as dynamic

Because you want to have control on when they are loaded. Which is not necessarily: the first time they are used.

[ah-yu]

I see. Let me think about it. Anyway, thanks for your time and patience! @mattdamon108

[ah-yu]

Unlike javascript, rescript can access modules without importing them. Why for dynamic modules we should import them first and then use them? It really doesn't make sense to me. Why not just mark them as dynamic

Because you want to have control on when they are loaded. Which is not necessarily: the first time they are used.

I see. Thanks for the explanation.

[cristianoc]

First class module means that it can be taken as argument to function, right?

It means modules packaged as values. In order to package a module as a value you need to provide the signature (in the example, the full module type of the module). Then they can be used just like other values, in particular sent as arguments to functions. But in order to do anything with them, they need to be unpacked back into modules.