[studio] Export as ESM import

[prakhargupta1]

Problem

#316 is great but as soon as you export to code, you lose all the benefits of using Toolpad, you can no longer sync the output with the tool. Using an ESM import would allow developers to have a hybrid approach with their app.

Possible use cases:

1. Customer survey for external end-users. For this, developers would need to be able to use their own design system:

Benchmark

• Framer Handshake ESM import: https://www.framer.com/docs/guides/handshake/

import Card from "https://framer.com/m/Card-abcd.js@abcdef1234"

<Card foo="bar" />

• Plasmic HTML injection: https://docs.plasmic.app/learn/nextjs-quickstart/

<PlasmicRootProvider loader={PLASMIC} prefetchedData={plasmicData}>
  <PlasmicComponent component="PATH_OR_COMPONENT" />
</PlasmicRootProvider>

[Janpot]

Some things to think about:

• What exactly will be exported? a page? an app? what's the exact use-case?
• How does data fetching work?
• We're doing internal apps, what does this mean for authentication? Can you import a component of a private app?
• ...

[prakhargupta1]

From internal apps perspective, the developer might want to embed Toolpad app in their existing internal tool. I such a scenario, can they pass on auth params to grant access to Toolpad app? This is clearly more work for the developer, but they don't want other teams to log into a different platform.
So a single page should be exported.

https://www.jotform.com/help/how-to-embed-apps-to-a-website/
https://docs.appsmith.com/advanced-concepts/embed-appsmith-into-existing-application

From external apps perspective. I think making a publicly accessible through a URL >> making it embeddable.

[buremba]

Great emphasis on the the hybrid approach and the integration with internal tools. I'm also doing proof of concept with Toolpad and it looks great so far.

For the hybrid approach, I wonder if we can find a way to let the user switch between UI and code easily. I see that you call the YML & JSON as "output" and React & ESM as "export". Rather than treating the components created in UI as packages, we might also consider exporting to MDX from the internal JSON format and parsing MDX to the same JSON format for better readability and compatibility. MDX already supports compiling to JSX modules that can be imported in ESM. Since MDX can import modules and define export or Provider components, we can also compile $jsExpression easily. WDYT?

[Janpot]

I see that you call the YML & JSON as "output" and React & ESM as "export".

I don't see the yaml/json files just as "output", they also serve as "input". They are the underlying data model that describes a toolpad application. It's hand editable but that's not the intention for day-to-day use. You can hand edit it as a power user to fulfil tasks such as bulk editing large applications, generate an application from higher level requirements,... The main requirement for it is to be declarative. We've experimented a bit and it doesn't seem likely that it's very feasable to use a non-declarative language as the data model for a Toolpad application (Also see utopia.app or blocksUI)

We see "export" or "eject" as a one-way action only, producing runnable code where the focus would be on one of the following use cases:

• Exported code could be more performant than the Toolpad runtime, you'd use it to create runnable artifacts to run applications in production. Exporting in CI every time you deploy and e.g. packaging it up as a docker container, or directly importing in an existing React application
• You outgrow the capabilities of Toolpad and wish to continue the pro-code way. You don't want to start from scratch and eject the application as a starting point to continue the pro-code way. This is an advanced case of the previous point where an extra constraint is that the output must be human readable and clean code.

[buremba]

Thanks for the answer Jan! I agree that most tools separate the "exporting" feature in a way that you can't go back to the visual editor but I'm trying to highlight that it doesn't necessarily need to be that way.

Both utopia.app and blocksUI generate React components with JSX, which is very flexible in a way a visual editor can't easily be used on top. This is because you might have variables, inline functions, outer loops, etc.

Let's go over the following page:

apiVersion: v1
kind: page
spec:
  id: od4ulyy
  title: dashboard
  display: shell
  content:
    - component: Text
      name: markdown
      props:
        mode: text
        value:
          $$jsExpression: |
            mydata
        variant: h1
        loading: false
    - component: PageRow
      name: pageRow
      props:
        justifyContent: start
      children:
        - component: Chart
          name: chart
          layout:
            columnSize: 1.3388946819603753
          props:
            data:
              - label: dataSeries1
                kind: line
                data:
                  $$jsExpression: |
                    [{ count: 10 }]
            sx:
              bgcolor: background.paper
        - component: Chart
          name: chart1
          props:
            data:
              - label: dataSeries1
                kind: line
                data: []
          layout:
            columnSize: 0.6611053180396247
 queries:
    - name: mydata
      query:
        function: dasdas.ts#default
        kind: local
      parameters:
        - name: message
          value: test

It can be converted to MDX as follows:

import dasdas from 'dasdas'

export const mydata = dasdas('test');

# {mydata}
<PageRow justifyContent="start"/>
  <Layout columnSize=1.3388946819603753>
      <Chart id="chart1" data={ [{ count: 10 }] kind={'line'} sx={{bgcolor: 'background.paper'}}/>
  </Layout>
  <Layout columnSize=0.6611053180396247>
      <Chart id="chart2" kind={'line'}  data=[] data={query.data}/>
  </Layout>
</PageRow>

The MDX above is completely valid and I believe covers what's represented in the YML. We can restrict the export syntax anywhere but on top of the file similar to imports to simplify parsing and avoid edge cases. If there is problems with export syntax, the data can even be defined in frontmatter as well.

The cool part of that since the above is a valid MDX, it's portable similar to React-generated code that can be used outside of Toolpad. Toolpad's custom data providers/components are flexible enough that I might not need to take my "exported" code and continue the development outside of Toolpad so MDX is kinda the output layer here, which is much more readable compared to YML and JSON, which are much harder to read. I can easily read this MDX and understand the layout unline the YML counterpart. I might be missing important details though, WDYT?

[Janpot]

Since JSX inside is written MDX you will encounter the exact same problems as in utopia.app unless you heavily restrict what counts as "valid Toolpad MDX" which instantly makes it not portable. Generating MDX out of a Toolpad application definition is the easy part. The hard part is updating an MDX file from the drag&drop editor after it has been hand edited.

We have discussed the topic of "sync visual editor and code" extensively internally and made some prototypes. We don't really see many possibilities for significant improvement over existing attempts, even with MDX. I highly encourage you to experiment further in this problem space though.

[buremba]

I see the point you make but I'm not sure I agree with "restricting features in MDX makes it not portable" because even when you look at frameworks such as Gatsby, Next.js, and Remix, not all the MDX features are supported by them or provide a mechanism to enable/disable features (etc. functions, export syntax etc.). While this kind of restriction might hurt DX, they can always write React and import it as a component from MDX IMO as a workaround bu it's still a win considering I never manage to edit the generated YML file due to the complexity (maybe YML validator in VSCode improves it, will try that as well)

I will try to prototype something using MDX parser as it sounds to be a cool project as well when I have time and probably start with defining data in the frontmatter rather than variable approach due to the complexity. Thanks a lot!

This is what I'm thinking as a start:

---
 queries:
    - name: mydata
      query:
        function: dasdas.ts#default
        kind: local
      parameters:
        - name: message
          value: test
---

# {mydata}
<PageRow justifyContent="start"/>
  <Layout columnSize=1.3388946819603753>
      <Chart id="chart1" data={ [{ count: 10 }] kind={'line'} sx={{bgcolor: 'background.paper'}}/>
  </Layout>
  <Layout columnSize=0.6611053180396247>
      <Chart id="chart2" kind={'line'}  data=[] data={query.data}/>
  </Layout>
</PageRow>

and restrict export, any function callback, etc. to simplify parsing MDX, which is the hard part as you stated.

[buremba]

Another question; is it possible to just import Toolpad's runtime renderer as a library and render pages without the full editor? I remember us talking about it but if you can guide me on where to look at, etc. I would appreciate!

I think the custom server implementation helps mounting the data, etc. but I'm hoping to just import a component and pass the YML/JSON to render the UI without any server requirement. The current implementation of the custom servers seems to require express framework, which I don't use and I don't need the runtime-rpc anyways as REST data provider works fine for me.

I plan to use Toolpad Editor as it is in my DEV environment and just focus on Toolpad YML -> MDX and then MDX -> Toolpad YML conversion and only use Toolpad's runtime renderer in my front-end app if possible. It would help me reduce the scope to keep it simpler.

[Janpot]

Another question; is it possible to just import Toolpad's runtime renderer as a library and render pages without the full editor? I remember us talking about it but if you can guide me on where to look at, etc. I would appreciate!

It's not public API and not under any sort of semver guarantees, but You could take a look at https://github.com/mui/mui-toolpad/blob/aa4abf4d87bdc4ef8a0792fc05bdb774459028e3/packages/toolpad-app/src/runtime/ToolpadApp.tsx#L1528
It does require some things from the build environment if you want to make use of custom components.