Allow binding to environment variables #1859

apedroferreira · 2023-04-05T20:49:02Z

Allow binding HTTP request headers to environment variables (or any other property besides request headers, if we enable it to).
Includes tests and documentation.

Screen.Recording.2023-05-17.at.19.07.14.mov

…http-queries

bharatkashyap

Looks good!

packages/toolpad-app/src/toolpadDataSources/rest/server.ts

…n-http-queries

Janpot · 2023-05-09T07:31:39Z

packages/toolpad-app/src/toolpadDataSources/rest/client.tsx

@@ -363,10 +363,10 @@ function QueryEditor({
 () => ({
 parameters: previewParams,
 process: {
- env,
+ env: Object.fromEntries(envVarNames.map((varName) => [varName, '<SECRET>'])),


Oh, we're evaluating some bindings on the client side...
Then I guess it isn't possible to only send the variable names?
😕 This is not great

So it's not secure this way? I thought it only uses the real environment values if it runs server-side :/ Shouldn't that be possible?

Ah, yes client-side it won't use the real values in this case, yeah...

I can show <SECRET> and use the real values in the client anyway if that's still safe.

I think we can do either one of those:

Send the whole env like you did before and let it be

Rethink how to use secrets in rest datasource and instead offer a new type of binding, next to javascript expression, allow binding directly to an environment variable through a dropdown

I feel like we can go with the first option "Send the whole env like you did before and let it be".
But let me know if you think the second one is better for any reason and it would not be that much extra work anyway.

Some arguments I can think of that would be in favor for the second solution:

It's easier for us to detect which environment variables an app expects to be run under. That allows us to warn on any missing ones, e.g. when a user checks out an app for the first time. It could somewhat be extracted from the javascript bindings by regex, or static analysis, but it wouldn't be a foolproof solution.

We could more fine grained specify where exactly we want this to be used. e.g. only in authentication/headers

Thinking long term, suppose we decide that whatever we came up with in this PR was a mistake, we can migrate from $env: MY_VAR to $jsExpression: 'process.env.MY_var' and guarantee no breaking change, we can't do that on the inverse.

Fair points, ok I'll change it to be a new type of binding, having it as $env sounds consistent with the other types.

This reverts commit e490e94.

…n-http-queries

Janpot

Looking good. I'd change a few things:

avoid the layout shift when switching tabs in the binding editor

Screen.Recording.2023-05-14.at.12.11.39.mov
Make it a free input with autocomplete. We can give an indication when we don't find the environment variable in the env file, but I feel like we should allow input. As a follow up we can prompt the user for a value if we don't find it in the env file and add it automatically
Update the rest-basic test with some checks for environment variables.

Additionally we should update the HTTP query docs. This can be done separately as well if you want

Janpot · 2023-05-13T16:07:49Z

packages/toolpad-app/src/server/schema.ts

@@ -301,6 +308,7 @@ export const META = {
 definitions: {
 Json: jsonSchema,
 JsExpressionBinding: jsExpressionBindingSchema,
+ EnvBinding: envBindingSchema,


Just a note, but after beta we will have to think about a migration strategy, because this is not backwards compatible

But it's not a breaking change, just a new type of binding, right? Why would it break with previous versions?

An app created with this schema won't work in older Toolpad versions. Older Toolpad versions don't know what $env means

Oh, I thought we only needed to make to make old schemas be able to migrate to newer versions...
If we have backwards compatibility like this I guess we will need to migrate, yep.

The general idea would be:

old schema + new toolpad => we should be able to internally migrate and display the app + give option for user to migrate files to new version, or do it automatically

new schema + old toolpad => either all features of the schema are supported in Toolpad, or otherwise we need to tell the user "This app as created in newer Toolpad, please upgrade Toolpad to version X if you want to run/edit this application"

At first thought the 2nd option (showing the user a message) seems more sane to me and not that unusual?
But we can try the first one if it's not that hard, also depending on how common it would be for users to try to use new schemas in old Toolpad versions.

also depending on how common it would be for users to try to use new schemas in old Toolpad versions.

This scenario could happen when people copy toolpad apps from one repo to another

packages/toolpad-app/src/toolpadDataSources/rest/client.tsx

packages/toolpad-app/src/toolpad/AppEditor/BindingEditor.tsx

packages/toolpad-app/src/toolpadDataSources/rest/server.ts

…n-http-queries

Janpot

this is great!

prakhargupta1 · 2023-05-22T09:05:29Z

This should be very helpful! I'll resume work on some of the public apps that needed this feature.
cc: @oliviertassinari

oliviertassinari · 2023-05-22T15:56:29Z

Ah, yes, awesome, we can move more apps https://github.com/mui/mui-private/issues/267. It's different than #1952 but would still solve the problem.

Ability to access process in HTTP query bindings

18c5ab2

apedroferreira added the new feature New feature or request label Apr 5, 2023

apedroferreira self-assigned this Apr 5, 2023

apedroferreira added 4 commits April 5, 2023 21:49

Merge remote-tracking branch 'origin/master' into use-process-env-in-…

89ceabf

…http-queries

Out of place comment

c9d0a6a

Merge remote-tracking branch 'origin/master' into use-process-env-in-…

f9a7dfa

…http-queries

Remove unused import

025031f

apedroferreira marked this pull request as ready for review April 6, 2023 12:36

apedroferreira added 2 commits April 10, 2023 18:55

Merge branch 'master' into use-process-env-in-http-queries

1d6230e

Fix build

b016d0d

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 12, 2023

Merge remote-tracking branch 'origin/master' into use-process-env-in-…

36e6706

…http-queries

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 14, 2023

apedroferreira requested review from a team and Janpot April 14, 2023 14:03

This comment was marked as outdated.

Sign in to view

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 20, 2023

Merge remote-tracking branch 'origin/master' into use-process-env-in-…

5f504cd

…http-queries

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 27, 2023

apedroferreira added 3 commits April 27, 2023 23:36

Use introspection query

e41f73c

Fixes

888689f

Fix types

9ea7edb

bharatkashyap approved these changes Apr 28, 2023

View reviewed changes

Janpot reviewed May 3, 2023

View reviewed changes

packages/toolpad-app/src/toolpadDataSources/rest/server.ts Outdated Show resolved Hide resolved

apedroferreira added 2 commits May 9, 2023 08:17

Retrieve env var names only in introspection

e490e94

Merge remote-tracking branch 'upstream/master' into use-process-env-i…

27b578e

…n-http-queries

Janpot reviewed May 9, 2023

View reviewed changes

apedroferreira added 3 commits May 9, 2023 11:09

Revert "Retrieve env var names only in introspection"

547e1d6

This reverts commit e490e94.

Merge remote-tracking branch 'upstream/master' into use-process-env-i…

2b89c28

…n-http-queries

Use env bindings (still need to fix types)

d666aaa

apedroferreira added 2 commits May 12, 2023 19:11

Fix types

175b749

Merge remote-tracking branch 'upstream/master' into use-process-env-i…

c6fabf8

…n-http-queries

apedroferreira requested a review from Janpot May 12, 2023 18:13

Remove env binding option from URL input

80a3a92

Janpot requested changes May 14, 2023

View reviewed changes

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label May 15, 2023

Merge remote-tracking branch 'upstream/master' into use-process-env-i…

a51702f

…n-http-queries

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label May 16, 2023

apedroferreira added 2 commits May 16, 2023 22:47

Use env var names, autocomplete input

d6ef1f4

Adjustments to last commit

661d393

This comment was marked as resolved.

Sign in to view

apedroferreira added 4 commits May 16, 2023 23:33

Better autocomplete + prevent layout shift

7a8aed5

Better responsive behavior

8a784ab

Test + fixes and improvements

afc70e4

Update docs

ee4de96

apedroferreira changed the title ~~Ability to access process in HTTP query bindings~~ Allow binding to environment variables May 17, 2023

Maybe fix tests

2ec74a6

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label May 17, 2023

Merge remote-tracking branch 'upstream/master' into use-process-env-i…

e26bc8a

…n-http-queries

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label May 17, 2023

apedroferreira requested a review from Janpot May 17, 2023 21:33

Janpot approved these changes May 17, 2023

View reviewed changes

apedroferreira merged commit f257e90 into master May 18, 2023

apedroferreira deleted the use-process-env-in-http-queries branch May 18, 2023 15:05

Janpot mentioned this pull request Jun 1, 2023

Issues with environment variables #2099

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow binding to environment variables #1859

Allow binding to environment variables #1859

apedroferreira commented Apr 5, 2023 •

edited

Loading

This comment was marked as outdated.

bharatkashyap left a comment

Janpot May 9, 2023 •

edited

Loading

apedroferreira May 9, 2023

apedroferreira May 9, 2023

apedroferreira May 9, 2023

Janpot May 9, 2023

apedroferreira May 9, 2023

Janpot May 10, 2023

apedroferreira May 10, 2023

Janpot left a comment •

edited

Loading

Janpot May 13, 2023 •

edited

Loading

apedroferreira May 15, 2023

Janpot May 15, 2023

apedroferreira May 15, 2023 •

edited

Loading

Janpot May 16, 2023

apedroferreira May 16, 2023 •

edited

Loading

Janpot May 16, 2023 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

Janpot left a comment

prakhargupta1 commented May 22, 2023

oliviertassinari commented May 22, 2023 •

edited

Loading

Allow binding to environment variables #1859

Allow binding to environment variables #1859

Conversation

apedroferreira commented Apr 5, 2023 • edited Loading

This comment was marked as outdated.

bharatkashyap left a comment

Choose a reason for hiding this comment

Janpot May 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Janpot left a comment • edited Loading

Choose a reason for hiding this comment

Janpot May 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apedroferreira May 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apedroferreira May 16, 2023 • edited Loading

Choose a reason for hiding this comment

Janpot May 16, 2023 • edited Loading

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

Janpot left a comment

Choose a reason for hiding this comment

prakhargupta1 commented May 22, 2023

oliviertassinari commented May 22, 2023 • edited Loading

apedroferreira commented Apr 5, 2023 •

edited

Loading

Janpot May 9, 2023 •

edited

Loading

Janpot left a comment •

edited

Loading

Janpot May 13, 2023 •

edited

Loading

apedroferreira May 15, 2023 •

edited

Loading

apedroferreira May 16, 2023 •

edited

Loading

Janpot May 16, 2023 •

edited

Loading

oliviertassinari commented May 22, 2023 •

edited

Loading