[DX] Read .env files in the functions directory

[kentcdodds]

I configure env variables in the netlify UI, but locally I use .gitignored .env files and https://www.npmjs.com/package/dotenv-cli to run netlify-lambda with the right env variables.

It'd be cool if netlify-dev-plugin could solve this problem somehow :)

[philhawksworth]

Perhaps netlify dev could access and locally use environment variables defined in the Netlify UI, but allow any environment variables defined in a local .env file to override those found remotely.

And if this could happen automatically by using dotenv or similar internally without requiring the developer to manually include that, it would feel like a nice simplification to me.

[swyxio]

sooo.. i just found out that netlify dev ALREADY does this 🤯 https://github.com/netlify/netlify-dev-plugin/blob/master/src/commands/dev/index.js#L196

this is crazy... NO MORE LOCAL .env VARIABLES!!!!!!!!!!!! JUST SET IT IN THE UI AND YOU'RE DONE

and to be clear, any set process.env variables will override - but we dont read the .env file for you, you'd have to do it on your own.

if this is satisfactory to @kentcdodds, we can close this, but if anyone feels strongly that we should auto-read a .env file in the root directory, i'm open to the idea but want buyin from everyone. also have to consider conventions like https://github.com/kentcdodds/kentcdodds.com which has .env.production and .env.development

[philhawksworth]

This is a pretty huge first step!

Handling the ability to do local overrides automatically would be a very useful utility and a benefit to using netlify dev but reasonable to have that as next iteration in my view.

[owenconti]

I'm finding that I cannot override any process.env variables when using dotenv in combination with netlify dev.

I need to check process.env.NODE_ENV to determine which callback URL to provide when authenticating users via Twitter:

const isProduction = process.env.NODE_ENV === 'production';

console.log('isProduction', isProduction);

if (!isProduction) {
  console.log('Loading dotenv config...');
  require('dotenv').config();
}

// later in the file...

new TwitterStrategy({
  callbackURL: process.env.SERVER_URL + '/auth/twitter/callback'
},

And I can confirm that dotenv is called (via the log):

Request from ::ffff:127.0.0.1: GET /.netlify/functions/auth/twitter
isProduction false
Loading dotenv config...

When using netlify-lambda serve this works fine, but with netlify dev (after running netlify link) all the environment variables are the "production" ones, not my local .env file ones.

[iamskok]

@owenconti were you able to figure it out?

[maoberlehner]

sooo.. i just found out that netlify dev ALREADY does this 🤯 https://github.com/netlify/netlify-dev-plugin/blob/master/src/commands/dev/index.js#L196

this is crazy... NO MORE LOCAL .env VARIABLES!!!!!!!!!!!! JUST SET IT IN THE UI AND YOU'RE DONE

Although this is indeed an interesting feature, I usually don't want (most of) the production or staging env variables in my local dev system. This is kinda the point of env variables: they're different between environments 😅

Automatically reading .env would be great!

[robnewton]

We are going to try to get around this for now by making a copy of the site in app.netlify.com which will basically act as a repo for local dev environment variables. We will link to that from the ntl cli so it will pull down those env vars.

I would much rather just be able to drop a gitignored env in the root of the site that overrides one or two environment variables selectively then uses the injected site values for the rest.

[uptownhr]

Not sure if this is the right thread for this but I also ran into env variable issues when trying out netlify dev. I would actually wish that .toml file environmental variables be used in netlify dev and made available not just to the site but also to the functions. This would provide a single place to handle environmental variables for both dev and production

[BrunoQuaresma]

Is there any workaround to get .env files working on netlify dev?

[bsgreenb]

Shouldn't Netlify Dev environment just be its own deploy context?

Also am personally finding that Netlify kind of clashes with my dotenv setup, so would definitely appreciate a dotenv friendly approach.

[dferber90]

I'm finding that I cannot override any process.env variables when using dotenv in combination with netlify dev.

And I can confirm that dotenv is called (via the log):

Request from ::ffff:127.0.0.1: GET /.netlify/functions/auth/twitter
isProduction false
Loading dotenv config...

@owenconti This happens because netlify dev will set the environment variables before dotenv gets a chance to run. dotenv doesn't overwrite existing environment variables (see https://www.npmjs.com/package/dotenv#what-happens-to-environment-variables-that-were-already-set), so the value defined in Netlify wins.

---

@BrunoQuaresma Here's a snippet which enables overwriting env vars defined in Netlify with env vars from .env. This only works for Netlify functions though, not for your client-side code. Note that this snippet would have to be pasted at the top of every function you want to use.

const dotenv = require("dotenv")
const envConfig = dotenv.config();
// "parsed" will be undefined when no .env file was present,
// in which case we need to skip the call to Object.entries
if (envConfig.parsed) {
  Object.entries(envConfig.parsed).forEach(([key, value]) => {
    process.env[key] = value;
  });
}

Or as a short version

const envConfig = require("dotenv").config();
Object.entries(envConfig.parsed || {}).forEach(
  ([key, value]) => (process.env[key] = value)
);

---

@sw-yx

but if anyone feels strongly that we should auto-read a .env file in the root directory, i'm open to the idea but want buyin from everyone

I too would appreciate out-of-the-box support for .env in netlify dev. I want to use most env vars defined in Netlify, while being able to overwrite specific ones using a .env file. My specific use-case is that I have a token for a payment provider, where I'd use a the sandbox token in development and the real one in production.

also have to consider conventions like https://github.com/kentcdodds/kentcdodds.com which has .env.production and .env.development

The concept of .env.production and .env.development doesn't really make sense to me in this case. To me .env.production are the env vars stored in Netlify, while .env.development is what I'd store in .env.

Some ideas/ramblings

What might be nice to have (from my limited understanding) is a way to provide env var overwrites for different deploy contexts through the Netlify UI. That would enable using different API keys for production and preview branches easily. Additionally netlify dev could have its own section there, where we could define overwrites of env vars for netlify dev as well. Then devs wouldn't need to pass .env files to new team members. I've always been wondering how other Netlify users are solving this. I'm aware of overwrites in netlify.toml, but I wouldn't want to commit those and therefore netlify.toml isn't an option for me here.

[ksaturnius]

From a security perspective, the reading of .env files would be a really nice addition. In particular, for our use case, we're using Storyblok as a headless cms. In order to interact with their content management api, there is a per-user api key required. It would be ideal for each dev to use their own key during development and have our infra/ops account owner on Storyblok generate the key that is used in production. I'm thinking there would be two parts to this:

1. Provide an optional "lock" mechanism in the Netlify admin that prevents the Netlify-cli from automatically injecting it's environment variables into local dev environments. This would entirely prevent exposure of any environment variables added via the Netlify admin outside of the Netlify pipeline.
2. Allow loading of .env files locally, which would automatically overwrite the values injected by the Netlify cli (in case the environment variables haven't already been "locked" as per above).

Also, to touch on @dferber90's "ramblings" section, my only concern about having netlify dev having it's own context is that it wouldn't really provide the same level of security the way a local .env file would. As in my example above with each developer needing their own oAuth key to interact with the api, as soon as a team member leaves, we would theoretically need to cycle that oAuth key. But if it's in a local .env file, as soon as we remove them as a member from our Storyblok site, the oAuth key would fail to work even if their account is still functional.

[DavidWells]

Env support is coming soon with #616 and new https://github.com/netlify/build pipeline

You will be able to reference environment variables in config files

key: ${env:MY_ENV_VAR}

[trevorblades]

An alternative technique that I've been using recently is setting my _DEV and _PROD environment variables in the UI, and then switching on process.env.NETLIFY_DEV in my lambda function, which is true when using netlify dev.

setClientSecret(
  process.env.NETLIFY_DEV
    ? process.env.CLIENT_SECRET_DEV
    : process.env.CLIENT_SECRET_PROD
);

I don't know how well-documented this NETLIFY_DEV env variable is, but it's a life saver!