Does viper require a config file in order to use ENV variables? #584
Comments
|
+1 just spent a lot of time finding why env variables don't bind to the config file. Simply adding the key to the yaml file made it work |
|
Looking at The example shows It would be nice to know if this is desired behavior, so I can build my applications accordingly. My main driver for avoiding a configuration file is that I do not wish to have configurations baked into my docker containers. I can deploy empty configurations, however, that is an additional (and unnecessary) stop in my build process. |
|
@jsirianni totally agree with you. I was surprised actually that I'm forced to configure |
|
The problem is actually Unmarshal works a bit differently: it collects the registered keys and passes them with their values to the mapstructure package which binds the values to the appropriate keys of the structure. In this case there is no "request" nothing to compare against. I agree that this is not optimal, but given Unmarshal was added later I can hardly categorize this as a regression. Unfortunately it's not easy to fix as environment variables are transformed in various ways:
Even if the prefix can be transformed back, the env key replacer cannot be "reversed".
(An alternative could be doing a "reverse decode": grab an empty struct and "decode" it into a In the meantime, using BindEnv manually is the only way unfortunately (or manual unmarshaling). Normally I define default configuration wherever possible, so it's really just those few keys I have to manually bind (database host, port, user, password, name; redis host and password). |
viper will not use env vars if no configuration file is found spf13/viper#584 As workaround we could manually bind/set a default for each configuration option using viper.SetDefault("key") and then generate a default config using viper.Get("key"). This manual solution is error prone and it will become increasingly difficult to maintain since the configuration options will grow, so we avoid it for now. Let's see if viper will solve this issue Fixes #35
|
Please see and follow the issue above. ☝️ |
|
I'm modeling my whole config as struct. To make all environment variables work, even when the config file does not contain the key I first load my defaults from a struct like this: |
This adds the ability to override config settings with an environment variable in the form "FEATMAP_<uppercase setting name>". e.g. `FEATMAP_PORT=8080` would override the `port` value in `conf.json`. Due to the way Viper works, this cannot load from *only* env vars, so the config file is still needed and the env vars can override it. See spf13/viper#584 To help protect users from this issue, this change also adds verification that essential config settings have been configured. # Tests - Test that configuration is still loaded from the config file - Test that environment variables override the values in the config file - Test that missing settings correctly report an error
This adds the ability to override config settings with an environment variable in the form "FEATMAP_<uppercase setting name>". e.g. `FEATMAP_PORT=8080` would override the `port` value in `conf.json`. Due to the way Viper works, this cannot load from *only* env vars, so the config file is still needed and the env vars can override it. See spf13/viper#584 To help protect users from this issue, this change also adds verification that essential config settings have been configured. # Tests - Test that configuration is still loaded from the config file - Test that environment variables override the values in the config file - Test that missing settings correctly report an error
|
Facing the same issue... and it seems to have a not easy solution :( |
you can simply use |
it works... thank you so much! |
|
I hope it's not a long answer to this problem (not tested with maps) Below code will:
which leads to all of these fields to get picked up from the environment package main
import (
"fmt"
"github.com/spf13/viper"
"reflect"
"strings"
)
type (
Settings struct {
Vitals Vitals
Excludes []string `yaml:"excludes,omitempty"`
}
Vitals struct {
Product PIdentifier
Project PIdentifier
}
PIdentifier struct {
Name string
Token string
}
)
func addKeysToViper(v *viper.Viper) {
var reply interface{} = Settings{}
t := reflect.TypeOf(reply)
keys := getAllKeys(t)
for _,key := range keys {
fmt.Println(key)
v.SetDefault(key, "")
}
}
func getAllKeys(t reflect.Type) []string {
var result []string
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
n := strings.ToUpper(f.Name)
if reflect.Struct == f.Type.Kind() {
subKeys := getAllKeys(f.Type)
for _, k := range subKeys {
result = append(result, n+"."+k)
}
} else {
result = append(result, n)
}
}
return result
}
func main() {
v := viper.New()
addKeysToViper(v)
v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
v.SetEnvPrefix("ABC")
v.AutomaticEnv()
fmt.Println(v.WriteConfigAs("out.yaml"))
}the printed output from the env vars used in this test: the resulting value for excludes: c
vitals:
product:
name: a
token: b
project:
name: ""
token: "" |
|
my workaround, seem silly but it worked func Parse(i interface{}) error {
r := reflect.TypeOf(i)
for r.Kind() == reflect.Ptr {
r = r.Elem()
}
for i := 0; i < r.NumField(); i++ {
env := r.Field(i).Tag.Get("mapstructure")
if err := viper.BindEnv(env); err != nil {
return err
}
}
return viper.Unmarshal(i)
} |
|
The above example from @qwerty22121998 is a solid work around for flat structures; however, it struggles in that it doesn't deal with nested structures (Go Playground Link). package main
import (
"fmt"
"os"
"reflect"
"strings"
"github.com/spf13/viper"
)
type Config struct {
Foo string `mapstructure:"FOO"`
Bar string `mapstructure:"BAR"`
Baz FooBar `mapstructure:"BAZ"`
}
type FooBar struct {
Foo string `mapstructure:"FOO"`
Bar string `mapstructure:"BAR"`
}
func scaffold() {
os.Setenv("FOO", "1")
os.Setenv("BAR", "A")
os.Setenv("BAZ.FOO", "2")
os.Setenv("BAZ.BAR", "B")
}
func parseEnv(i interface{}) error {
r := reflect.TypeOf(i)
for r.Kind() == reflect.Ptr {
r = r.Elem()
}
for i := 0; i < r.NumField(); i++ {
env := r.Field(i).Tag.Get("mapstructure")
if err := viper.BindEnv(env); err != nil {
return err
}
}
return viper.Unmarshal(i)
}
func main() {
scaffold()
var c Config
if e := parseEnv(&c); e != nil {
panic(e)
}
fmt.Println(fmt.Sprintf("%+v", c))
}The above will output: {Foo:1 Bar:A Baz:{Foo: Bar:}}
Program exited.I would suggest making a few changes. In particular, we want to check if the current property is a struct and recursively try and build the environment variables. If you want to use this strategy, I would propose changes like the following (Go Playground Link): package main
import (
"fmt"
"os"
"reflect"
"github.com/spf13/viper"
)
type Config struct {
Foo string `mapstructure:"FOO"`
Bar string `mapstructure:"BAR"`
Baz FooBar `mapstructure:"BAZ"`
}
type FooBar struct {
Foo string `mapstructure:"FOO"`
Bar string `mapstructure:"BAR"`
Baz FooBarBaz `mapstructure:"BAZ"`
}
type FooBarBaz struct {
Foo string `mapstructure:"FOO"`
Bar string `mapstructure:"BAR"`
}
func scaffold() {
os.Setenv("FOO", "1")
os.Setenv("BAR", "A")
os.Setenv("BAZ.FOO", "2")
os.Setenv("BAZ.BAR", "B")
os.Setenv("BAZ.BAZ.FOO", "3")
os.Setenv("BAZ.BAZ.BAR", "C")
}
func parseEnv(i interface{}, parent, delim string) error {
// Retrieve the underlying type of variable `i`.
r := reflect.TypeOf(i)
// If `i` is of type pointer, retrieve the referenced type.
if r.Kind() == reflect.Ptr {
r = r.Elem()
}
// Iterate over each field for the type. By default, there is a single field.
for i := 0; i < r.NumField(); i++ {
// Retrieve the current field and get the `mapstructure` tag.
f := r.Field(i)
env := f.Tag.Get("mapstructure")
// By default, set the key to the current tag value. If a parent value was passed in
// prepend the parent and the delimiter.
if parent != "" {
env = parent + delim + env
}
// If it's a struct, only bind properties.
if f.Type.Kind() == reflect.Struct {
t := reflect.New(f.Type).Elem().Interface()
parseEnv(t, env, delim)
continue
}
// Bind the environment variable.
if e := viper.BindEnv(env); e != nil {
return e
}
}
return viper.Unmarshal(i)
}
func main() {
scaffold()
var c Config
if e := parseEnv(&c, "", "."); e != nil {
panic(e)
}
fmt.Println(fmt.Sprintf("%+v", c))
}The above will output: {Foo:1 Bar:A Baz:{Foo:2 Bar:B Baz:{Foo:3 Bar:C}}}
Program exited.For anyone else who ends up searching up this issue and coming here, this is because Viper, at its core, is primarily meant to handle file-based environment configurations. In fact, the If you want to use environment variables by themselves you need to either:
I would probably recommend sticking to one of the above solutions which binds variables into one of the internal maps used by Viper instead of using some recursive method in which you uses the |
|
I frequently come back to this issue because we still have codes that use viper. For those who are tired of these issues (which essentially come from bad design), you should have a look at koanf. |
Thanks, this helped me alot |
Hi, I have an application that can use both a config file and env variable.
My
initConfig()method looks like this:unmarshaldrops the config into a struct, and it works well.I am facing an issue where if my config file does not exist (or is empty), viper seems to completely ignore any environment variables. I have verified that viper can see the env variables, by printing them to standard out. My workaround is to deploy the config file, with all the correct keys but no values. Viper then correctly overrides the config file with my environment variables.
Am I wrong to assume that Viper should be able to use environment variables when no config file is found?
The text was updated successfully, but these errors were encountered: