Remove viper for cleanenv

[lzap]

Complete rewrite of our config handling. Configuration is now done via configs/api.env (pbapi), configs/worker.env (pbworker) and configs/test.env (integration tests). There is also configs/example.env file which is generated via a new makefile target generate-example-config.

Everything, literally everything is now defined in a single file internal/config/config.go - the whole structure, types, env variable name, default value and also optionally description.

ACTION NEEDED: New configuration needs to be deployed for development setups. There should be no changes needed for Clowder, except if we have a typo in an environmental variable.

Do not merge, this is the first cut. I will fix bugs and after review I would like to do few more TODO comments which is a small refactoring which leads from my review of all values. I want to do this in a single PR so people don't need to update configuration multiple times as I want to rename and move few around.

[lzap]

The "Rename clowder service to 'api'" commit was pushed accidentally, will remove.

[ezr-ondrej]

I wonder why We need different config file for api/worker, I guess most configuration in development is identical, right?

But I'm fine eighter way, it just means I need to define my DB three times, its inconvenient, but I can live with it 🤷

Apart of that 💘 it

[lzap]

Please do not merge, still working on refactoring those useless config and improving some others.

Well it seems logical to me, in clowder env you also have separate configs so I thought it's a reasonable to have this for development too. I can implement fallback, if worker.env does not exist load api.env.

[lzap]

I pushed a new version which removes EXIT_ON_PANIC (it was useless the app should never exit) and removes environment in favour or helper methods IsClowder and IsProductionClowder. The idea is, we only seem to need to check if the app is running in clowder environment or not. In one case tho (error when HTTP proxy is used) we do want to allow this in ephemeral environment because that's how IQE connects to sources. So the second helper is a WIP to check that, clowder should provide some kind of environment name I hope (there is a variable I want to test).

[lzap]

I will push new commits from now on, sorry, I haven't realized you started reviewing. The patch is smaller than I expected tho.

[lzap]

Just to let you know, those amended commits - I am just trying to break the CI so I can actually see the value in clowder.Metadata.Env variable, I guess it could be something like "ephemeral" but I need to confirm this :)

[lzap]

Ah we do NOT use HTTP proxy when in clowder, so it's even more simple, I do not need any IsProductionClowder function.

[lzap]

Ok removed, @ezr-ondrej how about this? I will only convert logger to string and heartbeat to time.Duration in separate commits.

[ezr-ondrej]

Nice, great refactoring, much easier to read then viper :)
The files make sense, I'd expect one main and then per-process, but api is kinda the main one, right?

Tho shouldn't the api go first in the calls? now if the var is set in worker and api, the api will win for worker, right? :)

[lzap]

Yeah I refactored and changed the order, initially I thought it is not possible to override values but looks like it does work.

[lzap]

Done, this is now ready for the final review and testing. Had to rebase one more time, there was a typo configs/api.env vs config/api.env (singular). I really think using singular is more appropriate, this was not my first time I screw this up.

If you want, I can rename the directory in another commit since people will have to update their configuration files anyway, it will be more disturbing change tho (makefile changes likely). Not a huge deal tho, two dozens of lines.

[ezr-ondrej]

Not a huge deal tho, two dozens of lines.

If it's not as big, I'd be for singular as well... I've messed up few times already as well

EDIT: Just keep in mind configs came from the "common go repo structure"

[lzap]

EDIT: Just keep in mind configs came from the "common go repo structure"

Yeah but sources-api-go use config, I feel like config is more appropriate.

[lzap]

Added a new commit with renamed configuration directory.

[ezr-ondrej]

Needs rebase, but I wonder how we should anounce these breaking changes to devel env? 🤔 shall we compose some list of recent changes in the repo? So devs who were away for a while can go to that list?
It could be simple list that links the PR for details ....

[lzap]

Will do, please merge this one first before these:

• Update to app-common-go 1.6.4 #259
• Improve HTTP client transport creation #254

There will be conflicts and this one is pain to rebase (multiple commits).

[ezr-ondrej]

The app starts, connects to DB and conects to AWS.
Thanks 👍 nice cleanup :)