Allow Proxy Configuration in config.json #93

dave-tucker · 2017-05-16T14:42:14Z

This is moby/moby#32966 re-openend against docker/cli
The above PR was a redux of moby/moby#30588 rebased and after the repo move!

- What I did

Added a mechanism to allow HTTP/HTTPS/FTP/NO proxy variables to be configured in config.json. These are then automatically populated in a docker run command as environment variables or as build-args in docker build!

This makes it easier for users suffering behind an HTTP Proxy as they only need to configure this once as opposed to creating custom aliases or forgetting the args and being denied buildage/runnage.

Updates #30323

- How I did it

Extended the configfile with some new proxy related variables.

These are scoped per Docker Host with a default catchall.

The config file is then read by the build or run command and the necessary variables are exported.

A -e flag or --build-arg provided on the CLI will override these default settings.

- How to verify it

Create a config.json with { "proxies": {"default": { "httpProxy": "http://127.0.0.1:3001" }}}
Create a Dockerfile

FROM busybox
RUN echo $HTTP_PROXY
CMD echo $HTTP_PROXY

docker --config /path/to/config build -t configtest .
Verify that the HTTP_PROXY variable is as configured
docker run --rm configtest env
Verify that the HTTP_PROXY variable is as configured

- Description for the changelog

Added HTTP/HTTPS/FTP proxy configuration to config.json

- A picture of a cute animal (not mandatory but encouraged)

dnephin · 2017-05-16T14:52:39Z

cli/command/container/run.go

 	"github.com/docker/docker/api/types"
 	"github.com/docker/docker/api/types/container"
+	"github.com/docker/docker/opts"


#82, this should use github.com/docker/cli/opts

dnephin · 2017-05-16T14:55:12Z

cli/command/container/run.go

@@ -295,3 +299,44 @@ func runStartContainerErr(err error) error {

 	return statusError
 }
+
+func parseProxyConfig(cfg *configfile.ConfigFile, host string, o opts.ListOpts) opts.ListOpts {


A more descriptive variable name for o would be great.

Since we're only using this as a map[string]string, either passing in the []string or map[string]string would be good.

dnephin · 2017-05-16T14:58:08Z

cli/command/container/run.go

+	}
+
+	config, _ := cfg.Proxies[cfgKey]
+	permitted := map[string]*string{


Does this need to be a *string? It seems it's only storing string.

It only stores a string... but as we're assigning to *string further down in the code, it's easier to make it *string here as you can't take the address of map values (e.g &permitted["HTTP_PROXY"])

dnephin · 2017-05-16T15:11:14Z

cli/command/image/build.go

@@ -469,3 +471,34 @@ func replaceDockerfileTarWrapper(ctx context.Context, inputTarStream io.ReadClos

 	return pipeReader
 }
+
+func createBuildArgs(cfg *configfile.ConfigFile, host string, o opts.ListOpts) map[string]*string {


There's a lot of duplication between this and parseProxyConfig(). I think that should be easy to fix.

If you move createBuildArgs() to cli/config/configfile and make it ParseProxyConfig(), you could call it directly from this package, and in container/run.go you could call it and perform the last step of converting to a ListOpts again.

A test case for the new function would be great as well.

dnephin

Design LGTM

The client user config seems like a reasonable place to provide proxy information considering the current method is from client environment variables.

dave-tucker · 2017-05-16T15:49:35Z

Thanks @dnephin I'll address your review comments and add some unit tests tomorrow

dave-tucker · 2017-05-17T14:59:55Z

@dnephin comments addressed. I realised I'd touched vendored code directly when git did it's apply magic. I've opened moby/moby#33253 to cover the moby/moby side of the changes. This will fail CI until such time as I can vendor and updated moby/moby/client.

dnephin · 2017-05-17T15:03:42Z

We actually pass in host from cli/command/cli.go, so we could refactor that to store the Host on ServerInfo struct, instead of pulling it from the client.

dnephin · 2017-05-17T15:05:19Z

cli/config/configfile/file.go

@@ -152,6 +163,39 @@ func (configFile *ConfigFile) Save() error {
 	return configFile.SaveToWriter(f)
 }

+// ParseProxyConfig computes proxy configuration by retreiving the config for the provided host and
+// then checking this against any environment variables provided to the container
+func (configFile *ConfigFile) ParseProxyConfig(host string, runOpts opts.ListOpts) map[string]*string {


This function only needs a []string since it's calling GetAll() on runOpts and not using it in any other way. Could we have it accept a []string instead? That should make testing easier as well.

dnephin · 2017-05-17T15:08:18Z

cli/config/configfile/file_test.go

+	proxyConfig := cfg.ParseProxyConfig("/var/run/docker.sock", opts.NewListOpts(nil))
+
+	if *proxyConfig["HTTP_PROXY"] != httpProxy {
+		t.Fatalf("HTTP_PROXY value is incorrect.\nGot: %s\nExpected: %s\n", *proxyConfig["HTTP_PROXY"], httpProxy)


This would be a lot less verbose, and would avoid any misleading error messages by using assertions:

assert.Equal(t, httpProxy, proxyConfig["HTTP_PROXY"]) ...

You can even do it as a single assertion by creating an "expected" struct to match proxyConfig:

assert.Equal(t, expected, proxyConfig)

codecov-io · 2017-05-24T11:02:54Z

Codecov Report

Merging #93 into master will increase coverage by 0.04%.
The diff coverage is 59.37%.

@@            Coverage Diff             @@
##           master      #93      +/-   ##
==========================================
+ Coverage   45.67%   45.72%   +0.04%     
==========================================
  Files         171      171              
  Lines       11513    11542      +29     
==========================================
+ Hits         5259     5278      +19     
- Misses       5947     5956       +9     
- Partials      307      308       +1

dave-tucker · 2017-05-24T11:18:11Z

@dnephin requested changes made. I've also vendored a newer moby/moby to pick up moby/moby#33253. CI is now green ✅

dave-tucker · 2017-06-06T09:46:34Z

ping @dnephin - does this need more attention or is it good to merge?

dnephin

Design LGTM

cc @tonistiigi

tonistiigi · 2017-06-06T16:57:34Z

LGTM

This commit modifies config.json to allow for any proxies allowed in build-args to be configured. These values will then be used by default as build-args in docker build. Signed-off-by: Dave Tucker <dt@docker.com>

dave-tucker · 2017-06-20T08:53:34Z

Rebased. It'd be nice for this to merge soon, assuming it passes CI 🙏

cpuguy83

LGTM

thaJeztah · 2017-07-21T21:11:20Z

ping @dave-tucker @mstanleyjones looks like this change didn't make it into the documentation; there's a section in the reference docs describing the options that are available in config.json https://github.com/docker/cli/blob/master/docs/reference/commandline/cli.md

As per the implementation in docker/cli#93 the proxy server addresses are specified per docker daemon or as defaults. Updated the example to show the default option. Fixes docker#4686

* Update documentation of proxy server configuration As per the implementation in docker/cli#93 the proxy server addresses are specified per docker daemon or as defaults. Updated the example to show the default option. Fixes #4686 * Fix table layout of proxy environment variables

Docker 17.07 and up allow the CLI to be configured to set default proxy env-vars to be used (both as build-arg and as env for docker run), see docker/cli#93, so setting these here should be redundant. If someone needs these env-vars set, they should be configured in the cli's `~/.docker/config.json` instead. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Docker 17.07 and up allow the CLI to be configured to set default proxy env-vars to be used (both as build-arg and as env for docker run), see docker/cli#93, so setting these here should be redundant. If someone needs these env-vars set, they should be configured in the cli's `~/.docker/config.json` instead. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: ec9c8545f8408e747f283a33d4552187f6cec13f Component: engine

Docker 17.07 and up allow the CLI to be configured to set default proxy env-vars to be used (both as build-arg and as env for docker run), see docker/cli#93, so setting these here should be redundant. If someone needs these env-vars set, they should be configured in the cli's `~/.docker/config.json` instead. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

GordonTheTurtle added the status/0-triage label May 16, 2017

dave-tucker force-pushed the proxyConfig branch from cf31ff2 to 9ddc38e Compare May 16, 2017 15:11

dnephin reviewed May 16, 2017

View reviewed changes

dnephin added status/2-code-review status/1-design-review and removed status/0-triage status/2-code-review labels May 16, 2017

dnephin reviewed May 16, 2017

View reviewed changes

dave-tucker mentioned this pull request May 17, 2017

client: Expose ClientHost to client users moby/moby#33253

Merged

dave-tucker force-pushed the proxyConfig branch from 9ddc38e to 03aba8f Compare May 17, 2017 14:54

dnephin reviewed May 17, 2017

View reviewed changes

dave-tucker force-pushed the proxyConfig branch 2 times, most recently from 3de2451 to 10f5490 Compare May 24, 2017 11:02

dnephin reviewed Jun 6, 2017

View reviewed changes

Allow Proxy Configuration in config.json

35f1e30

This commit modifies config.json to allow for any proxies allowed in build-args to be configured. These values will then be used by default as build-args in docker build. Signed-off-by: Dave Tucker <dt@docker.com>

dave-tucker force-pushed the proxyConfig branch from 10f5490 to 35f1e30 Compare June 20, 2017 08:52

cpuguy83 approved these changes Jun 20, 2017

View reviewed changes

cpuguy83 merged commit 6f6ccbd into docker:master Jun 20, 2017

GordonTheTurtle added this to the 17.07.0 milestone Jun 20, 2017

thaJeztah added impact/changelog docs/revisit labels Jul 21, 2017

alb-i986 mentioned this pull request Sep 6, 2017

Proxy settings in config.json not working: "cannot unmarshal string into Go struct field ConfigFile.proxies of type configfile.ProxyConfig" #507

Closed

somcsel mentioned this pull request Sep 21, 2017

Documentation of proxy configuration via config.json does not match the implementation docker/docs#4686

Closed

somcsel mentioned this pull request Sep 21, 2017

Update documentation of proxy server configuration docker/docs#4687

Merged

dave-tucker mentioned this pull request Sep 22, 2017

Improve UX around http-proxy for those in corporate environment moby/moby#30323

Open

thaJeztah mentioned this pull request Dec 5, 2017

Add kubernetes support to docker/cli #721

Merged

thaJeztah mentioned this pull request Dec 15, 2017

Proxy configuration is now in stable as well docker/docs#5520

Merged

djs55 mentioned this pull request Jan 11, 2018

HTTP Proxy support is broken after upgrade to Version 17.12.0-ce-mac46 (21698) docker/for-mac#2386

Closed

danielpanteleit mentioned this pull request Jan 23, 2018

Support for HTTP proxy configuration via config.json docker/compose#5601

Closed

thaJeztah mentioned this pull request Jan 25, 2018

docker build auto pass proxy env vars moby/moby#36102

Closed

thaJeztah mentioned this pull request Feb 13, 2018

Docker run works behind proxy, and Docker build does not moby/moby#24697

Closed

thaJeztah mentioned this pull request Dec 13, 2018

Fix proxy-configuration being ignored on docker create #1573

Merged

thaJeztah mentioned this pull request Jan 19, 2019

Enhancement: add proxy configuration to contexts #1621

Open

thaJeztah mentioned this pull request Mar 14, 2019

Add the ability to specify global build args moby/moby#27949

Closed

thaJeztah mentioned this pull request Aug 31, 2019

Cannot initiate the connection to archive.ubuntu.com:80 moby/moby#20394

Closed

This was referenced Oct 28, 2019

Support host.docker.internal in dockerd on Linux moby/moby#40007

Merged

Pushing Environment Variables to all Docker Containers moby/moby#16699

Closed

thaJeztah mentioned this pull request Dec 10, 2019

Add docs for proxy configuration in config.json #2224

Merged

thaJeztah mentioned this pull request Jan 17, 2020

[19.03 backport] assorted documentation changes #2268

Merged

thaJeztah mentioned this pull request Jul 1, 2020

Publish the latest updates from master docker/docs#11070

Merged

ChristianTackeGSI mentioned this pull request Sep 22, 2020

FR: Read /etc/docker/config.json in addition to ~/.docker/config.json #2738

Open

thaJeztah mentioned this pull request Nov 25, 2020

Proxy not working if password contains special character(%) docker/for-linux#1149

Closed

3 tasks

thaJeztah mentioned this pull request Sep 24, 2021

Makefile: remove passing proxy env-vars moby/moby#42880

Merged

thaJeztah mentioned this pull request Oct 20, 2021

allow setting env vars only at build time #3345

Closed

thaJeztah mentioned this pull request Feb 10, 2022

Client config.json proxy settings ignored docker/buildx#947

Closed

thaJeztah mentioned this pull request Sep 16, 2023

Set "always used build arg" in the (daemon) configuration file moby/moby#27605

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow Proxy Configuration in config.json #93

Allow Proxy Configuration in config.json #93

dave-tucker commented May 16, 2017 •

edited by thaJeztah

Loading

dnephin May 16, 2017

dnephin May 16, 2017

dnephin May 16, 2017

dave-tucker May 17, 2017

dnephin May 16, 2017

dnephin left a comment

dave-tucker commented May 16, 2017

dave-tucker commented May 17, 2017

dnephin commented May 17, 2017

dnephin May 17, 2017

dnephin May 17, 2017

codecov-io commented May 24, 2017 •

edited

Loading

dave-tucker commented May 24, 2017

dave-tucker commented Jun 6, 2017

dnephin left a comment

tonistiigi commented Jun 6, 2017

dave-tucker commented Jun 20, 2017

cpuguy83 left a comment

thaJeztah commented Jul 21, 2017

Allow Proxy Configuration in config.json #93

Allow Proxy Configuration in config.json #93

Conversation

dave-tucker commented May 16, 2017 • edited by thaJeztah Loading

dnephin May 16, 2017

Choose a reason for hiding this comment

dnephin May 16, 2017

Choose a reason for hiding this comment

dnephin May 16, 2017

Choose a reason for hiding this comment

dave-tucker May 17, 2017

Choose a reason for hiding this comment

dnephin May 16, 2017

Choose a reason for hiding this comment

dnephin left a comment

Choose a reason for hiding this comment

dave-tucker commented May 16, 2017

dave-tucker commented May 17, 2017

dnephin commented May 17, 2017

dnephin May 17, 2017

Choose a reason for hiding this comment

dnephin May 17, 2017

Choose a reason for hiding this comment

codecov-io commented May 24, 2017 • edited Loading

Codecov Report

dave-tucker commented May 24, 2017

dave-tucker commented Jun 6, 2017

dnephin left a comment

Choose a reason for hiding this comment

tonistiigi commented Jun 6, 2017

dave-tucker commented Jun 20, 2017

cpuguy83 left a comment

Choose a reason for hiding this comment

thaJeztah commented Jul 21, 2017

dave-tucker commented May 16, 2017 •

edited by thaJeztah

Loading

codecov-io commented May 24, 2017 •

edited

Loading