Problem: dependent build fails behind an unauthenticated proxy

[d-led]

Problem

During a CI build behind a proxy (no auth parameters, just HTTPS_PROXY set, getting dependencies works, however building cldr fails with:

[error] Timeout connecting to 'raw.githubusercontent.com' to download 'https://raw.githubusercontent.com/elixir-cldr/cldr/v2.37.0/priv/cldr/locales/de.json'. Connection time exceeded 60000ms.
Generating MyModule.Cldr for 3 locales named [:de, :en, :und] with a default locale named :en
== Compilation error in file lib/my_module/cldr.ex ==
** (RuntimeError) Locale definition was not found for :de
    (ex_cldr 2.37.0) lib/cldr/locale/loader.ex:109: Cldr.Locale.Loader.do_get_locale/2
    (ex_cldr 2.37.0) lib/cldr/config/config.ex:2315: Cldr.Config.decimal_formats_for/2
    (elixir 1.14.4) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (ex_cldr 2.37.0) lib/cldr/config/config.ex:2302: Cldr.Config.decimal_format_list/1
    (ex_cldr_numbers 2.31.0) lib/cldr/number/formatter/decimal_formatter.ex:935: Cldr.Number.Formatter.Decimal.define_to_string/1
    (ex_cldr_numbers 2.31.0) lib/cldr/number/backend/decimal_formatter.ex:52: Cldr.Number.Backend.Decimal.Formatter.define_number_module/1
    (ex_cldr_numbers 2.31.0) lib/cldr/number/backend/compiler.ex:11: Cldr.Number.Backend.define_number_modules/1
    (ex_cldr 2.37.0) lib/cldr/config/config.ex:2632: anonymous fn/3 in Cldr.Config.define_provider_modules/1
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

as httpc options are not set in Cldr.Install.install_locale_name

Solution

• Try setting a proxy during build
  • similar to esbuild or mix
• If anyone needs further options for httpc, another mechanism might be preferable

[d-led]

no scheme selection as the base_url is known in advance

[kipcole9]

@d-led , thanks for very for the PR. I think this is best added to cldr_utils since that provides HTTP support for the Cldr family. I'll leave this PR open until I make the change in cldr_utils and will close this PR then.

Thanks for the contribution, greatly appreciated.

[kipcole9]

I've published cldr_utils version 2.23.0 that incorporates your PR. Would you consider mix deps.update cldr_utils and confirming it operates as you expect?

I have also added a commit that adds :http_proxy as a valid configuration key in config.exs under the :ex_cldr key.

[d-led]

Makes sense. Thx. Will do now

[d-led]

@kipcole9 the upgraded cldr_utils fix the problem (in the docker/kaniko build with https_proxy set). Thanks!

[d-led]

getting a strange build in local docker without proxy. gotta follow up:

#0 0.566 --== Compiling ==--
#0 1.937 Compiling 21 files (.ex)
#0 2.577 
#0 2.577 == Compilation error in file lib/my_module/cldr.ex ==
#0 2.577 ** (FunctionClauseError) no function clause matching in String.to_charlist/1    
#0 2.577     
#0 2.577     The following arguments were given to String.to_charlist/1:
#0 2.577     
#0 2.577         # 1
#0 2.577         nil
#0 2.577     
#0 2.577     Attempted function clauses (showing 1 out of 1):
#0 2.577     
#0 2.577         def to_charlist(+string+) when -is_binary(string)-
#0 2.577     
#0 2.577     (elixir 1.14.4) lib/string.ex:2530: String.to_charlist/1
#0 2.577     (cldr_utils 2.23.0) lib/cldr/http/http.ex:271: Cldr.Http.get_with_headers/2
#0 2.577     (cldr_utils 2.23.0) lib/cldr/http/http.ex:120: Cldr.Http.get/2
#0 2.577     (ex_cldr 2.37.0) lib/cldr/install.ex:104: Cldr.Install.do_install_locale_name/3
#0 2.577     (elixir 1.14.4) lib/enum.ex:975: Enum."-each/2-lists^foreach/1-0-"/2
#0 2.577     (ex_cldr 2.37.0) lib/cldr/install.ex:29: Cldr.Install.install_known_locale_names/1
#0 2.577     (ex_cldr 2.37.0) lib/cldr.ex:82: Cldr.install_locales/1
#0 2.577     (ex_cldr 2.37.0) expanding macro: Cldr.Backend.Compiler.__before_compile__/1

no such error on mix compile or mix phx.server.

I seem to be unable to pin the problem down. Both environment variables are not set in the docker build. Why does the http.ex:271 case match at all 🤔 ?

↓

the env var was set to whitespace somehow. Not a fault in cldr_utils

[d-led]

upd: HTTPS_PROXY seems to be set to whitespace in my local docker setup somehow, thus causing esbuild proxy logic to fail too. I think, the PR can be closed.

[kipcole9]

Thanks @d-led, I'll close the PR. And make the code more resilient to invalid HTTP_PROXY env vars in cldr_utils. Thanks for your patience.

[kipcole9]

I've published cldr_utils version 2.23.1 that makes Cldr.Http.get/2 resilient in the face of invalid https proxy URLs. Hopefully that gets you back in business. If the proxy URL is invalid then that fact is logged and to request continues without the proxy being set (and therefore possibly still failing).

[d-led]

superb. Thanks!

For posterity: the combination that didn't work:

Dockerfile:

ARG HTTPS_PROXY=
ENV HTTPS_PROXY=${HTTPS_PROXY}
ENV https_proxy=${HTTPS_PROXY}

#...
RUN ./build_release.sh

build_release.sh:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

//...

mix.compile

for some reason, the variables seem to be set, although empty, causing System.get_env("HTTPS_PROXY") not to return nil but an empty string (needs follow up in erlang, I think).

Fixed the build script with the following:

if [[ "$HTTPS_PROXY" = "" ]]; then unset HTTPS_PROXY; fi