Not able to build FIPS compliant envoy proxy.

[abhishekguptaprog]

I am trying to build envoy proxy from release v1.28.5.

I am running into issues when working with following flags.

If I pass the flag DENVOY_SSL_FIPS , build completes but it does not generate a fips compliant image.
bazel build -c opt envoy --define boringssl=fips --define tcmalloc=gperftools --config=clang --copt=-DENVOY_SSL_FIPS

If I use this flag --define boringssl=fips, I get below error.

[agup@ad4e77b48ae7 envoy]$ bazel build -c opt envoy --define boringssl=fips --define tcmalloc=gperftools --config=clang --verbose_failures
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:envoy (1001 packages loaded, 50295 targets configured).
INFO: Found 1 target...
ERROR: /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/external/boringssl_fips/BUILD.bazel:25:8: Executing genrule @boringssl_fips//:build failed: (Exit 7): bash failed: error executing command (from target @boringssl_fips//:build) 
  (cd /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/1/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm \
    CC=clang \
    CXX=clang++ \
    LLVM_CONFIG=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin/llvm-config \
    PATH=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin:/usr/share/Modules/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/home/agup/.local/bin:/home/agup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel/external/boringssl_fips.genrule_cmd bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a bazel-out/k8-opt/bin/external/boringssl_fips/ssl/libssl.a')

Configuration: 1bf6ab3752083dda4f42398f340543e9d61f03a62240966fd97985f371f1f1d8
 Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/1/execroot/envoy/external/boringssl_fips /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/1/execroot/envoy
Target //source/exe:envoy-static failed to build
ERROR: /home/agup/envoy/source/exe/BUILD:25:16 Linking source/exe/envoy-static failed: (Exit 7): bash failed: error executing command (from target @boringssl_fips//:build) 
  (cd /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/1/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm \
    CC=clang \
    CXX=clang++ \
    LLVM_CONFIG=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin/llvm-config \
    PATH=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin:/usr/share/Modules/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/home/agup/.local/bin:/home/agup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel/external/boringssl_fips.genrule_cmd bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a bazel-out/k8-opt/bin/external/boringssl_fips/ssl/libssl.a')

Configuration: 1bf6ab3752083dda4f42398f340543e9d61f03a62240966fd97985f371f1f1d8
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 105.672s, Critical Path: 65.05s
INFO: 22 processes: 5 internal, 17 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

I have tried to use clang 14+ and 15+
linux ninja version is 1.12.1

[abhishekguptaprog]

Another concern i have is with the documentation provided here.
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl.html
It says that boringssl=fips is only supported for x86_64 arch. We need to build image for both arm and amd64.

BoringSSL can be built in a [FIPS-compliant mode](https://boringssl.googlesource.com/boringssl/+/master/crypto/fipsmodule/FIPS.md), following the build instructions from the [Security Policy for BoringCrypto module](https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3678.pdf), using --define boringssl=fips Bazel option. Currently, this option is only available on Linux-x86_64.

[phlax]

llvm used by our ci when building/testing this is currently 14.0.0 (same in 1.28) - did you try using the envoy build container - ie ./ci/run_envoy_docker.sh - this would guarantee correct host versions

its not immediately obvious from the posted logs/error what the issue is

re arm support i believe this should be doable - altho may require some arm-specific setup for building the boringssl module

there is a ticket here related to arm/fips support #27620

cc @ggreenway

[phlax]

for ref these are the flags that we currently test the fips build with

envoy/.bazelrc

Lines 352 to 370 in ce53be3

	## Compile-time-options testing
	# Right now, none of the available compile-time options conflict with each other. If this
	# changes, this build type may need to be broken up.
	build:compile-time-options --define=admin_html=disabled
	build:compile-time-options --define=signal_trace=disabled
	build:compile-time-options --define=hot_restart=disabled
	build:compile-time-options --define=google_grpc=disabled
	build:compile-time-options --define=boringssl=fips
	build:compile-time-options --define=log_debug_assert_in_release=enabled
	build:compile-time-options --define=path_normalization_by_default=true
	build:compile-time-options --define=deprecated_features=disabled
	build:compile-time-options --define=tcmalloc=gperftools
	build:compile-time-options --define=zlib=ng
	build:compile-time-options --define=uhv=enabled
	build:compile-time-options --config=libc++20
	build:compile-time-options --test_env=ENVOY_HAS_EXTRA_EXTENSIONS=true
	build:compile-time-options --@envoy//bazel:http3=False
	build:compile-time-options --@envoy//source/extensions/filters/http/kill_request:enabled

[abhishekguptaprog]

run_envoy_docker.sh requires docker support on VM. We do not have access to docker on ol8.
We have access to podman.
And i see the --define=boringssl=fips in .bazelrc as well

[phlax]

ol8?

you most likely can use the envoy build container with podman - without using the script

[abhishekguptaprog]

Sorry for the confusion, let me give some more details.
I have hosted linux box with Linux 7. Linux 7 has support for docker.
Our Linux box 8 do not support docker any more.
In production environment we use linux 8 images.
currently I am using a docker container with Linux 8 image and building the proxy inside the container on my hosted linux 7.
without --define=boringssl=fips flag, it does generate the envoy-static but when i check version its not BORINGSSL-FIPS.
Is there any thing we can twist in our options to make this work?

[phlax]

Linux 7

are you referring to redhat or somesuch - pretty sure Linux 7 doesnt exist

we provide a build container both to allow building in a variety of environments and as a canonical source of build requirements

without --define=boringssl=fips flag

iiuc this will not build a fips-compliant binary

Is there any thing we can twist in our options to make this work?

the flags posted above are what we test with - unfortunately more than just the fips build are being tested there so most may not be necessary

[abhishekguptaprog]

Yes its Rhel based linux distribution.

[abhishekguptaprog]

Can we customize run_envoy_docker to pull which base linux image we want to pull and run the build on ?

[phlax]

you can - atm its not ideally set up for this - but the build image is set by this line

envoy/ci/run_envoy_docker.sh

Line 91 in 89f0328

export ENVOY_BUILD_IMAGE="${IMAGE_NAME}:${IMAGE_ID}"

not sure how that helps tho - the official build image guarantees tested host versions and that script requires docker rather than podman

[ggreenway]

The FIPS build is always a bit fragile because the FIPS components require a specific compiler and toolchain, which is different from what the rest of Envoy is compiled with. One thing that may help is trying to compile Envoy with the FIPS-required compiler (https://github.com/envoyproxy/envoy/blob/release/v1.28/bazel/external/boringssl_fips.genrule_cmd#L35).

[abhishekguptaprog]

But documentation says to use clang version 14+ ?

[phlax]

FIPS determines that the crypto lib must be built with clang 14.0.0 (afaiaa)

Envoy currently uses clang 14.0.0 for the rest of the build

in my testing of trying to update clang elsewhere it has failed - iirc when it tries to link the built crypto libs - so for the avoidance of issues best thing is to make sure you are building with clang 14.0.0 everywhere

[abhishekguptaprog]

It seems to have passed the stage where it used to fail after changing value to 12.0.0. Lets see if the envoy-static gets generated and what is the version.

[abhishekguptaprog]

It failed with same error. It is still showing version of clang used as 15.
Is it because of executing bazel/setup_clang.sh ?
I had changed VERSION=12.0.0 in
bazel/external/boringssl_fips.genrule_cmd

[phlax]

I had changed VERSION=12.0.0 in
bazel/external/boringssl_fips.genrule_cmd

i believe this would make the binary non-FIPs compliant

i think you need to leave the genrule_cmd alone and just make sure you have llvm 14 installed on your host/build system

this is the known good setup, if you still have problems with this, at least we can compare to the known working baseline

[abhishekguptaprog]

Can you please suggest me what can I do to fix my build. I want to pick up patches applied in 1.20.x because of which I had picked 1.28.5 envoy.
Please let me know if you want any other info.

[abhishekguptaprog]

I had tried to use ci/run_envoy_docker.sh and ran into multiple issues while trying to change the base container image.

[phlax]

the point about using the container is specifically not to use your own container image - its to start with an environment that is tested and known to work - im struggling to understand why you would want to do that

[abhishekguptaprog]

In production we are using only RHEL based oracle linux containers.
That is a requirement for us.

[phlax]

but you dont need to build with that - certainly at least while testing your build setup

[abhishekguptaprog]

I had executed again with --sandbox_debug
the output is very similar to issue below.

https://www.reddit.com/r/EnvoyProxy/comments/1ecu6q0/envoy_build_with_boringsslfips_failed/?rdt=62982

how do we configure the correct boring ssl version/path.

[agup@ad4e77b48ae7 envoy]$ bazel build -c opt envoy --define boringssl=fips --define tcmalloc=gperftools --config=clang --verbose_failures --sandbox_debug
INFO: Analyzed target //:envoy (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Action external/com_google_googleapis/google/devtools/cloudtrace/v2/trace.grpc.pb.h:
bazel-out/k8-opt/bin/external/com_google_googleapis/external/com_google_googleapis: warning: directory does not exist.
INFO: From Action external/opencensus_proto/opencensus/proto/agent/trace/v1/trace_service.grpc.pb.h:
bazel-out/k8-opt/bin/external/opencensus_proto/external/opencensus_proto: warning: directory does not exist.
ERROR: /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/external/boringssl_fips/BUILD.bazel:25:8: Executing genrule @boringssl_fips//:build failed: (Exit 7): process-wrapper failed: error executing command 
  (cd /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm \
    CC=clang \
    CXX=clang++ \
    LLVM_CONFIG=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin/llvm-config \
    PATH=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin:/usr/share/Modules/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/home/agup/.local/bin:/home/agup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    TMPDIR=/tmp \
  /home/agup/.cache/bazel/_bazel_agup/install/a09dbb90c658248f08f9aa0eba11997d/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/stats.out' /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel/external/boringssl_fips.genrule_cmd bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a bazel-out/k8-opt/bin/external/boringssl_fips/ssl/libssl.a')
/home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/execroot/envoy/external/boringssl_fips /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/execroot/envoy
Target //source/exe:envoy-static failed to build
ERROR: /home/agup/envoy/source/exe/BUILD:25:16 Linking source/exe/envoy-static failed: (Exit 7): process-wrapper failed: error executing command 
  (cd /home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm \
    CC=clang \
    CXX=clang++ \
    LLVM_CONFIG=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin/llvm-config \
    PATH=/home/agup/clang+llvm-15.0.0-x86_64-linux-gnu-rhel-8.4/bin:/usr/share/Modules/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/home/agup/.local/bin:/home/agup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    TMPDIR=/tmp \
  /home/agup/.cache/bazel/_bazel_agup/install/a09dbb90c658248f08f9aa0eba11997d/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/home/agup/.cache/bazel/_bazel_agup/5a4ae7bd0cdd0afbae2fbe185187e245/sandbox/processwrapper-sandbox/2480/stats.out' /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel/external/boringssl_fips.genrule_cmd bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a bazel-out/k8-opt/bin/external/boringssl_fips/ssl/libssl.a')
INFO: Elapsed time: 1839.665s, Critical Path: 131.29s
INFO: 4698 processes: 2485 internal, 1 local, 2211 processwrapper-sandbox, 1 worker.
FAILED: Build did NOT complete successfully

[phlax]

if you dont follow the steps i suggested im not sure i can help

[abhishekguptaprog]

i think you need to leave the genrule_cmd alone and just make sure you have llvm 14 installed on your host/build system

how do we leave out the genrule_cmd? I had tried with llvm 14 as well. I can revert back to 14.

[phlax]

start with what works - use the build container, dont change any versions and use ~the same flags as tested in our ci

once you have a working build, you can start to change things

[phlax]

re genrule_cmd - that is what ~guarantees the FIPS-compliance - unless you really know what you are doing you should not change anything there

[ggreenway]

re genrule_cmd - that is what ~guarantees the FIPS-compliance - unless you really know what you are doing you should not change anything there

Stated even more directly: if you change anything in the FIPS genrule, it is unlikely you'll get a FIPS-compliant build.