test/streaming.cc takes minutes to compile #275

CodingCanuck · 2022-03-28T21:26:40Z

This file: https://github.com/3rdparty/eventuals-grpc/blob/main/test/streaming.cc

Takes several minutes to compile as part of this build:

bazel test //test:grpc

Sample output from a not-yet-done bazel build with default compilation option (5 minutes and counting!):

[22 / 24] Compiling test/streaming.cc; 307s processwrapper-sandbox

My wild guess as to what's going on: we have some long eventual chains in this file. Does that result in slow compilation times due to heavy template nesting? If so, here's a resource on how to profile time spent in the compiler: stackoverflow.com/questions/15818281/profiling-template-metaprogram-compilation-time

It looks like we're building using GCC. I wonder if Clang's any faster?

(This issue is a recreated version of 3rdparty/eventuals-grpc#73 now that eventuals-grpc is a read-only repo)

The text was updated successfully, but these errors were encountered:

CodingCanuck · 2022-03-28T21:26:53Z

From an offline discussion with @benh : a likely hypothesis is that the long eventuals chains in streaming.cc are indeed causing slow compilation times.

One workaround he proposes is to split streaming.cc into many files (one file per test) so there's at most one long eventual per test (a similar thing was done for eventuals tests in #182 ).

CodingCanuck · 2022-04-05T19:50:29Z

Compiling the output files generated by cc_eventuals_library also takes a super long time, e.g.: https://github.com/reboot-dev/respect/runs/5839858123?check_suite_focus=true

[1,964 / 2,126] Compiling external/com_github_reboot_dev_respect_api/respect/v1/object_store.eventuals.cc; 312s local, remote-cache ... (4 actions running)

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck · 2022-04-11T20:58:47Z

Hmm, I'm no longer able to reproduce these multi-minute compile times.

clang (3 runs):

3 runs of:
echo "" >> test/streaming.cc && time dazel build --config=clang //test:grpc

real    0m30.302s
user    0m0.143s
sys     0m0.062s

real    0m29.914s
user    0m0.128s
sys     0m0.083s

real    0m30.148s
user    0m0.123s
sys     0m0.086s

gcc (3 runs):

1 run of:
echo "" >> test/streaming.cc && time dazel build //test:grpc

real    0m39.810s
user    0m0.140s
sys     0m0.068s

real    0m39.954s
user    0m0.127s
sys     0m0.088s

real    0m39.792s
user    0m0.113s
sys     0m0.096s

I'm going to close this: 30s isn't great but it's tolerable.

(I think that compiling code generated from cc_eventuals_library is still slow, but we can file additional issues for that as needed)

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck · 2022-04-28T18:06:40Z

@onelxj it sounds like you're working on the slow compile times of foo.eventuals.cc noted in #275 (comment) : nice!!

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc. TESTED=added a clang-only change to `eventuals/just.h` and saw the build succeed on all platforms: https://github.com/3rdparty/eventuals/actions/runs/2598422120 Clang-only change was: ```c++ ```

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc. Make the `build_and_test` action run a clean before building so that output from one compiler isn't used in a build for a different compiler. This fixes the build errors reported in #324. TESTED=added a clang-only change to `eventuals/just.h` and saw the build succeed on all platforms: https://github.com/3rdparty/eventuals/actions/runs/2598422120 Clang-only change was: ```c++ \#ifndef __clang__ \#error "Must compile with clang" \#endif ``` Fixes #324. Fixes #331.

CodingCanuck · 2022-07-11T16:55:17Z

test/grpc/streaming.cc is also showing up in these strange compilation failures: https://github.com/3rdparty/eventuals/runs/7286455244?check_suite_focus=true

[3,030 / 3,044] Compiling test/grpc/streaming.cc; 90s local, remote-cache ... (4 actions running)
[3,038 / 3,044] Compiling test/grpc/streaming.cc; 250s local, remote-cache ... (3 actions, 2 running)
ERROR: /home/runner/work/eventuals/eventuals/test/grpc/BUILD.bazel:35:8: Compiling test/grpc/streaming.cc failed: (Exit 1): gcc failed: error executing command 
  (cd /home/runner/.cache/dazel/eventuals/execroot/com_github_3rdparty_eventuals && \
  exec env - \
    PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    *** \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g '-std=c++0x' -MD -MF bazel-out/k8-dbg/bin/test/grpc/_objs/grpc/streaming.pic.d '-frandom-seed=bazel-out/k8-dbg/bin/test/grpc/_objs/grpc/streaming.pic.o' -fPIC '-DRAPIDJSON_HAS_STDSTRING=1' -DGLOG_NO_ABBREVIATED_SEVERITIES -DASIO_STANDALONE -DCURL_STATICLIB -DCARES_STATICLIB -iquote . -iquote bazel-out/k8-dbg/bin -iquote external/com_github_3rdparty_stout -iquote bazel-out/k8-dbg/bin/external/com_github_3rdparty_stout -iquote external/com_github_google_glog -iquote bazel-out/k8-dbg/bin/external/com_github_google_glog -iquote external/com_github_gflags_gflags -iquote bazel-out/k8-dbg/bin/external/com_github_gflags_gflags -iquote external/com_google_protobuf -iquote bazel-out/k8-dbg/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-dbg/bin/external/zlib -iquote external/com_google_absl -iquote bazel-out/k8-dbg/bin/external/com_google_absl -iquote external/boost -iquote bazel-out/k8-dbg/bin/external/boost -iquote external/com_github_kazuho_picojson -iquote bazel-out/k8-dbg/bin/external/com_github_kazuho_picojson -iquote external/com_github_tencent_rapidjson -iquote bazel-out/k8-dbg/bin/external/com_github_tencent_rapidjson -iquote external/com_github_tl_expected -iquote bazel-out/k8-dbg/bin/external/com_github_tl_expected -iquote external/com_github_chriskohlhoff_asio -iquote bazel-out/k8-dbg/bin/external/com_github_chriskohlhoff_asio -iquote external/boringssl -iquote bazel-out/k8-dbg/bin/external/boringssl -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-dbg/bin/external/com_github_grpc_grpc -iquote external/com_google_googleapis -iquote bazel-out/k8-dbg/bin/external/com_google_googleapis -iquote external/upb -iquote bazel-out/k8-dbg/bin/external/upb -iquote external/com_github_cncf_udpa -iquote bazel-out/k8-dbg/bin/external/com_github_cncf_udpa -iquote external/com_envoyproxy_protoc_gen_validate -iquote bazel-out/k8-dbg/bin/external/com_envoyproxy_protoc_gen_validate -iquote external/com_github_cares_cares -iquote bazel-out/k8-dbg/bin/external/com_github_cares_cares -iquote external/envoy_api -iquote bazel-out/k8-dbg/bin/external/envoy_api -iquote external/opencensus_proto -iquote bazel-out/k8-dbg/bin/external/opencensus_proto -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-dbg/bin/external/com_googlesource_code_re2 -iquote external/com_github_google_googletest -iquote bazel-out/k8-dbg/bin/external/com_github_google_googletest -iquote external/bazel_tools -iquote bazel-out/k8-dbg/bin/external/bazel_tools -Ibazel-out/k8-dbg/bin/external/com_github_google_glog/_virtual_includes/glog -Ibazel-out/k8-dbg/bin/external/com_github_gflags_gflags/_virtual_includes/gflags -Ibazel-out/k8-dbg/bin/external/com_google_protobuf/_virtual_includes/descriptor_proto -isystem external/com_github_3rdparty_stout/include -isystem bazel-out/k8-dbg/bin/external/com_github_3rdparty_stout/include -isystem external/com_google_protobuf/src -isystem bazel-out/k8-dbg/bin/external/com_google_protobuf/src -isystem external/zlib/zlib/include -isystem bazel-out/k8-dbg/bin/external/zlib/zlib/include -isystem external/boost -isystem bazel-out/k8-dbg/bin/external/boost -isystem external/boost/libs/math/include_private -isystem bazel-out/k8-dbg/bin/external/boost/libs/math/include_private -isystem external/com_github_kazuho_picojson -isystem bazel-out/k8-dbg/bin/external/com_github_kazuho_picojson -isystem external/com_github_tencent_rapidjson/include -isystem bazel-out/k8-dbg/bin/external/com_github_tencent_rapidjson/include -isystem external/com_github_tl_expected/include -isystem bazel-out/k8-dbg/bin/external/com_github_tl_expected/include -isystem bazel-out/k8-dbg-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/include -isystem external/com_github_chriskohlhoff_asio/asio/include -isystem bazel-out/k8-dbg/bin/external/com_github_chriskohlhoff_asio/asio/include -isystem bazel-out/k8-dbg/bin/external/com_github_libuv_libuv/libuv/include -isystem bazel-out/k8-dbg/bin/external/com_github_curl_curl/libcurl_boringssl_linux_macos/include -isystem external/boringssl/src/include -isystem bazel-out/k8-dbg/bin/external/boringssl/src/include -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-dbg/bin/external/com_github_grpc_grpc/include -isystem external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem bazel-out/k8-dbg/bin/external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem external/com_github_grpc_grpc/src/core/ext/upbdefs-generated -isystem bazel-out/k8-dbg/bin/external/com_github_grpc_grpc/src/core/ext/upbdefs-generated -isystem external/com_github_grpc_grpc/third_party/xxhash -isystem bazel-out/k8-dbg/bin/external/com_github_grpc_grpc/third_party/xxhash -isystem external/com_github_grpc_grpc/third_party/address_sorting/include -isystem bazel-out/k8-dbg/bin/external/com_github_grpc_grpc/third_party/address_sorting/include -isystem external/com_github_cares_cares/include -isystem bazel-out/k8-dbg/bin/external/com_github_cares_cares/include -isystem external/com_github_cares_cares -isystem bazel-out/k8-dbg/bin/external/com_github_cares_cares -isystem external/com_github_google_googletest/googlemock -isystem bazel-out/k8-dbg/bin/external/com_github_google_googletest/googlemock -isystem external/com_github_google_googletest/googlemock/include -isystem bazel-out/k8-dbg/bin/external/com_github_google_googletest/googlemock/include -isystem external/com_github_google_googletest/googletest -isystem bazel-out/k8-dbg/bin/external/com_github_google_googletest/googletest -isystem external/com_github_google_googletest/googletest/include -isystem bazel-out/k8-dbg/bin/external/com_github_google_googletest/googletest/include '-fsanitize=address' -DADDRESS_SANITIZER -O1 -g -fno-omit-frame-pointer '-std=c++17' -Werror -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c test/grpc/streaming.cc -o bazel-out/k8-dbg/bin/test/grpc/_objs/grpc/streaming.pic.o)
# Configuration: 89bbfd33b1d499fc6805225676f8c4fe1aaa564b0024adb28c700d0f73feea6a
# Execution platform: @local_config_platform//:host
gcc: fatal error: Killed signal terminated program cc1plus
compilation terminated.
INFO: Elapsed time: 1238.193s, Critical Path: 372.03s
INFO: 3043 processes: 2337 remote cache hit, 651 internal, 55 local.
FAILED: Build did NOT complete successfully

https://stackoverflow.com/a/67644572 makes me think that this error occurs when the compiler OOMs.

So I think in addition to slow compilation time, complex eventuals structures are also hitting the compiler's memory limits.

I'm optimistic that clang might also help here, but it does worry me that non-trivial eventuals use stretches C++ compilers to their limits: this might make the compilation experience pretty rough for a non-trivial codebase that makes heavy use of eventuals.

(Notably, the above compilation error occurs when using --config=asan under dazel)

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc. Make the `build_and_test` action run a clean before building so that output from one compiler isn't used in a build for a different compiler. This fixes the build errors reported in #324. TESTED=added a clang-only change to `eventuals/just.h` and saw the build succeed on all platforms: https://github.com/3rdparty/eventuals/actions/runs/2598422120 Clang-only change was: ```c++ \#ifndef __clang__ \#error "Must compile with clang" \#endif ``` Fixes #324. Fixes #331.

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc. Remove the `--spawn_strategy=local` flag, allowing bazel to default to what I think is a `sandboxed` strategy (which is preferred above `local` in bazel's automatic strategy selection: https://blog.bazel.build/2019/06/19/list-strategy.html#after-bazel-0-27 ). This `spawn_strategy` change fixes the build errors reported in #324. TESTED=added a clang-only change to `eventuals/just.h` and saw the build succeed on all platforms: https://github.com/3rdparty/eventuals/actions/runs/2598422120 Clang-only change was: ```c++ \#ifndef __clang__ \#error "Must compile with clang" \#endif ``` Fixes #324.

CodingCanuck added a commit that referenced this issue Apr 11, 2022

Compile with clang by default

3630d11

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck mentioned this issue Apr 11, 2022

Compile with clang by default #323

Merged

CodingCanuck closed this as completed Apr 11, 2022

CodingCanuck added a commit that referenced this issue Apr 20, 2022

Compile with clang by default

c2910c3

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Apr 20, 2022

Compile with clang by default

5155627

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

2c8e9bb

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

ce11b75

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

e15d1c6

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

f3db610

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

fb5c51f

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck added a commit that referenced this issue Jul 1, 2022

Compile with clang by default

9d29bc2

This should speed up template compilation times and improve #275. This also ensures that clang is used on all platforms. Before this PR, our CI servers were building with gcc.

CodingCanuck mentioned this issue Jul 12, 2022

Flaky compiler OOM when compiling files with long eventuals pipelines #454

Open

CodingCanuck mentioned this issue Jul 20, 2022

Measure compiler time and memory usage when compiling code that uses eventuals #482

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test/streaming.cc takes minutes to compile #275

test/streaming.cc takes minutes to compile #275

CodingCanuck commented Mar 28, 2022

CodingCanuck commented Mar 28, 2022

CodingCanuck commented Apr 5, 2022

CodingCanuck commented Apr 11, 2022

CodingCanuck commented Apr 28, 2022

CodingCanuck commented Jul 11, 2022

test/streaming.cc takes minutes to compile #275

test/streaming.cc takes minutes to compile #275

Comments

CodingCanuck commented Mar 28, 2022

CodingCanuck commented Mar 28, 2022

CodingCanuck commented Apr 5, 2022

CodingCanuck commented Apr 11, 2022

CodingCanuck commented Apr 28, 2022

CodingCanuck commented Jul 11, 2022