Ubuntu build failing when VELOX_ENABLE_PARQUET flag is enabled

[agrawalreetika]

Problem description

Ubuntu build failing when VELOX_ENABLE_PARQUET flag is enabled.
#10321

System information

Operating System :
Ubuntu
22.04.4
LTS

CMake log

No response

[yingsu00]

@agrawalreetika Have you tried to manually build it on our Ubuntu fyre machine? Did it give the same error?

[agrawalreetika]

Yes @yingsu00, I tried it on Ubuntu fyre machine as well.

Command :

make NUM_THREADS=4 MAX_HIGH_MEM_JOBS=2 MAX_LINK_JOBS=2 EXTRA_CMAKE_FLAGS=" -DVELOX_DEPENDENCY_SOURCE=BUNDLED -DTREAT_WARNINGS_AS_ERRORS=1 -DENABLE_ALL_WARNINGS=1 -DVELOX_BUILD_MINIMAL=OFF -DVELOX_BUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Debug -DVELOX_ENABLE_PARQUET=ON " debug

Error :

FAILED: velox/dwio/parquet/tests/reader/velox_dwio_parquet_reader_benchmark
: && /usr/bin/c++ -mavx2 -mfma -mavx -mf16c -mlzcnt -std=c++17 -mbmi2 -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wnon-virtual-dtor        -Wno-implicit-fallthrough          -Wno-class-memaccess          -Wno-comment          -Wno-int-in-bool-context          -Wno-redundant-move          -Wno-array-bounds          -Wno-maybe-uninitialized          -Wno-unused-result          -Wno-format-overflow          -Wno-strict-aliasing -Werror -g -Wl,-export-dynamic velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/FileHandle.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HiveConfig.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HiveConnector.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HiveConnectorUtil.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HiveDataSink.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HiveDataSource.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/HivePartitionUtil.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/PartitionIdGenerator.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/SplitReader.cpp.o velox/connectors/hive/CMakeFiles/velox_hive_connector.dir/TableHandle.cpp.o velox/dwio/parquet/tests/reader/CMakeFiles/velox_dwio_parquet_reader_benchmark.dir/ParquetReaderBenchmarkMain.cpp.o -o velox/dwio/parquet/tests/reader/velox_dwio_parquet_reader_benchmark  velox/dwio/parquet/tests/reader/libvelox_dwio_parquet_reader_benchmark_lib.a  velox/dwio/parquet/libvelox_dwio_parquet_reader.a  velox/dwio/parquet/libvelox_dwio_parquet_writer.a  velox/exec/tests/utils/libvelox_exec_test_lib.a  velox/exec/libvelox_exec.a  _deps/folly-build/libfolly.a  velox/dwio/common/tests/utils/libvelox_dwio_common_test_utils.a  velox/vector/tests/utils/libvelox_vector_test_lib.a  velox/exec/tests/utils/libvelox_exec_test_lib.a  velox/exec/tests/utils/libvelox_temp_path.a  lib/libgtest.a  lib/libgtest_main.a  lib/libgmock.a  _deps/gflags-build/libgflags_nothreads_debug.a  -lstdc++fs  velox/common/file/tests/libvelox_file_test_utils.a  velox/tpch/gen/libvelox_tpch_gen.a  velox/tpch/gen/dbgen/libvelox_dbgen.a  velox/functions/prestosql/aggregates/libvelox_aggregates.a  velox/functions/lib/aggregates/libvelox_functions_aggregates.a  velox/serializers/libvelox_presto_serializer.a  velox/row/libvelox_row_fast.a  velox/dwio/parquet/libvelox_dwio_parquet_reader.a  velox/dwio/parquet/reader/libvelox_dwio_native_parquet_reader.a  velox/dwio/parquet/thrift/libvelox_dwio_parquet_thrift.a  velox/dwio/parquet/libvelox_dwio_parquet_writer.a  velox/dwio/parquet/writer/libvelox_dwio_arrow_parquet_writer.a  velox/dwio/parquet/writer/arrow/libvelox_dwio_arrow_parquet_writer_lib.a  velox/dwio/parquet/writer/arrow/generated/libvelox_dwio_arrow_parquet_writer_thrift_lib.a  velox/dwio/parquet/writer/arrow/util/libvelox_dwio_arrow_parquet_writer_util_lib.a  third_party/arrow_ep/install/lib/libarrow.a  /usr/lib/x86_64-linux-gnu/libthrift.a  velox/dwio/dwrf/writer/libvelox_dwio_dwrf_writer.a  /usr/lib/x86_64-linux-gnu/liblz4.so  /usr/lib/x86_64-linux-gnu/liblzo2.so  velox/connectors/hive/iceberg/libvelox_hive_iceberg_splitreader.a  velox/dwio/catalog/fbhive/libvelox_dwio_catalog_fbhive.a  velox/dwio/orc/reader/libvelox_dwio_orc_reader.a  velox/dwio/dwrf/reader/libvelox_dwio_dwrf_reader.a  velox/dwio/dwrf/common/libvelox_dwio_dwrf_common.a  velox/dwio/common/compression/libvelox_dwio_common_compression.a  /usr/lib/x86_64-linux-gnu/libsnappy.so  /usr/lib/x86_64-linux-gnu/libzstd.so  velox/dwio/dwrf/utils/libvelox_dwio_dwrf_utils.a  velox/dwio/dwrf/proto/libvelox_dwio_dwrf_proto.a  velox/connectors/hive/libvelox_hive_partition_function.a  velox/connectors/hive/storage_adapters/s3fs/libvelox_s3fs.a  velox/connectors/hive/storage_adapters/hdfs/libvelox_hdfs.a  velox/connectors/hive/storage_adapters/gcs/libvelox_gcs.a  velox/connectors/hive/storage_adapters/abfs/libvelox_abfs.a  velox/dwio/common/tests/utils/libvelox_dwio_common_test_utils.a  velox/parse/libvelox_parse_parser.a  velox/duckdb/conversion/libvelox_duckdb_parser.a  velox/duckdb/conversion/libvelox_duckdb_conversion.a  velox/parse/libvelox_parse_expression.a  velox/exec/libvelox_exec.a  velox/parse/libvelox_parse_utils.a  velox/functions/libvelox_function_registry.a  _deps/duckdb-build/src/libduckdb_static.a  _deps/duckdb-build/third_party/fsst/libduckdb_fsst.a  _deps/duckdb-build/third_party/fmt/libduckdb_fmt.a  _deps/duckdb-build/third_party/libpg_query/libduckdb_pg_query.a  _deps/duckdb-build/third_party/re2/libduckdb_re2.a  _deps/duckdb-build/third_party/miniz/libduckdb_miniz.a  _deps/duckdb-build/third_party/utf8proc/libduckdb_utf8proc.a  _deps/duckdb-build/third_party/hyperloglog/libduckdb_hyperloglog.a  _deps/duckdb-build/third_party/fastpforlib/libduckdb_fastpforlib.a  _deps/duckdb-build/third_party/mbedtls/libduckdb_mbedtls.a  _deps/duckdb-build/extension/jemalloc/libjemalloc_extension.a  velox/dwio/common/libvelox_dwio_common.a  velox/common/io/libvelox_common_io.a  velox/dwio/common/encryption/libvelox_dwio_common_encryption.a  _deps/protobuf-build/libprotobufd.a  /usr/lib/x86_64-linux-gnu/libz.so  velox/type/fbhive/libvelox_type_fbhive.a  velox/functions/prestosql/registration/libvelox_functions_prestosql.a  velox/functions/prestosql/libvelox_functions_prestosql_impl.a  velox/common/hyperloglog/libvelox_common_hyperloglog.a  velox/functions/lib/libvelox_functions_lib.a  velox/external/md5/libmd5.a  velox/functions/prestosql/types/libvelox_presto_types.a  velox/functions/prestosql/json/libvelox_functions_json.a  velox/functions/lib/libvelox_functions_lib_date_time_formatter.a  _deps/libstemmer/src/libstemmer/libstemmer.a  velox/functions/lib/libvelox_is_null_functions.a  velox/expression/libvelox_expression.a  velox/core/libvelox_core.a  velox/connectors/libvelox_connector.a  velox/vector/arrow/libvelox_arrow_bridge.a  velox/common/caching/libvelox_caching.a  velox/core/libvelox_config.a  velox/expression/libvelox_expression_functions.a  velox/expression/type_calculation/libvelox_type_calculation.a  velox/expression/signature_parser/libvelox_signature_parser.a  velox/functions/lib/libvelox_functions_util.a  /usr/lib/x86_64-linux-gnu/libdouble-conversion.so.3.1  _deps/simdjson-build/libsimdjson.a  velox/dwio/common/exception/libvelox_dwio_common_exception.a  velox/vector/tests/utils/libvelox_vector_test_lib.a  velox/vector/libvelox_vector.a  velox/buffer/libvelox_buffer.a  velox/common/memory/libvelox_memory.a  velox/common/time/libvelox_time.a  velox/type/libvelox_type.a  velox/common/encode/libvelox_encode.a  velox/type/tz/libvelox_type_tz.a  velox/common/serialization/libvelox_serialization.a  velox/external/date/libvelox_external_date.a  velox/common/base/libvelox_status.a  _deps/re2-build/libre2.a  velox/common/base/libvelox_exception.a  velox/common/file/libvelox_file.a  velox/common/base/libvelox_common_base.a  velox/common/testutil/libvelox_test_util.a  velox/common/compression/libvelox_common_compression.a  velox/common/process/libvelox_process.a  velox/common/base/libvelox_exception.a  velox/common/file/libvelox_file.a  velox/common/base/libvelox_common_base.a  velox/common/testutil/libvelox_test_util.a  velox/common/compression/libvelox_common_compression.a  velox/common/process/libvelox_process.a  _deps/folly-build/libfolly.a  _deps/boost-build/libs/random/libboost_random.a  _deps/boost-build/libs/context/libboost_context.a  _deps/boost-build/libs/filesystem/libboost_filesystem.a  _deps/boost-build/libs/program_options/libboost_program_options.a  _deps/boost-build/libs/thread/libboost_thread.a  _deps/boost-build/libs/atomic/libboost_atomic.a  _deps/boost-build/libs/date_time/libboost_date_time.a  _deps/boost-build/libs/chrono/libboost_chrono.a  _deps/boost-build/libs/container/libboost_container.a  _deps/boost-build/libs/exception/libboost_exception.a  /usr/lib/x86_64-linux-gnu/libdouble-conversion.so  _deps/gflags-build/libgflags_debug.a  _deps/glog-build/libglogd.a  /usr/lib/x86_64-linux-gnu/libevent.so  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libssl.so  /usr/lib/x86_64-linux-gnu/libcrypto.so  /usr/lib/x86_64-linux-gnu/libbz2.so  /usr/lib/x86_64-linux-gnu/liblz4.so  /usr/lib/x86_64-linux-gnu/libzstd.so  /usr/lib/x86_64-linux-gnu/libsnappy.so  /usr/lib/x86_64-linux-gnu/libdwarf.so  /usr/lib/x86_64-linux-gnu/libsodium.so  -ldl  /usr/lib/x86_64-linux-gnu/libunwind.so  _deps/fmt-build/libfmtd.a  _deps/glog-build/libglogd.a  lib/libgtest_main.a  lib/libgtest.a  _deps/gflags-build/libgflags_nothreads_debug.a  _deps/folly-build/folly/libfollybenchmark.a  -lstdc++fs && :
/usr/bin/ld: _deps/folly-build/folly/libfollybenchmark.a(Benchmark.cpp.o): in function `folly::detail::BenchmarkingStateBase::doSetUpPerfScoped(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const':
/root/velox/_build/debug/_deps/folly-src/folly/Benchmark.cpp:858: undefined reference to `folly::detail::PerfScoped::PerfScoped(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
/usr/bin/ld: _deps/folly-build/folly/libfollybenchmark.a(Benchmark.cpp.o): in function `folly::detail::BenchmarkingStateBase::setUpPerfScoped() const':
/root/velox/_build/debug/_deps/folly-src/folly/Benchmark.cpp:867: undefined reference to `folly::detail::PerfScoped::PerfScoped()'
/usr/bin/ld: _deps/folly-build/folly/libfollybenchmark.a(Benchmark.cpp.o): in function `std::pair<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<folly::detail::BenchmarkResult, std::allocator<folly::detail::BenchmarkResult> > > folly::detail::BenchmarkingStateBase::runBenchmarksWithPrinter<folly::(anonymous namespace)::BenchmarkResultsPrinter>(folly::(anonymous namespace)::BenchmarkResultsPrinter*) const':
/root/velox/_build/debug/_deps/folly-src/folly/Benchmark.cpp:881: undefined reference to `folly::detail::PerfScoped::~PerfScoped()'
/usr/bin/ld: /root/velox/_build/debug/_deps/folly-src/folly/Benchmark.cpp:881: undefined reference to `folly::detail::PerfScoped::~PerfScoped()'
collect2: error: ld returned 1 exit status
[170/226] Linking CXX executable velox/dwio/parquet/tests/reader/velox_parquet_e2e_filter_test
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:103: build] Error 1
make[1]: Leaving directory '/root/velox'
make: *** [Makefile:107: debug] Error 2

[bdice]

I can reproduce this issue as well.

[majetideepak]

I looked into this and the issue seems to be that the bundled folly install is not installing the libfolly.a library in the correct location.
_build/debug/_deps/folly-build/libfolly.a
_build/debug/_deps/folly-build/folly/libfollybenchmark.a

CC: @assignUser

[majetideepak]

The lib folly location is fine. There is no issue with other benchmarks.
The issue is definitely with the link order. libfolly.a should follow libfollybenchmark.a but it isn't. I will look into it more.

[assignUser]

link order.

That is influenced by the order of thing ins target_link_libraries so maybe folly::folly needs to be moved back.

[majetideepak]

so maybe folly::folly needs to be moved back

This was my first workaround but did not seem to work. Due to some transitive dependency, libfolly.a is not even immediately before the libfollybenchmark.a

... libfolly.a ... linux-gnu/libzstd.so /usr/lib/x86_64-linux-gnu/libsnappy.so /usr/lib/x86_64-linux-gnu/libdwarf.so /usr/lib/x86_64-linux-gnu/libsodium.so -ldl /usr/lib/x86_64-linux-gnu/libunwind.so _deps/fmt-build/libfmtd.a _deps/glog-build/libglogd.a lib/libgtest_main.a lib/libgtest.a _deps/gflags-build/libgflags_nothreads_debug.a _deps/folly-build/folly/libfollybenchmark.a -lstdc++fs