ARROW-7863: [C++][Python][CI] Ensure running HDFS related tests #6432
Conversation
libhdfs3 backend may be broken. It causes SEGV. So libhdfs3 backend test is disabled for now.
|
@ursabot crossbow submit test-conda-python-3.7-hdfs-2.9.2 |
|
AMD64 Conda Crossbow Submit (#90851) builder has been succeeded. Revision: b54664c Submitted crossbow builds: ursa-labs/crossbow @ ursabot-509
|
.env
Outdated
| @@ -35,6 +35,7 @@ PANDAS=latest | |||
| DASK=latest | |||
| TURBODBC=latest | |||
| HDFS=2.9.2 | |||
| LIBHDFS3=2.3 | |||
There was a problem hiding this comment.
libhdfs3 2.3 is not supported at all. Version 2.2.31 is the last one we support but I'm not sure if that still works with the current set of conda-forge dependencies / compilers. We might propose reverting the feedstock to the last "good" version that uses the libhdfs C ABI
conda-forge/libhdfs3-feedstock@97fd932#diff-ae18fe4e01e61a9862ceaf165003213a
There was a problem hiding this comment.
I can build that for you when you provide me with a source tarball at a semi-official location. The original one seems to be vanished.
|
Also to be honest I would be OK with removing support for libhdfs3 altogether since it is abandonware |
| function use_libhdfs_dir() { | ||
| unset HADOOP_HOME | ||
| export ARROW_LIBHDFS_DIR=$libhdfs_dir | ||
| } |
There was a problem hiding this comment.
@kou Could you explain the difference between the two? (use_hadoop_home vs. use_libhdfs_dir)
Are there two sources for libhdfs?
There was a problem hiding this comment.
We need to find ${HADOOP_HOME}/lib/native/libhdfs.so when user uses HADOOP_HOME.
We need to find ${ARROW_LIBHDFS_DIR}/libhdfs.so when user uses ARROW_LIBHDFS_DIR.
If HADOOP_HOME is defined, ARROW_LIBHDFS_DIR isn't used. So we need to test them separately.
|
OK. I'll remove libhdfs3 support. |
|
We used to test libhdfs3 support until conda forge provided the right version, but in the last iteration I've removed it. |
|
@ursabot crossbow submit test-conda-python-3.7-hdfs-2.9.2 |
|
AMD64 Conda Crossbow Submit (#91370) builder has been succeeded. Revision: 9620d03 Submitted crossbow builds: ursa-labs/crossbow @ ursabot-515
|
It's no longer maintained.
kou
force-pushed
the
cpp-hdfs-test-robust
branch
from
9620d03 to
b311877
Compare
|
@ursabot crossbow submit test-conda-python-3.7-hdfs-2.9.2 |
|
AMD64 Conda Crossbow Submit (#91379) builder has been succeeded. Revision: b311877 Submitted crossbow builds: ursa-labs/crossbow @ ursabot-516
|
|
@ursabot crossbow submit test-conda-python-3.7-hdfs-2.9.2 |
|
AMD64 Conda Crossbow Submit (#91421) builder has been succeeded. Revision: 85b45ad Submitted crossbow builds: ursa-labs/crossbow @ ursabot-518
|
|
Done. |
|
+1 |
libhdfs3 backend may be broken. It causes SEGV. So libhdfs3 backend
test is disabled for now.