hdfs: should not require hostname

[efiop]

using hdfs://path/to/something is pretty normal when you are trying to access your local hdfs, but dvc thinks that path is a hostname and fails to connect. We could think of a way to separate those or maybe just document it somewhere (and or provide a nicer hint). First idea would be to make users use localhost or 127.0.0.1 as a hostname in those cases, but that won't actually work. What would work is using default, as that is the default value in pyarrow.connect https://arrow.apache.org/docs/python/generated/pyarrow.hdfs.connect.html and I was able to verify that hdfs://default/path/to/something works great.

[anderl80]

Thank you for helping figuring this out!

[grco-xp]

have you tried using get-url from an hdfs path such as hdfs://default/path/to/something?
i am working with dvc==1.6.6 and having the following exception:

dvc.tree.base.RemoteCmdError: hdfs command 'hadoop fs -checksum hdfs://default/path/to/something' finished with non-zero return code 1': b'checksum: Invalid host name: local host is: (unknown); destination host is: "default":8020; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost\n'

the HDFSTree get_file_hash method tries to checksum the full url hdfs://default/path/to/something, instead of the hdfs:///path/to/something path.

[efiop]

Hi @grco-xp ! Are you able to use that url with hadoop CLI?