v2.4.0

This release contains numerous contributed bugfixes, as well a few changes in functionality:

• The default checksum for written blocks has been changed from CRC32 to CRC32C, to match the behavior of the java client. (thanks @symious)
• The hdfs binary now supports test (#292) and ls -R (#289), per the posix spec. (thanks @gardenia)

v2.3.0

This release, along with several bugfixes, contains new functionality:

• Client.Truncate (#73, 2f11406) (thanks @junjieqian!)
• Client.ServerDefaults (039ab59)

It contains one potentially dangerous behaviour change, in b02ab58: FileWriter.Close now will correctly propagate errors in the situation where the namenode has not yet received all acks from the datanodes. Close returns a specific error in that case, ErrReplicating, which the client can either ignore or use in a retry loop. See also IsErrReplicating, for checking this case.

v2.2.1

This release contains several contributed bugfixes, notably the longstanding HDP issues described in #260.

v2.2.0

This overdue release contains a spate of bugfixes, as well as two major features contributed by others:

• #236: Data transfer encryption support, for fully kerberized clusters.
• #238: Extended attributes ("XAttr") support

v2.1.1

This is a small bugfix release, fixing a case where the lease renewal code didn't run, and another bug where a goroutine was possibly leaked on a call to Close. Both fixes are in 1cf907d.

v2.1.0

This is a small release. In addition to some bug fixes, it includes a few useful features:

• #153, #154, #208, 574b0ba: Automatic renewal of file leases and heartbeats for write streams. Both features are necessary for writes to open filesover a long period.
• #144, #170, 1c841f7: Support for SASL-wrapped RPC communication with the namenode.
• #205: Support for snapshots.
• 6f7e441: A new method, RemoveAll, and a fix such that Remove is not recursive. Please be careful to check your usage of said function, as this is a fairly major change to behavior.

It also officially includes support for CDH6, although it probably worked before.

v2.0.0

This is a major release, including multiple breaking interface changes and new features.

The library is now structured as a go module. To use it, use the import path github.com/colinmarc/hdfs/v2.

Kerberos support

Added in #133, with lots of help from @Shastick and @staticmukesh. This adds basic kerberos authentication using to the library and the command line client - to the latter with support for ccaches. This much-requested feature should be ready for production use, but I would love your feedback and/or bug reports.

Timeouts and other useful configuration

Client now has two new options, NamenodeDialFunc and DatanodeDialFunc, which can be used to replace net.Dial and set timeouts and keepalives and other useful things. You can also use SetDeadline on FileReader and FileWriter to enforce i/o timeouts. See #139 for more information.

Your hdfs.Client can be made to respect your Hadoop configuration with the hdfs.ClientOptionsFromConf method. This looks for relevant options from the configuration and tries to configure the client to match. While this doesn't do that much right now, it may be expanded to other things in the future.

Incompatable interface changes

The rpc package is now internal; keeping the interface backwards compatible was too difficult, and nothing in that subpackage was really useful externally anyway. rpc.NamenodeError which was possibly the only useful export, now implements an interface, hdfs.Error.

The configuration parsing/loading code and the HadoopConf type have been moved into their own package, hadoopconf. The interfaces are also slightly tweaked.

Finally, all deprecated methods, such as hdfs.NewForUser, have been removed.

v1.1.3

This is a hotfix release, fixing an obvious bug introduced in the last release in 5160e9f.

v1.1.2

This is a minor bugfix release, but contains one possibly breaking change.

As of #123 (change proposed by @hollow), the client will now use hostnames to connect to datanodes (when available), rather than IP addresses. If you experience problems with that, you may have an issue with your DNS configuration.

v1.1.1

This (long-overdue) release contains a host of minor bug fixes, as well as a few new features:

• #98 - hdfs put - now reads from stdin. (submitted by @Shastick)
• #101 - client has a new method, Walk, like filepath.Walk. (submitted by @Shastick)
• #107 (fixed in 84dbd09) - FileWriter now exposes Flush, for syncing data to disk.

The other notable interface change is the deprecation of NewForUser and NewForConnection in 0f30457.