put can read from stdin by specifying the '-' source file #98
Conversation
No tests written in go for now, though :/
|
Hi @Shastick, Thanks for the PR! This looks like a reasonable approach, but can you add some tests and fix the build, to start? Also, please run |
|
(If you rebase, the protoc problem in travis should be fixed) |
|
Thanks for the feedback. Will do this asap :) |
(this should fix travis issues, hopefully)
For some reason, make on travis recently started deciding that the *.pb.go files were older than the *.proto files, ever though they originate from the same `git checkout`. The easiest way to fix this is simply to touch them right before we run make, so that they look newer.
|
Hoping the test is enough: I did not test scenarios where it should fail (existing file/directory with the same name) |
… existing file or directory)
Shastick
force-pushed
the
put-stdin
branch
3 times, most recently
from
6c1a9f1
to
50c093e
Compare
|
Added test cases for expected failures |
cmd/hdfs/put.go
Outdated
| if existing.IsDir() { | ||
| fatal(fmt.Errorf("can't write from STDIN into a directory: %s", dest)) | ||
| } else { | ||
| fatal(fmt.Errorf("won't write to an existing file: %s", dest)) |
There was a problem hiding this comment.
Both branches here can be just fatal(&os.PathError{"put", dest, os.ErrExist})
cmd/hdfs/put.go
Outdated
| } | ||
| } | ||
|
|
||
| func putFromStdIn(client *hdfs.Client, dest string) { |
There was a problem hiding this comment.
Nitpick: this should be putFromStdin (without the I capitalized), since the standard library spells it that way.
| assert_equal $SHA `shasum < $BATS_TMPDIR/mobydick_stdin_test.txt | awk '{ print $1 }'` | ||
| } | ||
|
|
||
| @test "put from stdin into existing file" { |
There was a problem hiding this comment.
How does the normal hadoop -fs behave in this case, out of curiosity?
There was a problem hiding this comment.
Just had a try:
put: `/tmp/testme': File exists
cmd/hdfs/put.go
Outdated
| } | ||
|
|
||
| mode := 0755 | os.ModeDir | ||
| // make sure parent dir exists |
There was a problem hiding this comment.
This and the next few comments are unnecessary. Also, for future reference, comments should be in complete sentences.
|
The failure seems unrelated (only happened on one of the 4 runs) for a rm test: update: meh, did not see that these were marked "allowed failures". |
|
Merged in 47855e4. Thanks so much for the contributions! |
* implement client options, support multiple namenodes with failover * Fix an alignment bug causing us to send very small packets The faulty math in block_write_stream was causing us to always send 512 byte packets. * rename host to namenodeHost * document standby retry behaviour * include detail in namenode error * deprecate WrapNamenodeConnection * replace goto behaviour with for loop * fallback to error from last failure * travis: bump go version to 1.x * Regenerate proto files (this should fix travis issues, hopefully) * travis: touch *.pb.go before running make For some reason, make on travis recently started deciding that the *.pb.go files were older than the *.proto files, ever though they originate from the same `git checkout`. The easiest way to fix this is simply to touch them right before we run make, so that they look newer. * travis: attempt to fix hadoop tarball caching * Fix comment for CopyToRemote Fixes colinmarc#89, colinmarc#90 * Deprecate NewForUser and NewForConnection * Add the ability for 'put -' to read from stdin Fixes colinmarc#44, closes colinmarc#98 * Convert from govendor to dep * Update dependencies * Tweak put tests Add a very basic test, and put everything in the same directory so it gets cleaned up properly. * Improve conf loading and surrounding tests - Fix a bug where configuration wasn't loaded from specified paths - Only search one path at a time, rather than deduping from all - Correctly handle fs.default.name - Improve test coverage and documentation * Fix the merge from upstream * more fixes to merge master * more fixes
Here is a small proposal for
puting viaSTDIN, as requested in #44.I'm relatively new to GO (and the github workflow as well...) and more than happy to read any comment!