Add InvalidSourceTap to catch all cases for no good path.#1458
Add InvalidSourceTap to catch all cases for no good path.#1458
Conversation
|
We have an internal version of this Tap with the same name. It handles some of the quirks required to work with cascading's MultiInputFormat and also unique identifiers. If you could consolidate the two, that'd be great. |
There was a problem hiding this comment.
This can extend just SourceTap.
|
@rubanm Didn't know we already have an internal version. Will take a look and consolidate them. |
|
Do we have a test yet that hits this code path in |
|
@benpence Added a unit-test. Thanks! |
There was a problem hiding this comment.
goodHdfsPaths will return Nil in this case right? We need to pass hdfsPaths here I think so that openForRead throws with the right message. Can you also update the comment above.
|
LGTM. @johnynek could you take a look as well? Thanks. |
|
Gentle ping. |
|
LGTM merging this. @reconditesea thanks! The internal version of this at Twitter has been working fine so I don't foresee any problems. Also with this change, opening an Iterator via toIterator on an invalid source will throw InvalidSourceException (instead of returning an iterator on an invalid hdfs datapath, which in turn, throws hadoop's InvalidInputException). |
Add InvalidSourceTap to catch all cases for no good path.
3 participants
To close #1454.