spark agent #272 fix duplicated attribute ids #281
Conversation
- remove deprecated symbol :/
|
|
||
| val duplicates = operation.output.groupBy(_.exprId).collect { case (x, List(_,_,_*)) => x } | ||
| if (duplicates.nonEmpty) { | ||
| logError(s"Duplicated attributes found in Join operation output, ExprIds of duplicates: $duplicates") |
There was a problem hiding this comment.
We log it as an Error, but there is no recovery logic and we just carry on. Shouldn't we log it as Warning instead?
|
|
||
| override def build(): DataOperation = { | ||
|
|
||
| val duplicates = operation.output.groupBy(_.exprId).collect { case (x, List(_,_,_*)) => x } |
There was a problem hiding this comment.
The operation.output type is Seq, but the List(_,_,_*) would only work for Lists, not for other Seqs.
We should use a more generic type destructor Seq(_, _, _*). Or even better, since we are only interested in count but not the values itself, we could communicate this logic more clearly:
| val duplicates = operation.output.groupBy(_.exprId).collect { case (x, List(_,_,_*)) => x } | |
| val duplicates = operation.output.groupBy(_.exprId).collect { case (exprId, attrs) if attrs.length > 1 => exprId } |
|
Kudos, SonarCloud Quality Gate passed!
|
|
Amazing @cerveada :). Do you guys have plans to deploy a new version of the agent? |
|
I'm working on it at the moment. Will trigger the release shortly after finishing some testing. |
|
amazing :). thanks a lot @wajda |
/: