TEZ-4503: Warn about large conf properties in payload #308
+162
−19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
|
💔 -1 overall
This message was automatically generated. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@abstractdog Can you please have a look in this PR when you get the chance. Thanks! |
abstractdog
reviewed
Oct 17, 2023
| public static final String TEZ_LOGGING_PROPERTY_SIZE_THRESHOLD = | ||
| TEZ_PREFIX + "logging.property.size.threshold"; | ||
| public static final int TEZ_LOGGING_PROPERTY_SIZE_THRESHOLD_DEFAULT = 512 * 1024; | ||
| /** |
There was a problem hiding this comment.
nit, we tend to add a line between every config properties, even if they are related to each other
abstractdog
reviewed
Oct 17, 2023
| private static final boolean PROPERTY_MASK; | ||
|
|
||
| static { | ||
| TezConfiguration c = new TezConfiguration(); |
There was a problem hiding this comment.
I'm afraid this approach will become confusing hard to handle over time
a new TezConfiguration() in a static initializer will always pick up whatever is present in tez-site.xml, hence we don't really have control over that: I think if we want to be able to use the size threshold, we need to achieve that from the actual config, which might need a bit of refactoring as this method receives Configuration as an iterable (and we need to see the threshold and mask property beforehand without iterating over the values)
There was a problem hiding this comment.
I reworked the PR (https://github.com/apache/tez/pull/308/commits/17e07e7c714d670cfbd5e15eb3c56f33a022f2ec,https://github.com/apache/tez/pull/308/commits/687bd3b22f26288c0f49d40d05588d4f17a7b2fa) to allow using the size threshold from within the configuration object.
Instead of using the central Tez configuration (tez-site.xml) for selecting the threshold and masking for logging large properties check for the respective entries in each individual configuration object.
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The changes were tested by running
mvn test -Dtest=TestTezUtilsand manually inspecting the output under:tez-common/target/surefire-reports/org.apache.tez.common.TestTezUtils-output.txt.