TEZ-4039: Tez should inject dag id, query id into MDC #98

abstractdog · 2021-02-01T05:32:47Z

No description provided.

prasanthj · 2021-02-01T07:19:01Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+   */
+  @ConfigurationScope(Scope.AM)
+  @ConfigurationProperty
+  public static final String TEZ_LOG_PATTERN_LAYOUT_AM = TEZ_PREFIX + "log.pattern.layout.am";


Use TEZ_AM_PREFIX instead?

prasanthj · 2021-02-01T07:19:33Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+   */
+  @ConfigurationScope(Scope.AM)
+  @ConfigurationProperty
+  public static final String TEZ_LOG_PATTERN_LAYOUT_TASK = TEZ_PREFIX + "log.pattern.layout.task";


Use TEZ_TASK_PREFIX instead?

prasanthj · 2021-02-01T07:20:16Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+   * By this option, user can easily override the logging pattern which is applied in
+   * TezContainerLogAppender in tasks, regardless of the environmental settings.
+   */
+  @ConfigurationScope(Scope.AM)


Scope should be Vertex since this task specific config

prasanthj · 2021-02-01T07:21:49Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+  public static final String TEZ_MDC_CUSTOM_KEYS_DEFAULT = "";
+
+  /**
+   * Comma separated list of keys, which can used for defining keys in Configuration. Tez will try


nit: list of values?

no, these are still keys, but keys in Configuration

tez.mdc.custom.keys=queryId tez.mdc.custom.keys.values.from=hive.query.id <- here we are

I'm trying to clarify this in javadoc

prasanthj · 2021-02-01T07:22:25Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+  public static final String TEZ_MDC_CUSTOM_KEYS_DEFAULT = "";
+
+  /**
+   * Comma separated list of keys, which can used for defining keys in Configuration. Tez will try


nit: list of values?

(same comment as above)

prasanthj · 2021-02-01T07:29:56Z

tez-common/src/main/java/org/apache/tez/util/LoggingUtils.java

+
+  /**
+   * NonClonableHashtable is a special class for hacking the log4j MDC context. By design, log4j's
+   * MDC uses a ThreadLocalMap, which clones parent thread's context before propagating it to child


Isn't this configurable via isThreadContextMapInheritable system property?

unfortunately, isThreadContextMapInheritable is slightly different from what I needed:

Set the system property `log4j2.isThreadContextMapInheritable` to `true` to enable child threads to inherit the Thread Context Map.

this is used for inheritance, which is needed, child thread inherits parent's context, this is fine, the problem is that it inherits and clones the map, which is against my implementation...
my implementation makes it possible to define a global MDC context in DAGAppMaster and TezChild in the main thread, and all threads inherit that, but when a new dag (DAGAppMaster) or task attempt (TezChild) comes, modifying the global thread's context should have an effect on all threads' context, but with cloning, it won't have:
https://github.com/apache/log4j/blob/trunk/src/main/java/org/apache/log4j/helpers/ThreadLocalMap.java#L34-L41

instead, I choose to define 1 context in the main thread and propagating it to all child threads (which is automatic due to ThreadLocalMap behavior by default), but I only need to init the logging context once for every dag/taskattempt. Without cloning, a single change will change the MDC contents of all threads in the JVM.

abstractdog · 2021-02-06T14:54:01Z

thanks for the review @prasanthj, I tried to address everything you mentioned

Change-Id: I4ba3a5ff69ab680f7f65b4dfd438e1ab82d182bc

rbalamohan · 2022-10-24T11:39:11Z

tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

+   */
+  @ConfigurationScope(Scope.VERTEX)
+  @ConfigurationProperty
+  public static final String TEZ_LOG_PATTERN_LAYOUT_TASK = TEZ_TASK_PREFIX + "log.pattern.layout";


Is there any default value for this? (e.g If the user wants to reset it back to old value)

good catch, user should be able to turn this off by setting it to an empty string, I'm reflecting this in the configuration

rbalamohan · 2022-10-24T11:39:54Z

tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java

+    patternConfig.set(TezConfiguration.TEZ_TASK_LOG_LEVEL, "debug");
+    patternConfig.set(TezConfiguration.TEZ_LOG_PATTERN_LAYOUT_AM,
+        "%d{ISO8601} [%p] [%t (queryId=%X{queryId} dag=%X{dagId})] |%c{2}|: %m%n");
+    patternConfig.set(TezConfiguration.TEZ_LOG_PATTERN_LAYOUT_TASK,


Can you check for the case when the value can be reset back to original value? Plz refer to my comment earlier in TezConfiguration.

yeah, added testcase

rbalamohan · 2022-10-24T11:42:01Z

tez-common/src/main/java/org/apache/tez/util/LoggingUtils.java

+    mdcContext.set(new NonClonableHashtable<String, String>());
+
+    try {
+      final Constructor<?>[] constructors = org.apache.log4j.MDC.class.getDeclaredConstructors();


log4j or slf4j?

log4j, intentionally, the whole hack implemented in this method is log4j specific

rbalamohan · 2022-10-24T11:43:12Z

tez-common/src/main/java/org/apache/tez/util/LoggingUtils.java

+      // don't want to fail on incorrect mdc key settings, but warn in app logs
+      if (mdcKey.isEmpty() || mdcKeysValuesFrom.length < i + 1) {
+        LOG.warn("cannot set mdc key: {}", mdcKey);
+        break;


Did you mean break or continue here?

if we hit this it means that mdcKey is most probably empty, so we had:

tez.mdc.custom.keys=

as it's very unlikely that the user set something like this:

tez.mdc.custom.keys=,something_which_matters

if we split string by commas and find empty string, it makes no sense to loop further, or at least I cannot figure out a valid use-case

rbalamohan

Left minor comments.

abstractdog · 2022-10-24T12:29:37Z

Left minor comments.

thanks a lot, fixed in https://github.com/apache/tez/pull/98/commits/01adba3cfca7bbd27d6bbacbbd9bb3041d03ab3cplease let me know if you're fine with them (and my comments)

tez-yetus · 2022-10-25T10:01:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	36m 3s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	5m 51s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 13s	master passed
+1 💚	compile	3m 51s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	3m 50s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 37s	master passed
+1 💚	javadoc	3m 51s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 36s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	6m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 46s	the patch passed
+1 💚	compile	1m 54s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 54s	the patch passed
+1 💚	compile	1m 41s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 41s	the patch passed
+1 💚	checkstyle	0m 14s	The patch passed checkstyle in tez-api
+1 💚	checkstyle	0m 13s	The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 14s	The patch passed checkstyle in tez-runtime-internals
+1 💚	checkstyle	0m 22s	tez-dag: The patch generated 0 new + 51 unchanged - 1 fixed = 51 total (was 52)
+1 💚	checkstyle	0m 15s	The patch passed checkstyle in tez-tests
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	1m 39s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 35s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	findbugs	4m 12s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 19s	tez-api in the patch passed.
+1 💚	unit	0m 30s	tez-common in the patch passed.
+1 💚	unit	0m 39s	tez-runtime-internals in the patch passed.
+1 💚	unit	5m 20s	tez-dag in the patch passed.
+1 💚	unit	39m 35s	tez-tests in the patch passed.
+1 💚	asflicense	1m 18s	The patch does not generate ASF License warnings.
		142m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-98/10/artifact/out/Dockerfile
GITHUB PR	#98
JIRA Issue	TEZ-4039
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 7f925c3d5f80 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `fc5b6e9`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-98/10/testReport/
Max. process+thread count	1383 (vs. ulimit of 5500)
modules	C: tez-api tez-common tez-runtime-internals tez-dag tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-98/10/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

rbalamohan · 2022-10-25T15:31:08Z

LGTM. +1

prasanthj reviewed Feb 1, 2021

View reviewed changes

abstractdog added 2 commits October 21, 2022 10:01

TEZ-4039: Tez should inject dag id, query id into MDC

23d1287

Change-Id: I4ba3a5ff69ab680f7f65b4dfd438e1ab82d182bc

addressing PR comments

717d739

abstractdog force-pushed the TEZ-4039 branch from 03ee238 to 717d739 Compare October 21, 2022 08:04

This comment was marked as outdated.

Sign in to view

post-rebase fixes

25134f3

abstractdog force-pushed the TEZ-4039 branch from 42e6bd1 to 25134f3 Compare October 21, 2022 11:01

This comment was marked as outdated.

Sign in to view

checkstyle

bc6e39d

This comment was marked as outdated.

Sign in to view

values.from -> conf.props for better understanding

1200ae8

This comment was marked as outdated.

Sign in to view

rbalamohan reviewed Oct 24, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

addressing PR comments

01adba3

abstractdog force-pushed the TEZ-4039 branch from 9dec3b6 to 01adba3 Compare October 25, 2022 07:38

abstractdog merged commit dbbd406 into apache:master Oct 26, 2022

This was referenced Nov 27, 2023

Improve test reliability by resolving nondeterministic of HashMap #316

Open

TEZ-4521: Partition stats should be always uncompressed size #317

Merged

tez-yetus mentioned this pull request Dec 4, 2023

TEZ-4522: Use OpenFile where FileStatus is available. #318

Merged

tez-yetus mentioned this pull request Dec 11, 2023

TEZ-3268: Provide a demuxer sample app that uses fair routing #320

Open

tez-yetus mentioned this pull request Jan 8, 2024

TEZ-4039: Addendum: Tez should inject dag id, query id into MDC. #325

Merged

This was referenced Jan 29, 2024

TEZ-4538 : Upgrade netty to 4.1.100 due to CVE-2023-44487 #333

Merged

TEZ-4540: Reading proto data more than 2GB from multiple splits fails #334

Merged

tez-yetus mentioned this pull request Feb 6, 2024

TEZ-4529: Apache Tez Release 0.10.3: update SNAPSHOT versions, DOAP file #335

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEZ-4039: Tez should inject dag id, query id into MDC #98

TEZ-4039: Tez should inject dag id, query id into MDC #98

abstractdog commented Feb 1, 2021

prasanthj Feb 1, 2021

prasanthj Feb 1, 2021

prasanthj Feb 1, 2021

prasanthj Feb 1, 2021

abstractdog Feb 6, 2021

prasanthj Feb 1, 2021

abstractdog Feb 6, 2021

prasanthj Feb 1, 2021

abstractdog Feb 6, 2021

abstractdog commented Feb 6, 2021

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

rbalamohan Oct 24, 2022

abstractdog Oct 24, 2022

rbalamohan Oct 24, 2022

abstractdog Oct 25, 2022

rbalamohan Oct 24, 2022

abstractdog Oct 24, 2022

rbalamohan Oct 24, 2022

abstractdog Oct 24, 2022

rbalamohan left a comment

abstractdog commented Oct 24, 2022 •

edited

Loading

This comment was marked as outdated.

tez-yetus commented Oct 25, 2022

rbalamohan commented Oct 25, 2022

TEZ-4039: Tez should inject dag id, query id into MDC #98

TEZ-4039: Tez should inject dag id, query id into MDC #98

Conversation

abstractdog commented Feb 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abstractdog commented Feb 6, 2021

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbalamohan left a comment

Choose a reason for hiding this comment

abstractdog commented Oct 24, 2022 • edited Loading

This comment was marked as outdated.

tez-yetus commented Oct 25, 2022

rbalamohan commented Oct 25, 2022

abstractdog commented Oct 24, 2022 •

edited

Loading