HADOOP-17461. IOStatisticsContext + committer integration #4566

steveloughran · 2022-07-14T19:48:59Z

This is PR #4352 with a new patch to integrate with the MR committers, at least
as far as collecting read stats from job commits and including in _SUCCESS
the committer ITests collect these and print them.

move reference map and lookup to a IOStatisticsContextIntegration class
static method in IOStatisticsContext to relay lookup
add method to switch a thread's context; needed to aggregate worker thread
IO in threads doing work for committers without the need to explicitly
collect and pass back the stats
production code moves to the new methods
tests move to this and away from looking up the fields in the streams
stats are reset in s3a test setup
s3a committers collect data read stats during job commit and include
in summary statistics. This is only the stats when reading manifest
files, not the actual work.
tests to print the aggregate of all loaded success files in the run.

How was this patch tested?

new/modified tests

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

steveloughran · 2022-07-14T19:49:23Z

you are going to love this. Full end to end stats collection of IO during job commit of terasort.


2022-07-14 20:37:38,332 [JUnit] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:dumpFileSystemIOStatistics(137)) - Aggregate FileSystem Statistics counters=((action_executor_acquired=2)
(action_file_opened=26)
(action_http_get_request=26)
(action_http_head_request=112)
(audit_request_execution=250)
(audit_span_creation=144)
(directories_created=14)
(directories_deleted=14)
(fake_directories_deleted=4)
(files_created=2)
(files_deleted=16)
(ignored_errors=8)
(object_bulk_delete_request=4)
(object_delete_objects=34)
(object_delete_request=14)
(object_list_request=74)
(object_metadata_request=112)
(object_put_request=16)
(object_put_request_completed=16)
(op_create=2)
(op_delete=16)
(op_exists=2)
(op_exists.failures=2)
(op_get_file_status=40)
(op_glob_status=4)
(op_list_files=6)
(op_list_located_status=4)
(op_list_status=4)
(op_list_status.failures=2)
(op_mkdirs=14)
(op_open=30)
(store_io_request=251)
(stream_read_bytes=232084)
(stream_read_close_operations=26)
(stream_read_closed=26)
(stream_read_fully_operations=10)
(stream_read_opened=26)
(stream_read_operations=41)
(stream_read_operations_incomplete=23)
(stream_read_remote_stream_drain=26)
(stream_read_seek_policy_changed=26)
(stream_read_total_bytes=232084)
(stream_write_block_uploads=2));

gauges=((stream_write_block_uploads_pending=2));

minimums=((action_executor_acquired.min=0)
(action_file_opened.min=49)
(action_http_get_request.min=63)
(action_http_head_request.min=40)
(object_bulk_delete_request.min=173)
(object_delete_request.min=51)
(object_list_request.min=48)
(object_put_request.min=82)
(op_create.min=62)
(op_delete.min=52)
(op_exists.failures.min=110)
(op_get_file_status.min=53)
(op_glob_status.min=102)
(op_list_files.min=176)
(op_list_status.failures.min=172)
(op_list_status.min=62)
(op_mkdirs.min=313)
(stream_read_remote_stream_drain.min=0));

maximums=((action_executor_acquired.max=0)
(action_file_opened.max=100)
(action_http_get_request.max=139)
(action_http_head_request.max=206)
(object_bulk_delete_request.max=454)
(object_delete_request.max=70)
(object_list_request.max=667)
(object_put_request.max=123)
(op_create.max=75)
(op_delete.max=685)
(op_exists.failures.max=120)
(op_get_file_status.max=178)
(op_glob_status.max=126)
(op_list_files.max=285)
(op_list_status.failures.max=184)
(op_list_status.max=68)
(op_mkdirs.max=847)
(stream_read_remote_stream_drain.max=1));

means=((action_executor_acquired.mean=(samples=2, sum=0, mean=0.0000))
(action_file_opened.mean=(samples=26, sum=1515, mean=58.2692))
(action_http_get_request.mean=(samples=26, sum=1997, mean=76.8077))
(action_http_head_request.mean=(samples=112, sum=7160, mean=63.9286))
(object_bulk_delete_request.mean=(samples=4, sum=997, mean=249.2500))
(object_delete_request.mean=(samples=14, sum=810, mean=57.8571))
(object_list_request.mean=(samples=74, sum=7647, mean=103.3378))
(object_put_request.mean=(samples=16, sum=1497, mean=93.5625))
(op_create.mean=(samples=2, sum=137, mean=68.5000))
(op_delete.mean=(samples=16, sum=1935, mean=120.9375))
(op_exists.failures.mean=(samples=2, sum=230, mean=115.0000))
(op_get_file_status.mean=(samples=40, sum=3283, mean=82.0750))
(op_glob_status.mean=(samples=4, sum=459, mean=114.7500))
(op_list_files.mean=(samples=6, sum=1280, mean=213.3333))
(op_list_status.failures.mean=(samples=2, sum=356, mean=178.0000))
(op_list_status.mean=(samples=2, sum=130, mean=65.0000))
(op_mkdirs.mean=(samples=14, sum=5303, mean=378.7857))
(stream_read_remote_stream_drain.mean=(samples=26, sum=3, mean=0.1154)));


Process finished with exit code 0

hadoop-yetus · 2022-07-14T23:52:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 7s		trunk passed
+1 💚	compile	23m 14s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 44s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 22s		trunk passed
+1 💚	mvnsite	3m 47s		trunk passed
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 43s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 6s		trunk passed
+1 💚	shadedclient	22m 31s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	23m 3s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 47s		the patch passed
+1 💚	compile	22m 24s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 24s		the patch passed
+1 💚	compile	20m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 15s	/results-checkstyle-root.txt	root: The patch generated 5 new + 75 unchanged - 0 fixed = 80 total (was 75)
+1 💚	mvnsite	3m 44s		the patch passed
+1 💚	javadoc	2m 52s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 15s		the patch passed
+1 💚	shadedclient	22m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 47s		hadoop-common in the patch passed.
+1 💚	unit	3m 23s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 37s		The patch does not generate ASF License warnings.
		241m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/1/artifact/out/Dockerfile
GITHUB PR	#4566
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8b0e37433c5b 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 1e9e6600d862bd7c33f627a670509bcf47cfeaa7
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/1/testReport/
Max. process+thread count	3158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-07-15T14:58:10Z

Here are the stats for a magic run where we also measure list calls.

The fact that hsync is being called shows that the terasort output is also being picked up. not intentional.

raises a big issue: should these committers reset the context before task/job commit.

 commit.AbstractCommitITest (AbstractCommitITest.java:printStatistics(118)) - Aggregate job statistics counters=((action_executor_acquired=7)
(committer_bytes_committed=800042)
(committer_commit_job=10)
(committer_commits_completed=14)
(committer_jobs_completed=10)
(committer_magic_marker_put=7)
(committer_materialize_file=14)
(object_list_request=17)
(object_multipart_initiated=7)
(op_hsync=6)
(stream_write_block_uploads=7)
(stream_write_bytes=400021)
(stream_write_total_data=400021));

gauges=();

minimums=((action_executor_acquired.min=0)
(committer_commit_job.min=301)
(committer_magic_marker_put.min=70)
(committer_materialize_file.min=129)
(object_list_request.min=64)
(object_multipart_initiated.min=74));

maximums=((action_executor_acquired.max=11)
(committer_commit_job.max=643)
(committer_magic_marker_put.max=127)
(committer_materialize_file.max=478)
(object_list_request.max=319)
(object_multipart_initiated.max=97));

means=((action_executor_acquired.mean=(samples=14, sum=65, mean=4.6429))
(committer_commit_job.mean=(samples=10, sum=4974, mean=497.4000))
(committer_magic_marker_put.mean=(samples=7, sum=641, mean=91.5714))
(committer_materialize_file.mean=(samples=14, sum=3171, mean=226.5000))
(object_list_request.mean=(samples=17, sum=2098, mean=123.4118))
(object_multipart_initiated.mean=(samples=7, sum=598, mean=85.4286)));

this is going to overreport, but i don't think the committers should be interfering with the context which should really be managed by the app

hadoop-yetus · 2022-07-15T18:54:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 8s		trunk passed
+1 💚	compile	23m 4s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 21s		trunk passed
+1 💚	mvnsite	3m 44s		trunk passed
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 56s		trunk passed
+1 💚	shadedclient	22m 13s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	22m 43s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 45s		the patch passed
+1 💚	compile	22m 17s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 17s		the patch passed
+1 💚	compile	20m 48s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 13s	/results-checkstyle-root.txt	root: The patch generated 5 new + 76 unchanged - 0 fixed = 81 total (was 76)
+1 💚	mvnsite	3m 43s		the patch passed
+1 💚	javadoc	2m 50s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 43s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 10s		the patch passed
+1 💚	shadedclient	22m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 53s		hadoop-common in the patch passed.
+1 💚	unit	3m 26s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 37s		The patch does not generate ASF License warnings.
		241m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/2/artifact/out/Dockerfile
GITHUB PR	#4566
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 9bb736781002 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / bf6ee6f5eb507d306691088906e8e5623becc1a5
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/2/testReport/
Max. process+thread count	2433 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

* move reference map and lookup to a IOStatisticsContextIntegration class * static method in IOStatisticsContext to relay lookup * add method to switch a thread's context; needed to aggregate worker thread IO in threads doing work for committers without the need to explicitly collect and pass back the stats * production code moves to the new methods * tests move to this and away from looking up the fields in the streams * stats are reset in s3a test setup * s3a committers collect data read stats during job commit and include in summary statistics. This is only the stats when reading manifest files, not the actual work. * tests to print the aggregate of all loaded success files in the run. Change-Id: I604990f2132b76d38e85ca8b777630225c32158e

* Cost of scan/load of magic files in task commit are collected * S3A list iterators update the context stats of the thread they were created in in close() calls. * With close() passthrough working and TaskPool invoking it if the iterator is closeable. Change-Id: If0a0c2de08d52a74b7c1f9498716d423b97b4003

mehakmeet · 2022-07-18T04:32:05Z

should these committers reset the context before task/job commit.

In spark we would be resetting before the execution begins I assume, so having it reset in the committers could mess things up? Was thinking about the case like a disctp job, how do we reset the stats in that case?

steveloughran · 2022-07-18T18:13:17Z

exactly. distcp, good thought. let's not worry about collecting results there

* Resetting and collection of context IO stats is optional; off by default * TaskPool wired up to switch to any builder-supplied context. This makes it easier to use and lines us up for manifest committer to also adopt. * Add an ID field to the interface, mainly for logging. This is separate from the thread ID Change-Id: I3b57dc915c34b1234b200ad00d79adc119ff6919

hadoop-yetus · 2022-07-21T14:35:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 46s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 29s		trunk passed
+1 💚	compile	23m 11s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 54s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 19s		trunk passed
+1 💚	mvnsite	3m 45s		trunk passed
+1 💚	javadoc	2m 57s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 27s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 0s		trunk passed
+1 💚	shadedclient	22m 8s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	22m 40s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 46s		the patch passed
+1 💚	compile	22m 23s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 23s		the patch passed
+1 💚	compile	20m 59s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 59s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	4m 14s		the patch passed
+1 💚	mvnsite	3m 40s		the patch passed
+1 💚	javadoc	2m 53s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 25s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 7s		the patch passed
+1 💚	shadedclient	22m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 45s		hadoop-common in the patch passed.
+1 💚	unit	3m 13s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 37s		The patch does not generate ASF License warnings.
		242m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/4/artifact/out/Dockerfile
GITHUB PR	#4566
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 16cb6c89e5fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 40ca262a7183799ae076cef877511d597a1da3c3
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/4/testReport/
Max. process+thread count	1544 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-21T14:57:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 15s		trunk passed
+1 💚	compile	25m 14s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 50s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 34s		trunk passed
+1 💚	mvnsite	3m 12s		trunk passed
+1 💚	javadoc	2m 24s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 3s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 39s		trunk passed
+1 💚	shadedclient	24m 28s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	24m 55s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 42s		the patch passed
+1 💚	compile	24m 19s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 19s		the patch passed
+1 💚	compile	21m 51s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 51s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	4m 26s		the patch passed
+1 💚	mvnsite	3m 10s		the patch passed
+1 💚	javadoc	2m 16s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 5s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 51s		the patch passed
+1 💚	shadedclient	25m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 2s		hadoop-common in the patch passed.
+1 💚	unit	3m 16s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 16s		The patch does not generate ASF License warnings.
		252m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/5/artifact/out/Dockerfile
GITHUB PR	#4566
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0af63939de89 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / ac6a3fa248e00111cbeb94c6b767b7c2b80641c7
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/5/testReport/
Max. process+thread count	2457 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

+setting thread context to null resets it +move merging of fs stats into finally block, after streamStatistics close has been called to update final stats Change-Id: I913b6a473da12918025e7ec11d4168bd135f0fc5

This is important for testing IOStatisticsContext functionality in unit tests as well as a source of actual data. +fixed the checkstyle Change-Id: I4f429a6a81729027026dc46bd1519f90a145c205

hadoop-yetus · 2022-07-25T23:02:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 41s		trunk passed
+1 💚	compile	26m 1s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	22m 20s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 32s		trunk passed
+1 💚	mvnsite	3m 18s		trunk passed
+1 💚	javadoc	2m 25s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 6s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 47s		trunk passed
+1 💚	shadedclient	25m 7s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	25m 37s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 43s		the patch passed
+1 💚	compile	25m 2s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	25m 2s		the patch passed
+1 💚	compile	22m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 29s		root: The patch generated 0 new + 97 unchanged - 1 fixed = 97 total (was 98)
+1 💚	mvnsite	3m 13s		the patch passed
+1 💚	javadoc	2m 17s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 5s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 54s		the patch passed
+1 💚	shadedclient	24m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 33s		hadoop-common in the patch passed.
+1 💚	unit	3m 2s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 20s		The patch does not generate ASF License warnings.
		254m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/6/artifact/out/Dockerfile
GITHUB PR	#4566
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 950ed0434155 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9dd221d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/6/testReport/
Max. process+thread count	1448 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mehakmeet and others added 8 commits June 23, 2022 11:37

HADOOP-17461. Thread-level IOStatistics in S3A

2655a61

HADOOP-17461. IOStatisticsContext in S3ABlockOutputStream

471379f

HADOOP-17461. steve's review comments

9f5631b

HADOOP-17461. changing the property key

6cb9a81

HADOOP-18242. checkstyle + merge conflicts

bcb4c25

HADOOP-17461. resetting stats + review comments

232d6b9

HADOOP-17461. review comments + test fixes

82bc205

HADOOP-17461. changes required for spark wiring.

c2dcad1

steveloughran marked this pull request as draft July 15, 2022 08:46

steveloughran added 2 commits July 18, 2022 09:25

Mehakmeet Singh added 2 commits July 19, 2022 10:36

HADOOP-17461. javadocs and checkstyles

f54ad18

HADOOP-17461. moving IOStatisticsContext in a different package

6f7e72e

steveloughran force-pushed the s3/HADOOP-17461-iostatisticsContext branch from bf6ee6f to ef9643c Compare July 20, 2022 14:39

apache deleted a comment from hadoop-yetus Jul 21, 2022

steveloughran force-pushed the s3/HADOOP-17461-iostatisticsContext branch 2 times, most recently from 40ca262 to ac6a3fa Compare July 21, 2022 10:43

steveloughran and others added 4 commits July 22, 2022 08:01

HADOOP-17461. TaskPool automatically propagates context

359206d

+setting thread context to null resets it +move merging of fs stats into finally block, after streamStatistics close has been called to update final stats Change-Id: I913b6a473da12918025e7ec11d4168bd135f0fc5

HADOOP-17461. adding stream capability and enabling iocontext for test

81c0480

HADOOP-17461. review comments

cdc9915

HADOOP-17461. RawLocalFileSystem streams are IOStatisticsContext aware

9dd221d

This is important for testing IOStatisticsContext functionality in unit tests as well as a source of actual data. +fixed the checkstyle Change-Id: I4f429a6a81729027026dc46bd1519f90a145c205

steveloughran force-pushed the s3/HADOOP-17461-iostatisticsContext branch from ac6a3fa to 9dd221d Compare July 25, 2022 18:46

steveloughran closed this Jul 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17461. IOStatisticsContext + committer integration #4566

HADOOP-17461. IOStatisticsContext + committer integration #4566

steveloughran commented Jul 14, 2022 •

edited

Loading

steveloughran commented Jul 14, 2022

hadoop-yetus commented Jul 14, 2022

steveloughran commented Jul 15, 2022

hadoop-yetus commented Jul 15, 2022

mehakmeet commented Jul 18, 2022

steveloughran commented Jul 18, 2022

hadoop-yetus commented Jul 21, 2022

hadoop-yetus commented Jul 21, 2022

hadoop-yetus commented Jul 25, 2022

HADOOP-17461. IOStatisticsContext + committer integration #4566

HADOOP-17461. IOStatisticsContext + committer integration #4566

Conversation

steveloughran commented Jul 14, 2022 • edited Loading

How was this patch tested?

For code changes:

steveloughran commented Jul 14, 2022

hadoop-yetus commented Jul 14, 2022

steveloughran commented Jul 15, 2022

hadoop-yetus commented Jul 15, 2022

mehakmeet commented Jul 18, 2022

steveloughran commented Jul 18, 2022

hadoop-yetus commented Jul 21, 2022

hadoop-yetus commented Jul 21, 2022

hadoop-yetus commented Jul 25, 2022

steveloughran commented Jul 14, 2022 •

edited

Loading