HADOOP-16650. ITestS3AClosedFS failing -junit test thread.

[steveloughran]

Change-Id: Ia58f5e1dd57588a28081f9c4092c010b3132856a

tested: s3 ireland. now going to CP and test on the specific backport branch where I encountered the problem

[steveloughran]

confirmed; fixes leaks.

@bgaborg - could you take a look @ this? All I'm doing is stripping the junit and surefire threads out so they don't interfere with the results

[bgaborg]

Yes, I'll take a look.

[steveloughran]

thanks. full test run happy except for the root dir tests hinting that there was a test dir which shouldn't be there. All cleaned up and fsck is happy now :)

[bgaborg]

given the usage of string literals instead of final fields in the environment of the method, I'm ok with this change.
LGTM, running a full test before +1 and commit.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	213	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1583	trunk passed
+1	compile	42	trunk passed
+1	checkstyle	33	trunk passed
+1	mvnsite	56	trunk passed
+1	shadedclient	944	branch has no errors when building and testing our client artifacts.
+1	javadoc	26	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	57	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	33	the patch passed
+1	compile	26	the patch passed
+1	javac	26	the patch passed
+1	checkstyle	17	the patch passed
+1	mvnsite	32	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	885	patch has no errors when building and testing our client artifacts.
+1	javadoc	22	the patch passed
+1	findbugs	67	the patch passed
		_ Other Tests _
+1	unit	89	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		4237

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1639/1/artifact/out/Dockerfile
GITHUB PR	#1639
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0a975e812f5d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / a031388
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1639/1/testReport/
Max. process+thread count	338 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1639/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[bgaborg]

I have a new failure during the test:

[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 67.573 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 31.68 s  <<< FAILURE!
java.lang.AssertionError:
Expected no results from listLocatedStatus(/), but got 1 elements:
S3ALocatedFileStatus{path=s3a://cloudera-dev-gabota-versioned-bucket-ireland/fork-0008; isDirectory=true; modification_time=1570716744830; access_time=0; owner=gabota; group=gabota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId='']
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.assertNoElements(AbstractContractRootDirectoryTest.java:218)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:202)
	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 1.558 s  <<< FAILURE!
java.lang.AssertionError:
listStatus(/) vs listLocatedStatus(/) with
listStatus =S3AFileStatus{path=s3a://cloudera-dev-gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gabota; group=gabota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null listLocatedStatus = S3ALocatedFileStatus{path=s3a://cloudera-dev-gabota-versioned-bucket-ireland/fork-0008; isDirectory=true; modification_time=1570716746674; access_time=0; owner=gabota; group=gabota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId='']
S3ALocatedFileStatus{path=s3a://cloudera-dev-gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=1570716746674; access_time=0; owner=gabota; group=gabota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId=''] expected:<1> but was:<2>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

I don't think this is related. Any thoughts about this @steveloughran ?

[steveloughran]

That is exactly what I saw. In the past that generally means some of the test with/without s3guard has put a tombstone above a path which has a file deep underneath. Our test runs are doing a dump of what's there iner target/, if you haven't done a maven clean yet you should be able to find the output

[bgaborg]

This failure is not related but needs to be fixed.
We can handle this separately.
+1 on the patch. LGTM

[steveloughran]

thanks, merged.