Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests involving Hadoop/Accumulo do not work on Windows #2717

Closed
GCHQDeveloper314 opened this issue Jul 22, 2022 · 0 comments · Fixed by #2757
Closed

Tests involving Hadoop/Accumulo do not work on Windows #2717

GCHQDeveloper314 opened this issue Jul 22, 2022 · 0 comments · Fixed by #2757
Assignees
Labels
enhancement Improvement to existing functionality/feature

Comments

@GCHQDeveloper314
Copy link
Member

While looking at #2591 it was discovered that tests involving Hadoop/Accumulo do not work on Windows.

For Hadoop "Native IO is mandatory on Windows and without it you will not be able to get your installation working" (from Hadoop Wiki). This involves installing Hadoop libraries, setting the HADOOP_HOME environment variable and compiling and installing Native IO for windows. The Hadoop Native IO libraries for Windows are not distributed by Apache and must either be built from source or downloaded from a 3rd party.

Due to the above situation with Hadoop, all modules which have a test dependency on Hadoop fail to run tests on Windows. This includes at least hdfs-library,accumulo-store,accumulo-rest(plus most other rest modules), flink-library and all spark modules.

In addition there is what could be a separate problem with flink and JUnit where a temp directory cannot be deleted.

[ERROR] uk.gov.gchq.gaffer.flink.integration.operation.handler.AddElementsFromKafkaHandlerIT.shouldAddElementsWithByteArrayConsumer  Time elapsed: 8.093 s  <<< ERROR!
java.io.IOException: Failed to delete temp directory C:\Users\User\AppData\Local\Temp\junit5479111707429912277. The following paths could not be deleted (see suppressed exceptions for details): , kafkaLogDir, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:280)
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:188)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        ...
        Suppressed: java.nio.file.DirectoryNotEmptyException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277
                at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
                at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
                at java.nio.file.Files.delete(Files.java:1126)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:223)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:199)
                at java.nio.file.Files.walkFileTree(Files.java:2688)
                at java.nio.file.Files.walkFileTree(Files.java:2742)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
                ... 63 more
        Suppressed: java.nio.file.FileSystemException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277\kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex: The process cannot access the file because it is being used by another process.
                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
                at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
                at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
                at java.nio.file.Files.delete(Files.java:1126)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:218)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:199)
                at java.nio.file.Files.walkFileTree(Files.java:2670)
                at java.nio.file.Files.walkFileTree(Files.java:2742)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
                ... 63 more

There's also a potential path problem with path path processing in common-rest which may just be linked to Hadoop:

[ERROR] Failures:
[ERROR]   DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist:55
Expecting message to be:
  "Unable to deserialise graph config"
but was:
  "Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties"
Throwable that failed the check:
java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties
        at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
        at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
        at java.nio.file.Paths.get(Paths.java:84)
        at uk.gov.gchq.gaffer.store.StoreProperties.loadStoreProperties(StoreProperties.java:135)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.createGraphBuilder(DefaultGraphFactory.java:97)
        at uk.gov.gchq.gaffer.rest.factory.GraphFactory.createGraph(GraphFactory.java:55)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.getGraph(DefaultGraphFactory.java:71)
        at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at org.assertj.core.api.ThrowableTypeAssert.isThrownBy(ThrowableTypeAssert.java:59)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist(DefaultGraphFactoryTest.java:54)
@GCHQDeveloper314 GCHQDeveloper314 added the enhancement Improvement to existing functionality/feature label Jul 22, 2022
@GCHQDeveloper314 GCHQDeveloper314 added this to the v2_backlog milestone Jul 22, 2022
@lb324567 lb324567 self-assigned this Aug 24, 2022
lb324567 added a commit that referenced this issue Sep 12, 2022
lb324567 added a commit that referenced this issue Sep 14, 2022
lb324567 added a commit that referenced this issue Sep 14, 2022
lb324567 added a commit that referenced this issue Sep 15, 2022
lb324567 added a commit that referenced this issue Sep 29, 2022
lb324567 added a commit that referenced this issue Sep 29, 2022
@GCHQDeveloper314 GCHQDeveloper314 linked a pull request Oct 3, 2022 that will close this issue
@lb324567 lb324567 closed this as completed Oct 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement to existing functionality/feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants