[Bug]: Writing files fails when using Azure Data Lake Gen 2

[phishncode]

Apache Hop version?

2.5

Java version?

19.0.2

Operating system

Windows

What happened?

New install of Hop 2.5.0
setup to write to Azure Data Lake Gen 2

name
key

create a new work flow name demo

try to save the workflow to azure

org.apache.hop.core.exception.HopException:
Error validating file existence for 'azure:\datalakedemo\hop\demo.hwf'

Error saving workflow to file 'azure:\datalakedemo\hop\demo.hwf'

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf".
Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.saveAs(HopGuiWorkflowGraph.java:3257)
at org.apache.hop.ui.hopgui.delegates.HopGuiFileDelegate.fileSaveAs(HopGuiFileDelegate.java:153)
at org.apache.hop.ui.hopgui.HopGui.menuFileSaveAs(HopGui.java:751)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.apache.hop.ui.core.gui.BaseGuiWidgets.lambda$getListener$1(BaseGuiWidgets.java:216)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:89)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4256)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1066)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4054)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3642)
at org.apache.hop.ui.hopgui.HopGui.open(HopGui.java:474)
at org.apache.hop.ui.hopgui.HopGui.main(HopGui.java:352)

Caused by: org.apache.hop.core.exception.HopException:
Error saving workflow to file 'azure:\datalakedemo\hop\demo.hwf'

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf".
Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.save(HopGuiWorkflowGraph.java:3229)
at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.saveAs(HopGuiWorkflowGraph.java:3254)
... 12 more

Caused by: org.apache.hop.core.exception.HopFileException:

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf".
Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:327)
at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.save(HopGuiWorkflowGraph.java:3213)
... 13 more

Caused by: org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf".
at org.apache.commons.vfs2.provider.AbstractFileObject.createFile(AbstractFileObject.java:328)
at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:301)
at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:325)
... 14 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not write to "azure:///datalakedemo/hop/demo.hwf".
at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1277)
at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1239)
at org.apache.commons.vfs2.provider.AbstractFileObject.createFile(AbstractFileObject.java:322)
... 16 more
Caused by: com.microsoft.azure.storage.StorageException: Specified feature is not yet supported for hierarchical namespace accounts.
at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196)
at com.microsoft.azure.storage.blob.CloudPageBlob.create(CloudPageBlob.java:538)
at com.microsoft.azure.storage.blob.CloudPageBlob.openOutputStreamInternal(CloudPageBlob.java:1055)
at com.microsoft.azure.storage.blob.CloudPageBlob.openWriteNew(CloudPageBlob.java:930)
at org.apache.hop.vfs.azure.AzureFileObject.doGetOutputStream(AzureFileObject.java:450)
at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1273)
... 18 more

Issue Priority

Priority: 2

Issue Component

Component: VFS

[phishncode]

This shows that VFS is creating folders and writing small files to the directory

[hansva]

It seems not all features work on data lake gen 2, do you have similar options when using a gen 1 blob storage?

[phishncode]

Thank you for the quick reply. It looks like Azure does not allow gen 1 storage anymore.

Is this a problem with VFS or specifically with HOP. I could look into fixing the issue, I would just need some pointers.

[hansva]

It seems we are doing something that was working in Gen1 but no longer works in Gen2

this is the error thrown by Azure
Caused by: com.microsoft.azure.storage.StorageException: Specified feature is not yet supported for hierarchical namespace accounts.

The VFS implementation that we have written for azure can be found here:

https://github.com/apache/hop/tree/master/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure

[hansva]

should be solved in the near future by #3676

[hansva]

going to close this one we only support hierarchical namespaces. Have been using these in the past couple of weeks/months. We have a couple of more improvement tickets but we are actively working with azure blob storage

[hansva]

.take-issue