Update support for iRODS #17341

korydraughn · 2025-08-14T22:08:32Z

This PR updates support for iRODS by replacing Jargon (legacy iRODS library) with irods4j.

The foundational work was implemented by @MINGYJ, a recent iRODS intern. My commits are mainly polish and corrections around the use of the irods4j library.

Basic functionality is working - i.e. single stream uploads/downloads, renames, editing, etc.

The parallel transfer implementation doesn't appear to be working as intended. This is likely due to not having a full understanding of how the Cyberduck components fit together - i.e. Read/WriteFeature vs Upload/DownloadFeature.

Here are the steps for performing parallel transfer (i.e. multipart uploads) in iRODS.

https://docs.irods.org/5.0.1/system_overview/data_objects/#how-to-perform-a-parallel-write-over-the-zone-port

Putting in draft for now. Feedback and guidance on how to implement proper support for parallel transfer would be greatly appreciated.

Resolves #14449.

defaults/src/main/resources/default.properties

irods/bin/src/test/resources/iRODS (iPlant Collaborative).cyberduckprofile

irods/output.txt

irods/src/main/java/ch/cyberduck/core/irods/IRODSCopyFeature.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSDeleteFeature.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java

irods/src/test/java/ch/cyberduck/core/irods/IRODSExceptionMappingServiceTest.java

irods/src/test/java/ch/cyberduck/core/irods/input.txt

irods/src/test/java/ch/cyberduck/core/irods/output.txt

defaults/src/main/resources/default.properties

dkocher · 2025-09-01T18:47:12Z

Putting in draft for now. Feedback and guidance on how to implement proper support for parallel transfer would be greatly appreciated.

Discussed in #14449 (comment)

korydraughn · 2025-10-09T00:10:58Z

TODO: Update documentation for iRODS - https://docs.cyberduck.io/protocols/irods/

trel · 2025-10-09T00:12:39Z

updating the docs is a separate repo, would have its own PR

korydraughn · 2025-10-09T01:18:00Z

Rebased PR on top of master.

Verified the following:

Parallel/Single stream uploads and downloads works
Config options defined in new iRODS profile are honored
PAM authentication works via the pam_password auth scheme
Other operations work too - rename, delete, etc.

With this PR, users will not be able to calculate checksums on data in an iRODS zone. Cyberduck will report checksums in their hex form if they exist in iRODS. We could add an option to the iRODS profile which allows users to instruct Cyberduck to calculate checksums following an upload, but that didn't feel like the correct approach, mainly because profiles seem to be loaded once on program start and never reread.

As for the unit/integration tests, I'm not sure how the implementation can be tested without a real iRODS server.

Finally, this PR bumps the minimum iRODS version requirement to 4.3.2. A new version of irods4j will be needed before this is merged. See irods/irods4j#126. Will get a new version released ~~this week~~ soon.

profiles/default/iRODS.cyberduckprofile

irods/src/main/java/ch/cyberduck/core/irods/IRODSTimestampFeature.java

profiles/default/iRODS.cyberduckprofile

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java

irods/src/test/resources/iRODS.cyberduckprofile

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java

korydraughn · 2025-10-14T17:16:32Z

@dkocher When I move a file in or out of a directory, I see a new iRODS connection get created. It disappears eventually, but not cleanly.

Can you explain why Cyberduck creates a new connection every time a file is moved?

The move class is very similar to other implementations so it's not clear to me what causes the new connection.

dkocher · 2025-10-14T17:21:22Z

@dkocher When I move a file in or out of a directory, I see a new iRODS connection get created. It disappears eventually, but not cleanly.

Can you explain why Cyberduck creates a new connection every time a file is moved?

The move class is very similar to other implementations so it's not clear to me what causes the new connection.

You will need to return Statefulness.stateless in IRODSProtocol#getStatefulness.

irods/bin/pom.xml

korydraughn · 2025-10-14T17:36:32Z

@dkocher When would someone choose stateless or stateful?
If I change the protocol to stateless, how will that affect other operations?
If I continue using stateful, what needs to change to make sure the additional connections are closed immediately?

To provide some context, a single iRODS connection cannot be used to execute multiple API operations in parallel. A single request is sent and the server returns a response.

dkocher · 2025-10-15T06:27:05Z

If you can share credentials with us for your test environment we can run daily integration tests from our CI.

dkocher · 2025-10-15T06:30:33Z

@dkocher When would someone choose stateless or stateful? If I change the protocol to stateless, how will that affect other operations? If I continue using stateful, what needs to change to make sure the additional connections are closed immediately?

To provide some context, a single iRODS connection cannot be used to execute multiple API operations in parallel. A single request is sent and the server returns a response.

We will have to keep it stateful then as otherwise it will be attempted to use a single connection for multiple actions in parallel, i.e. when the user is browsing folders or a file transfer with multiple files in parallel.

I will need to review how we can still support the native copy feature implementation for iRODS that would not require a new connection.

korydraughn · 2025-10-15T14:05:45Z

If you can share credentials with us for your test environment we can run daily integration tests from our CI.

We don't have a CI system for people to hook into yet. We have a small set of tools which make it easy to launch one or more iRODS servers for testing. It's likely overkill for your needs though.

With that said, building a containerized environment for testing iRODS is pretty easy. I'm happy to put together a Docker compose project for the iRODS component. That will allow you to launch it on a local computer and have it available for testing.

We will have to keep it stateful then as otherwise it will be attempted to use a single connection for multiple actions in parallel, i.e. when the user is browsing folders or a file transfer with multiple files in parallel.

I will need to review how we can still support the native copy feature implementation for iRODS that would not require a new connection.

So, because the IRODSProtocol class reports iRODS as being stateful, does that mean every operation results in a new iRODS connection being created?

If that's true, why is it that the Move operation results in connections which do not disconnect? No other operation shares that behavior.

I added some log statements to my local build to try and box in what is leading to the additional connections, but it didn't help. However, it did reveal a high number of instantiations of IRODSMoveFeature. Any idea why the move operation is instantiated so much?

dkocher · 2025-10-15T14:56:44Z

We have other usages of Docker Compose containers in integration tests, thus that should be feasible.

korydraughn · 2025-10-15T15:42:35Z

Where should I place the Docker Compose project? Is the test directory for irods appropriate?

I figure you can move things around if needed.

korydraughn · 2025-10-15T19:55:19Z

@dkocher What triggers the Copy implementation?

Using the Duplicate option within the context menu doesn't appear to trigger it.

korydraughn · 2025-10-15T20:02:58Z

Docker compose project added under test directory.

Squashing everything down.

irods/src/main/java/ch/cyberduck/core/irods/IRODSAttributesFinderFeature.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSListService.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSChecksumUtils.java

dkocher · 2025-11-09T19:11:45Z

irods/src/main/java/ch/cyberduck/core/irods/IRODSConnectionUtils.java

+
+        options.clientServerNegotiation = preferences.getProperty(IRODSProtocol.CLIENT_SERVER_NEGOTIATION);
+        options.sslProtocol = preferences.getProperty(IRODSProtocol.TLS_PROTOCOL);
+        options.sslTruststore = preferences.getProperty(IRODSProtocol.TLS_TRUSTSTORE);


We don't want to support a trust store referenced like this but instead it should be possible to inject the trust store configured depending on the runtime environment you can obtain from the IRODSSession constructor parameter.

Do you have an example demonstrating that?

Like this was previously used with the Jargon library ¹.

Footnotes

https://github.com/iterate-ch/cyberduck/blob/13f08974faac1872330be7f41d33f9bfe3a25de7/irods/src/main/java/ch/cyberduck/core/irods/IRODSSession.java#L110 ↩

Will take a look at what Jargon is doing and see how that might fit into irods4j.

@dkocher How do I test SSL/TLS in Cyberduck?

Does Cyberduck configure the TrustManager for the user?

Is there documentation showing how to enable it?

That aligns with the changes I've made.

If a user uses nonstandard locations for certificates, how does that work?

If you support mTLS, it is required the certificate can be found in the user trust store on the system, e.g. the login keychain on macOS or the Windows Certificate Store.

To verify certificates the chain must be trusted, e.g. intermediate and/or root certificates are found in the system trust store. Otherwise the user is prompted to proceed when missing trust.

Based on what you said, my understanding is that Cyberduck requires certificates to be installed into a standard location.

Hypothetically, what happens if a user isn't allowed to install the necessary certs into the standard location?

Otherwise the user is prompted to proceed when missing trust.

Can you say more about this? What does it mean to "proceed with missing trust"?

Targeting macOS and Windows I am not aware the user could be restricted to add certificates to the keychain or certificate store respectively.

Ok. I'll remove the file based options for TLS and verify things work.

irods/src/test/java/ch/cyberduck/core/irods/IRODSDockerComposeManager.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSTimestamp.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSExceptionMappingService.java

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java

dkocher · 2025-11-09T20:17:01Z

irods/src/main/java/ch/cyberduck/core/irods/IRODSSession.java

 import java.text.MessageFormat;

-public class IRODSSession extends SSLSession<IRODSFileSystemAO> {
+public class IRODSSession extends SSLSession<IRODSConnection> {


Wondering if it would be feasible to return IRODSConnectionPool as the client instead to allow concurrent usage.

I'm sure that returning a connection pool is possible. The challenge is in sizing it and determining what should happen when all connections are exhausted.

Alternatively, you can remove the management of a connection from the session class and have the feature implementations create connections as needed. This reduces the need for coordination. The penalty for this is frequent authentication. The benefit is that you don't hold on to iRODS connections for long periods of time.

This PR will give you what you need to test your idea once merged though.

dkocher · 2025-11-10T08:29:06Z

That's surprising. Are you able to launch the docker compose setup from the command line directly?

What OS are you using? What version of docker are you running?

I have edited my comment above. Only reproducible on macOS.

korydraughn · 2025-11-10T18:10:16Z

I have edited my comment above. Only reproducible on macOS.

That's expected. iRODS does not support ARM, yet.

One way around this is to launch the docker compose project on a different computer and point the tests at it.

…it exception handling logic

…ng logic

dkocher · 2025-11-10T20:33:46Z

TODO: Update documentation for iRODS - https://docs.cyberduck.io/protocols/irods/

A pull request in the repository would be much appreciated. I am particular interested about compatibility with different iRODS server versions.

korydraughn · 2025-11-10T20:59:46Z

No problem. We'll make sure to open a PR to update the docs for iRODS.

I am particular interested about compatibility with different iRODS server versions.

irods4j targets iRODS 4.3.2 and later. That's one of the reasons why the iPlant/CyVerse deployment no longer works with this build. iRODS 4.3.5 is planned as the final release of the iRODS 4.3 series. iRODS 5.0.2 was released at the beginning of October.

irods4j is developed and maintained by the iRODS Consortium and tracks the server's feature set closely. Maintaining compatibility is very important for us and one of the reasons for developing irods4j. Support for iRODS in Cyberduck is guaranteed to be stronger with the switch to irods4j.

I'm especially happy with the introduction of the docker compose project. That will allow you and the rest of the Cyberduck team to improve and fix bugs in the iRODS component without needing to rely on external organizations or even the iRODS team (of course, we're happy to help :-) ).

korydraughn · 2025-11-13T15:19:26Z

NTS: Add test showing the PAM issue is resolved.

korydraughn · 2025-11-13T15:21:13Z

@dkocher Please see questions at #17341 (comment).

dkocher · 2025-11-19T20:51:03Z

The merge window for 9.3 is closing this week.

korydraughn · 2025-11-21T21:18:46Z

That's fine. No need to wait on this PR. It can be part of the next release.

Plus I still need to update the documentation.

korydraughn commented Aug 14, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

korydraughn commented Aug 14, 2025

View reviewed changes

defaults/src/main/resources/default.properties Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

dkocher changed the title ~~[#14449] Update support for iRODS~~ Update support for iRODS Sep 1, 2025

This comment was marked as resolved.

Sign in to view

korydraughn force-pushed the jargon_to_irods4j branch from 660266c to f914720 Compare October 9, 2025 00:37

korydraughn commented Oct 9, 2025

View reviewed changes

profiles/default/iRODS.cyberduckprofile Outdated Show resolved Hide resolved

profiles/default/iRODS.cyberduckprofile Outdated Show resolved Hide resolved

irods/src/main/java/ch/cyberduck/core/irods/IRODSTimestampFeature.java Show resolved Hide resolved

trel reviewed Oct 9, 2025

View reviewed changes

dkocher added this to the 9.3 milestone Oct 9, 2025

korydraughn mentioned this pull request Oct 9, 2025

iRODS: Add general purpose profile (main) iterate-ch/profiles#163

Open

1 task

trel reviewed Oct 9, 2025

View reviewed changes

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java Outdated Show resolved Hide resolved

trel reviewed Oct 9, 2025

View reviewed changes

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java Outdated Show resolved Hide resolved

trel reviewed Oct 9, 2025

View reviewed changes

irods/src/test/resources/iRODS.cyberduckprofile Show resolved Hide resolved

korydraughn commented Oct 13, 2025

View reviewed changes

irods/src/main/java/ch/cyberduck/core/irods/IRODSUploadFeature.java Outdated Show resolved Hide resolved

korydraughn commented Oct 14, 2025

View reviewed changes

irods/bin/pom.xml Outdated Show resolved Hide resolved

dkocher reviewed Nov 9, 2025

View reviewed changes

MINGYJ and others added 10 commits November 9, 2025 20:53

iRODS: Replace Jargon with irods4j

ce2f25e

iRODS: Fix bugs, expand options, and clean up implementation

2870ae7

iRODS: Add Docker Compose project for testing

99b1ac6

iRODS: Fix tests

e7bac88

squash w/ 2870ae7. rename timestamp class

03c9c17

squash w/ 2870ae7. give string literals names

c3f2506

squash w/ 2870ae7. use convenience function for boolean

fda6abb

squash w/ 2870ae7. get app name dynamically

1d455b9

squash w/ 2870ae7. remove override for logout() method

4b28090

squash w/ 2870ae7. wrap call to super.disconnect() in try-catch block

8bed7e5

korydraughn added 7 commits November 10, 2025 14:46

squash w/ 2870ae7. use enum

2a5ae40

squash w/ 2870ae7. react to status.isExists()

27f4dc9

squash w/ 2870ae7. add test for checksum utils

b3eb7f8

squash w/ 2870ae7. lock exception mapper to IRODSException

78664bd

squash w/ 2870ae7. use cyberduck threadpool for multipart upload. spl…

7b6176a

…it exception handling logic

squash w/ 2870ae7. handle redundant variables. split exception handli…

12beeda

…ng logic

squash w/ 2870ae7. split exception handling logic in feature impls

d6d3f9b

korydraughn force-pushed the jargon_to_irods4j branch from 43c5f08 to d6d3f9b Compare November 10, 2025 19:59

dkocher added the irods IRODS Protocol Implementation label Nov 19, 2025

dkocher modified the milestones: 9.3, 9.4 Nov 21, 2025

Uh oh!

Update support for iRODS #17341

Are you sure you want to change the base?

Update support for iRODS #17341

Uh oh!

Conversation

korydraughn commented Aug 14, 2025 • edited by dkocher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

dkocher commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

korydraughn commented Oct 9, 2025

Uh oh!

trel commented Oct 9, 2025

Uh oh!

This comment was marked as resolved.

korydraughn commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

korydraughn commented Oct 14, 2025

Uh oh!

dkocher commented Oct 14, 2025

Uh oh!

Uh oh!

korydraughn commented Oct 14, 2025

Uh oh!

dkocher commented Oct 15, 2025

Uh oh!

dkocher commented Oct 15, 2025

Uh oh!

korydraughn commented Oct 15, 2025

Uh oh!

dkocher commented Oct 15, 2025

Uh oh!

korydraughn commented Oct 15, 2025

Uh oh!

korydraughn commented Oct 15, 2025

Uh oh!

korydraughn commented Oct 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

korydraughn commented Aug 14, 2025 •

edited by dkocher

Loading

dkocher commented Sep 1, 2025 •

edited

Loading

korydraughn commented Oct 9, 2025 •

edited

Loading