01 25 2023 Implementers Meeting

Attendees:

• Arran Griffith
• Becky Andresen
• Rosalyn Metz
• Dan Field
• Peter Winkles
• Birkin Diana
• Daniel Salwerowicz
• Tom Wrobel

Notes:

New Fedora Member

• Dan Field introduced himself and told what he works with in addition to being a Fedora developer.
  • He's been working at the National Library of Whales for around 19 years and is currently in the position of Head of Software Engineering.

Outstanding Issues/PR's

• Integrate the new S3 transfer manager #86

• Related to above open issue #86/#88

  • The new S3 transfer manager does not support the same options as existing S3 client. [Issue 86 on ocfl-java (https://github.com/OCFL/ocfl-java/issues/86) and Pull request 88 on ocfl-java.

  • While the new transfer manager and CRT client lead to improvements in S3 performance, there are some issues with it.

  • The new transfer manager requires the new CRT client to run, however the CRT client is, as of now, an async-only client.

  • Previously ocfl-java used the sync S3 client, so it leads to breaking changes for S3 users of ocfl-java.

  • Additionally the CRT client does not support many options, for example path-style access.

  • Some users do use non-native clients and many do use old-style implementations and setops which can lead to issues.

  • This required thorough testing to make sure that none of the S3 users will be negatively affected.

  • There are issues that Becky Andersen has experienced with relation to it.

    • One of them being the fact that S3 client returned a generic S3 exception, rather than NoSuchKeyException.
    • Becky Andersen has experienced further issues and will be doing further testing to provide more information about them.

  • Might be good idea to find other S3 users and see if they can do some tests on their systems to see if they experience any issues.

• Issue With Empty Files Written To Azure Running BTRFS

  • User had an issue with ocfl-java was writing claiming to successfully write files even though the files were actually empty, leading to broken files.

• Issue #2023 on fcrepo

  • Ocfl-Java didn't see this issue as it was writing asynchronously, synchronous writing should help with the issue, and it has been already implemented.

  • However it makes the write operations run slower.

  • Need to consider whether ocfl-java should be the one writing synchronously or if it should be configurable option.

  • The source of the issue might be with Azure on Kubernetes and the attached persistent volume configured with BTRFS. It also had a mountoption to "compress" stored files.

  • That was most likely the cause that the write operations failed, as BTRFS might not like the fact that many small files have been written in quick succession.

  • The problem now is that the user now has a lot of files that need to be cleaned up.

  • The other worrying issue is that the error could happen and it would take a while for the user to notice it has happened.

  • So in theory they could go about expecting files to be in order without noticing anything was wrong, making it hard to repair the broken files.

• Peter would like input on making this function configurable, and what the default should be.

  • Feedback from Daniel Salwerowicz is to make the old option, asynchronous writing, the default, and synchronous writing an opt-in.

Updating To OCFL 1.1

As Becky pointed out the process of updating from OCFL version 1.0 to 1.1 has been very easy thanks to good documentation provided by Peter.

Implementer Updates

None

Next Meeting: March 15 @ 10am Eastern

• Arran to correct and send invite to those missing it.