Repotool2 sending incorrect headers to Sword2 for file deposits

[tomwrobel]

Related to #14

When repotool deposits a file, it is sending the 'in progress' flag incorrectly for file and file metadata actions. For object-level actions its behaviour is correct.

The in_progress flag should indicate that more activity is set to take place on an object. When set to False, it indicates that deposit has finished and that the receiving server (Hyrax) should act on the object.

See the scenarios below, in each case, the in_progress header was set correctly for object creation (set to True), and incorrectly for file creation and metadata update (set to False for intermediate actions and to True for the final action).

• depositing a single file
• depositing two files
• depositing three files
• redepositing a single file
• redepositing two files

Log file results are in the comment below.

[tomwrobel]

Note that files were added to the UI in number order (one.txt, two.txt, three.txt) etc., and were received in number order. It is not the case that the order of file save has thrown anything off

Depositing a single file to 1012334 (uuid_f99fda65-345b-498b-a040-260df3

fbc7dc)

Action	url (after sword/collections/default/works/)	method	in_progress value
Create new work	POST	uuid_f99fda65-345b-498b-a040-260df3fbc7dc	True
Deposit new file (eleven.txt)	POST	uuid_f99fda65-345b-498b-a040-260df3fbc7dc/file_sets	False
Set (actually update) binary file metadata for file	PUT	uuid_f99fda65-345b-498b-a040-260df3fbc7dc/file_sets/sympc247ds08k	True
Harvest information from Hyrax	GET	uuid_f99fda65-345b-498b-a040-260df3fbc7dc	nil

Depositing two files to 1012335 (uuid_f99fda65-345b-498b-a040-260df3

fbc7dc)

Action	url (after sword/collections/default/works/)	method	in_progress value
Create new work	POST	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db	True
Deposit first new file (one.txt)	POST	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets	False
Set (actually update) binary file metadata for file one.txt	PUT	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets/symppn89d656c	False
Deposit second new file (two.txt)	POST	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets	False
Set (actually update) binary file metadata for file two.txt	PUT	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets/symp1544bp080	True
Harvest information from Hyrax	GET	uuid_8b710238-a33d-43ab-85e2-b2dab44d08db	nil

Depositing three files to 1012336 (uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a)

Action	url (after sword/collections/default/works/)	method	in_progress value
Create new work	POST	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a	True
Deposit first new file (one.txt)	POST	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets	False
Set (actually update) binary file metadata for file one.txt	PUT	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/sympx059c732j	False
Deposit second new file (two.txt)	POST	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets	False
Set (actually update) binary file metadata for file two.txt	PUT	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/symphh63sv88g	False
Deposit third new file (three.txt)	POST	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets	False
Set (actually update) binary file metadata for file three.txt	PUT	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/sympxg94hp52h	True
Harvest information from Hyrax	GET	uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46aqq	nil

Redepositing a single file - add eleven.txt to 1012310 (uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4)

Action	url (after sword/collections/default/works/)	method	in_progress value
Deposit new file (eleven.txt)	POST	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets	False
Set (actually update) binary file metadata for file	PUT	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/symp8s45q877q	True
Harvest information from Hyrax	GET	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4	nil

Redepositing multiple files - and nine.txt and ten.txt to 1012310 (uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4)

Action	url (after sword/collections/default/works/)	method	in_progress value
Deposit first new file (nine.txt)	POST	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets	False
Set (actually update) binary file metadata for first file	PUT	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/symp1831cj92d	False
Deposit second new file (ten.txt)	POST	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets	False
Set (actually update) binary file metadata for second file	PUT	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/sympbg257f046	True
Harvest information from Hyrax	GET	uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4	nil

Note that files were added to the UI in number order (one.txt, two.txt, three.txt) etc. It is not the case that the order of file save has thrown anything off.

[tomwrobel]

@AndrewBennet there is a workaround for this: I can invert the meanings of the in_progress flag for filesets to match Repotool2's current behaviour. We have to be VERY careful to undo this once it's fixed though. Thoughts?

The workaround is to invert the significance of in_progress = true and in_progress=false in app/controllers/concerns/integrator/hyrax/file_sets_behaviour.rb. A comment has been left in that file to link to this ticket

[tomwrobel]

Workaround is in place: github.com/tomwrobel/willow_sword feature/ora_customizations commit commit 995c7ef4e3a24d901e3956672a56a8528a4420df

[AndrewBennet]

You are quite right - I've made the classic programming error of inverting my booleans (when dealing with the files only).

We should be able to fix in a 5.18 patch. I'll let you know which version this gets in to, so you can remove your workaround when upgraded.

[tomwrobel]

Brilliant! I'll move to post-release!

[AndrewBennet]

Just to let you know: we released this in the patch to 5.18 out today (and also included in a 5.19 patch). You'll need to coordinate your workaround-removal with the Elements upgrade, of course :)

[tomwrobel]

There is now a configuration flag in willow_sword that is set in Hyrax and will be set by ansible. If config.in_progress_workaround = true the workaround will be used, once fixed, this value can be removed or set to false