Hierarchical DIP support #130

mistydemeo · 2014-10-23T17:19:28Z

This adds support for a new form of structMap, the logical structMap.

Directories created via SIP arrange can now be assigned a level of description label, in addition to a name. If directories with levels of description exist when the METS is created, then a logical structMap is created alongside the physical structMap. The structure of the logical structMap is identical, but the type value for directories is now the assigned level of description instead of Directory. The logical structMap can be used to assign levels of description to records on import to access systems like AtoM.

Major changes in this branch:

AtoM upload configuration has been refactored, and now now form-based; the user no longer directly writes an argument string.
If an AtoM system has been configured for upload, then the levels of description can be fetched from AtoM for use in SIP arrange.
The hierarchy of levels of description can be configured in Archivematica. (For instance, in custom LOD hierarchies, to control what levels are higher than others.)
Levels of description can be assigned to directories when they are created in SIP arrange.
Logical structMap creation in createMETS2.

mistydemeo · 2014-10-23T17:19:40Z

This history needs to be cleaned up before merging.

Hwesta · 2014-10-23T22:41:53Z

src/MCPServer/share/mysql_dev_hierarchical_dip_upload.sql

@@ -0,0 +1,11 @@
+-- Set AtoM DIP upload defaults
+INSERT INTO DashboardSettings (name, value) VALUES ('dip_upload_atom_url', 'http://localhost/ica-atom');


This should probably be 'http://localhost/atom' instead, as well as below.

Hwesta · 2015-01-14T23:05:08Z

This depends on/works with #145

Hwesta · 2015-06-11T19:39:10Z

This has been rebased on top of qa/1.x as dev/issue-6988-hierarchical-dip Should we reset dev/issue-6988-atom-dip-config to be the same as dev/issue-6988-hierarchical-dip to update this PR, or open a new one?

mistydemeo · 2015-07-14T00:00:04Z

Rebased, and squashed many of the fix commits into their original feature commits.

mistydemeo · 2015-07-14T18:53:40Z

Tested and confirmed to be working again after the rebase.

Hwesta · 2015-07-16T21:10:49Z

src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py

@@ -827,7 +829,59 @@ def createFileSec(directoryPath, parentDiv, baseDirectoryPath, baseDirectoryName
            file_elem.set("DMDID", dspaceMetsDMDID)

    return structMapDiv
-
+
+def build_arranged_structmap(original_structmap, fileGroupIdentifier):


Docstring? Especially coming to this after a year I'm not sure what this function is supposed to do.

Hwesta · 2015-07-16T21:25:54Z

src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py

+    Logical arrangement is determined using the levels of description which were assigned to them during SIP arrange.
+
+    :param etree.Element original_structmap: the structMap on which the arranged structMap should be based.
+    :param str fileGroupIdentifier: the SIP's UUID


Especially since this is new code, maybe we can start referring to sip_uuid instead of the ever-confusing fileGroupIdentifier?

Conflicts: src/dashboard/src/components/administration/urls.py

Added a button, and stumped in logic, for editing a directory's level of description.

This tracks the level of description for a filesystem entry in a SIP, either a file or a directory.

This hierarchical structMap is created by assigning levels of description to a SIP during SIP arrangement.

Added a pop-up form in the SIP arrange interface that allows editing of a directory's metadata.

Added logic, during SIP creation from the arrange panel, to assign the SIP UUID to each DB entry for a directory's level of description.

Removed the UI restriction preventing assigning level of description, in the SIP arrange panel, to files.

Changed how file level of descriptions are stored so once a SIP is created the relative location becomes relative to the SIP directory (rather than, before it becomes a SIP, the arrange directory).

refs #7068, #7071 Remove the FileLevelOfDescription and replace it with attributes on SIPArrange. Instead of attaching the level of description to the FileLevelOfDescription, populate the new level_of_description field in SIPArrange. This is easier to find, set and update when the SIP is created, and less error prone when creating a SIP from not top level directories. Do not delete SIPArrange directories, as they are now needed for level of description information. In createMETS2, search based on the path relative to the objects directory. Query SIPArrange instead of FileLevelOfDescription.

refs #7068 Add SQL to update SIPArrange for new columns. syncdb will create these on a fresh install, but not on update.

refs #7068 Add the label from the directory the SIP was created from to the logical structMap's objects div. If there was an objects directory in the arranged SIP already, delete it. Delete TYPE=Directory from the SIP level div.

refs #7070 Fix get_atom_levels_of_description to work correctly with clear=False. New LoD get unique sortorders, starting at the max value of the existing sortorders, since swapping sortorder values will not work if they have the same value. Fix hiding of promote/demote arrows only when sortorder is the max or min value (instead of 1 or length, since values may be deleted). Align all demote arrows vertically. Use STATIC_URL instead of hardcoded value. Clean up _atom_levels_of_description_sort_adjust remove unnecessary iterate and inefficient swap. Misc other minor fixes.

Updated the AJAX feed for providing SIP arrange directory data with level of directory metadata.

Updated client-side UI logic to display level of description for directories.

Added disabling of SIP arrange panel action buttons when the user has highlighted an item that the action should not apply to (creating directories within files, for example). Added an error alert in the SIP arrange UI if someone tries to create a directory within a file. Commented and cleaned up JavaScript. Conflicts: src/dashboard/src/media/js/ingest/ingest_file_browser.js

If a open opens the originals file browser and the transfer backlog isn't yet searched, trigger a search.

Squash with Add API method to fetch levels of description?

Hwesta reviewed Oct 23, 2014
View reviewed changes

qubot force-pushed the dev/issue-6988-atom-dip-config branch 2 times, most recently from 21ede76 to 2134903 Compare December 9, 2014 19:26

qubot force-pushed the dev/issue-6988-atom-dip-config branch 2 times, most recently from 035c515 to 0196ab8 Compare January 5, 2015 20:01

qubot force-pushed the dev/issue-6988-atom-dip-config branch from 0196ab8 to 60ed467 Compare January 13, 2015 18:16

qubot force-pushed the dev/issue-6988-atom-dip-config branch 2 times, most recently from 63019a9 to cb717af Compare January 29, 2015 18:05

Hwesta mentioned this pull request Feb 18, 2015

Transfer/SIP creation feedback #171

Merged

jhsimpson added this to the 1.5.0 milestone Apr 21, 2015

qubot force-pushed the dev/issue-6988-atom-dip-config branch from 749b397 to 3f83f07 Compare June 24, 2015 17:13

qubot force-pushed the dev/issue-6988-atom-dip-config branch 4 times, most recently from 7d995fa to 033c9dd Compare July 13, 2015 23:55

qubot force-pushed the dev/issue-6988-atom-dip-config branch 4 times, most recently from 4a4f44b to b3b758e Compare July 14, 2015 18:43

Hwesta reviewed Jul 16, 2015
View reviewed changes

qubot force-pushed the dev/issue-6988-atom-dip-config branch 2 times, most recently from dc7e578 to c4303e6 Compare July 16, 2015 21:15

Hwesta reviewed Jul 16, 2015
View reviewed changes

qubot force-pushed the dev/issue-6988-atom-dip-config branch from c4303e6 to b64377b Compare July 16, 2015 21:38

Misty De Meo and others added 26 commits July 24, 2015 16:47

Add stub for viewing levels of description

c6d088a

Conflicts: src/dashboard/src/components/administration/urls.py

Add API to fetch JSON levels of description

6f1675c

Added stub for SIP arrange LOD editing.

913cd42

Added a button, and stumped in logic, for editing a directory's level of description.

Add API method to fetch levels of description from AtoM

21165eb

Add FileLevelOfDescription class

5cbfd99

This tracks the level of description for a filesystem entry in a SIP, either a file or a directory.

createMETS2: Write arranged structMap to METS, if present

c457a9f

This hierarchical structMap is created by assigning levels of description to a SIP during SIP arrangement.

Add API to store levels of description for SIP objects

ca8ba8c

Correct name of view to fetch levels of description from AtoM

37da648

Added AtoM level of description admin UI.

c7eb48e

Added level of description edit modal.

bf72491

Added a pop-up form in the SIP arrange interface that allows editing of a directory's metadata.

Assign SIP UUID to directory levels of desc.

c05c6dd

Added logic, during SIP creation from the arrange panel, to assign the SIP UUID to each DB entry for a directory's level of description.

Allow files to have levels of description.

1e56438

Remove restriction of assigning LOD to files.

65b0a32

Removed the UI restriction preventing assigning level of description, in the SIP arrange panel, to files.

Changed file level of descriptions storage.

5c0539e

Changed how file level of descriptions are stored so once a SIP is created the relative location becomes relative to the SIP directory (rather than, before it becomes a SIP, the arrange directory).

Add update SQL for main_siparrange

c1b58b5

refs #7068 Add SQL to update SIPArrange for new columns. syncdb will create these on a fresh install, but not on update.

Add SIP label to objects div

1e28ca4

refs #7068 Add the label from the directory the SIP was created from to the logical structMap's objects div. If there was an objects directory in the arranged SIP already, delete it. Delete TYPE=Directory from the SIP level div.

Directory AJAX feed provides directory metadata.

2e0d716

Updated the AJAX feed for providing SIP arrange directory data with level of directory metadata.

Re-enabled editing of file metadata.

6140364

Display level of description in SIP arrange.

083cf0c

Updated client-side UI logic to display level of description for directories.

Minor CSS tweak to SIP arrange paging links.

0ffd26b

Auto-search transfer backlog in originals panel.

d7b8dde

If a open opens the originals file browser and the transfer backlog isn't yet searched, trigger a search.

Cleanup fetch_levels_of_description_from_atom

f3f2e29

Squash with Add API method to fetch levels of description?

JS: Remove unnecessary monkey patch

98c8aa2

qubot force-pushed the dev/issue-6988-atom-dip-config branch from aead678 to 98c8aa2 Compare July 24, 2015 23:48

qubot merged commit 98c8aa2 into qa/1.x Jul 24, 2015

qubot deleted the dev/issue-6988-atom-dip-config branch July 24, 2015 23:55

ross-spencer mentioned this pull request Aug 30, 2018

MCPClient: Fix package extraction #1245

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hierarchical DIP support #130

Hierarchical DIP support #130

mistydemeo commented Oct 23, 2014

mistydemeo commented Oct 23, 2014

Hwesta Oct 23, 2014

mistydemeo Dec 9, 2014

Hwesta commented Jan 14, 2015

Hwesta commented Jun 11, 2015

mistydemeo commented Jul 14, 2015

mistydemeo commented Jul 14, 2015

Hwesta Jul 16, 2015

mistydemeo Jul 16, 2015

Hwesta Jul 16, 2015

		@@ -0,0 +1,11 @@
		-- Set AtoM DIP upload defaults
		INSERT INTO DashboardSettings (name, value) VALUES ('dip_upload_atom_url', 'http://localhost/ica-atom');

Hierarchical DIP support #130

Hierarchical DIP support #130

Conversation

mistydemeo commented Oct 23, 2014

mistydemeo commented Oct 23, 2014

Hwesta Oct 23, 2014

Choose a reason for hiding this comment

mistydemeo Dec 9, 2014

Choose a reason for hiding this comment

Hwesta commented Jan 14, 2015

Hwesta commented Jun 11, 2015

mistydemeo commented Jul 14, 2015

mistydemeo commented Jul 14, 2015

Hwesta Jul 16, 2015

Choose a reason for hiding this comment

mistydemeo Jul 16, 2015

Choose a reason for hiding this comment

Hwesta Jul 16, 2015

Choose a reason for hiding this comment