IIIF Viewers with IPFS

[daviddias]

IIIF (International Image Interoperability Framework) is hosting a conference at the Vatican on June 5-9. We (@diasdavid and @flyingzumwalt) will be present at that conference to talk about the benefits of using decentralized Web technologies that institutions like IIIF can benefit from.

We will be collaborating with @edsilv and @aeschylus to prepare a demo to showcase at the conference, of both Mirador and UniversalViewer using IPFS to exchange and update annotations made on the images. This issue will be used to coordinate this effort as a main central hub.

Mirador and UniversalViewer are both viewers of IIIF data that follow IIIF defined data model, they are created by independent researchers, but there is a lot of collaboration and idea sharing.

From the design thinking we've done, we know that we need:

• A way to propagate changes on the Annotation List
• A way to synchronize the state of the Annotation List for each document.

This converts into roughly the following milestones:

1. Mechanism to propagate changes on the annotation list (this can be using libp2p pubsub directly)
2. Synchronize the Annotation list through IPFS directly (using orbit-db or another CRDT library that is mounted on top of IPFS
3. Load the entire collection of images into the viewer through IPFS itself
4. Load and run the entire viewer using only js-ipfs, requiring no plugins installed.

This suggested that we just need to create primitives for get and put with the ability to follow changes or updated in a document, once that part is done, it can be fully integrated.

Other important pieces of context:

• Annotation list is a simple JSON document
• Web Annotation Data Model - https://www.w3.org/TR/annotation-model/
• @dignifiedquire's quick experiment done for a previous conference (might not be working at the moment) - https://github.com/dignifiedquire/anno
• Viewdir - a collection of modules shared between both viewers https://github.com/viewdir
• UV uses https://svelte.technology/
• More context on the origins of the proposed demo IIIF Annotation Demo for Vatican orbitdb/orbitdb#213
• CRDT Research repo https://github.com/ipfs/research-CRDT

[daviddias]

alright, just had a good chat with @pgte on this endeavor, he is in!

To move forward:

• @edsilv and @aeschylus, could you provide current Annotation List examples, changes, updates and so on.
• There is a lot of room to explore here, but I feel that the milestones described above are the best to start with, let me know if you feel something should be added from the start.
• We should set a weekly standup to make sure this demo is progressing and needs are met. @edsilv, @aeschylus, @flyingzumwalt would you be available to do this? I can set up a Doodle.

Let's try to all meet this week :)

[edsilv]

Great! Here's an example of an AnnotationList:

https://wellcomelibrary.org/annoservices/search/b18035723?q=gene

(http://iiif.io/model/shared-canvas/1.0/#AnnotationList)

This is the format Mirador uses to store annotations and the UV uses to display search results.

This will eventually be replaced with the Web Annotation Data Model as part of IIIF Presentation API 3.0, but this is still some way off:

IIIF/api#496

(https://www.w3.org/TR/annotation-model/#annotation-collection)

Milestone 3 states that the images should be loaded over IPFS. Currently we only have this very simple fully-IPFS-enabled IIIF manifest: http://universalviewer.io/examples/?manifest=https%3A%2F%2Fipfs.io%2Fipfs%2FQmYomyCpT1vKNovYvTjAStYq31JoaH8xdivjmWcDj5mpM1

Milestone 4 would require some work from @aeschylus to port Mirador to be able to be loaded over IPFS. Currently the UV can be loaded using the ipfs.io gateway. Are you suggesting that we bypass the gateway and use js-ipfs to load all of the viewer assets?

Perhaps in order to keep things manageable initially we could focus on enabling serverless communication between the viewers over js-ipfs, with the viewers themselves and IIIF manifests being loaded over HTTP. We could then move on to fully IPFS-ifying the viewers and IIIF manifests + assets?

I need to complete the work that will allow me to update the UV search results programmatically as opposed to on user interaction. This will be necessary to be able to respond to the IPFS annotation endpoint. I'm working on giving the UV an external API and expect this to be ready in the next few weeks.

We will also need to be able to specify the annotation endpoint to subscribe to somehow when starting the viewers. Perhaps a text field that we paste the name of the endpoint into?

A weekly standup sounds good!

[edsilv]

At the bottom of https://wellcomelibrary.org/annoservices/search/b18035723?q=gene you'll see a hits array. This is extra data to provide context to search results. We don't need to worry about this as the UV doesn't require it.

[daviddias]

Let's find the time for that weekly standup

@edsilv @pgte @flyingzumwalt @aeschylus please fill: https://doodle.com/poll/bqpzs9xxzfde25pc

Everyone else is welcome to add their availability if they are interested in actively participating in this endeavor, otherwise, we will have to favor the schedule of the people contributing. Thank you :)

[daviddias]

\o/ woot, we have a schedule! :)

Every friday at:

Zoom URL is: https://zoom.us/j/131133322

[aeschylus]

Great!

I'll look forward to speaking to everyone on the 28th. For reference, here is the link to a fork of orbitdb we've made where the main example has been tossed up to work as a textarea.

https://github.com/viewdir/orbit-db/tree/initial-hacking

More context is in orbitdb/orbitdb#213, noted above.

[flyingzumwalt]

Standup Friday April 21

Agenda

• Welcome
• Round of Intros
• Define target for next week

Misc:

Recent Article about using pubsub on IPFS

Notes

• Participants: @edsilv, @diasdavid, @flyingzumwalt, @pgte, @aeschylus

There are 2 main vierwers in the IIIF space

• Mirador (Stanford, Mellon-funded), which is more of a scholarly workbench and a tool for authoring annotations.
• Universal Viewer, aka "UV"(Wellcome Trust, British Library) - more of a generic viewer for many data formats (Image, PDF, Video, 3D objects...)

Main IIIF APIs:

• Images api - way to display images from an image service
• Presentation api - way of listing canvases (ie. a series of pages)

Other Apis: search, annotation, etc.

• new annotations API will use the WebAnnotations Model (w3c spec)
  • lets you create AnnotationList objects

The Demo: Serverless IIIF Image Annotations

• mirador producing annotations, writing them ot IPFS & UV showing those annotations -- completely serverless collaborative editing of the annotations

Work Done so Far

Drew and Ed got a basic version of UV running on orbit-db. they got it working but ran into troubles with orbit and confusion about what they should do

Questions:

• Will the IPFS daemon run in the browser? Will people have to run a go-ipfs daemon in order to use IPFS pubsub?
  • 2 attempts: one in november 2016 with @dignifiedquire. Then again in March 2017 using the latest orbit-db (hoping that they would be able to to pure js implementation)

Q: can we use js-ipfs?
A: you can use either. go-ipfs has some features that js-ipfs doesn't have

Current Work

NEED for demo:
Easy way for everyone in the room to get the demo code, including a working IPFS node

• needs ipfs-download.js, or something similar

Theatrics of the demo:

• everyone downloading ipfs binaries over HTTP will be slow
• everyone downloading IIIF tiles over HTTP will be slow
• everyone downloading IIIF tiles over IPFS ifs they have ipfs running will (should?) be much faster

Where the work will happen:

• @pgte will create a repo for this annotations-pubsub work in github.com/pgte/ ... we might create an "ipfs-labs" github org for this kind of work to live
• the first-pass version of this demo only relies on ipfs, which includes ipfs-pubsub. it doesn't need CRDTs, so doesn't need orbit-db
• if we have time before the demo, we will add CRDT support.

Action Items:

• figure out what the annotations pubsub API should look like
• get basic annotations on pubsub working in both viewers
  • updates are the whole blob of annotations (refresh the whole list on every change)
• prepare the theatrics of the demo
• Add CRDTs (deal with chronological order of annotations with many editors)

What Ed & Drew need:

• (pubsub consumer) a way for mirador & UV to listen for an event telling you that something has changed
• (pubsub producer) a way to send updates when you have changed something
• a way to declare a pubsub topic/channel and subscribe to it

[pgte]

Created repo, can move it later to org: https://github.com/pgte/ipfs-iiif-annotations

[edsilv]

Here's the event on sched: https://2017iiifconferencethevatican.sched.com/event/AChW/presentation-interoperable-peer-to-peer-research-with-iiif-and-ipfs-room-5

[daviddias]

Standup Friday April 28

Agenda

• Standup

Notes

• Participants: @edsilv, @diasdavid, @flyingzumwalt @brendam O'Brian
  • Pedro won't be able to make it today
• iiifDB https://github.com/pgte/ipfs-iiif-db
• Testing
  • Write down test cases for iiif-db with potential scenarios of how it is going to be used by uv and mirador (ed)
  • Use JEST (https://facebook.github.io/jest/) to test UV
• Explore how we can achieve pagination of annotations using IPLD
• Food for thought - namespace of the publisher instead of just contentId
• Ship an example of iiif-db using browserify (@pgte)
• Goal: Have a thing that @edsilv can use by next Tuesday //cc @pgte

Video: it is being rendered and uploaded

[edsilv]

Some more notes from the 28/04/17 standup:

We are currently using the Open Annotation spec which is what the UV and Mirador both support and is perfect for an MVP. However, IIIF will use the Web Annotation Data Model in future. This has support for AnnotationPages which would allow us to break up the data into more manageable chunks instead of a single large json blob.

It also supports agents which would allow us to identify and filter by user.

The model uses IRIs for ids, which means fs:/ipfs/<hash> is a valid identifier if we want to permit content addressing.

Also worth noting that AnnotationPages are based upon Activity Streams Collections. Activity Streams is a more generic specification and can be applied to many other types of actions.

Create a front end test harness? Use Jest: https://facebook.github.io/jest/

[daviddias]

@pgte has finished the first milestone, you can see it here: ipfs-shipyard/ipfs-iiif-db#1 (comment)

@edsilv @aeschylus can I get your ack? The example includes a demo video

[edsilv]

Hey, Looks great! I'll have a play with it before the standup :-)

…

On 4 May 2017 8:23 pm, "David Dias" ***@***.***> wrote: @pgte <https://github.com/pgte> has finished the first milestone, you can see it here: ipfs-shipyard/ipfs-iiif-db#1 (comment) <ipfs-shipyard/ipfs-iiif-db#1 (comment)> @edsilv <https://github.com/edsilv> @aeschylus <https://github.com/aeschylus> can I get your ack? The example includes a demo video — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#240 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABRceUwzHYN_O0C4eH6MkBWiJkysRO1aks5r2iWegaJpZM4M-UZr> .

[aeschylus]

Having a look now and getting our example put up.