Spec

[PVince81]

Metadata App

Description:

An app that manages all kind of metadata around files and folders and all kind of other objects in ownCloud. It is important that this is implemented based on concrete use-cases and not in an abstract way.

APIs

The metadata can be accessed via different APIs

• OCP API for other ownCloud apps
• OCS API for access from mobile and desktop

Types

• Favorites - This can be used to mark files/folders/objects as favorites. Favorites are personal metadata of a user
• Comments - This can be used to add comments to files/folder/objects. This comments are visible by everyone who has access to this file/folder/object
• Tags - Tagging of files/folders/objects. This tags are only visible for the user who added the tag to the file/folder/objects
• Attributes - No visible attributes as key-values that can be attached to a file/folder/object. This can be used to store exif data for a picture, id3 data for an mp3 or other attributes.

UI

• Add a Fav/UnFav button to the files app: Files app favorites owncloud/core#2368
• Add sidebar entry to list favorite files: Files app favorites owncloud/core#2368
• Should push every Fav/UnFav action to the activity stream Activity extension for favorites #3
• Adds a column to the activity page configuration:
  “send change notification only for favorited files/folders” Notify changes for favorites files owncloud/core#12872

Version 2.0:

• Add a UI for comments and tags
• Integration into other apps are optional
• Favs, Tags, Comments, Attributes should be exposed via webdav so that desktop and mobile clients can show them

Checklist

Here's the feature checklist for metadata:

• Step 1: favorites: Files app favorites owncloud/core#2368
• Step 2: tags: Design forward: Global grouping system (categories/tags/topics/…) [$20] owncloud/core#3812
• Step 3: metadata extraction / plugin system (future)

[PVince81]

Note: it seems that core already has a "Tags" class that could be reused to store/assign the data.
This needs to be looked into: https://github.com/owncloud/core/blob/master/lib/private/tags.php
If that class can be reused, the "metadata" app will be there to implement the other non-core metadata and UI.

Challenges

1. Find out whether the current Tags class suits the requirements
2. Files app (PHP + JS side) needs to be extensible enough to accomodate for displaying tags / stars

Open questions

1. Should tags be user-only or shareable ? (public/private tags concept)
2. Should tags be assigned to file ids or file paths ?
3. Should tags survive versioning/trashbin ?

[LukasReschke]

Might be related to owncloud/core#3812?

[PVince81]

Nice, I wasn't aware of that. Thanks for the link.

CC @owncloud/designers

[LukasReschke]

Attributes - No visible attributes as key-values that can be attached to a file/folder/object. This can be used to store exif data for a picture, id3 data for an mp3 or other attributes.

I really don't get the use-case for that, can you elaborate?

[PVince81]

I guess storing the metadata like exif/id3 could be used by apps (ex: music app) when indexing files instead of providing their own metadata facilities. @MorrisJobke what do you think ?

[LukasReschke]

I guess storing the metadata like exif/id3 could be used by apps (ex: music app) when indexing files instead of providing their own metadata facilities.

And how are they updated when the files are modified? - This requires then some hooks to update them then.

[PVince81]

Yes, the app could use write hooks to update metadata.
I guess the app itself should also have some kind of plugin system to make it possible for other apps to register their own metadata types and also provide functions to scan / read the metadata from the files.

But the first version of this app should focus mostly on favorites/starring files.

[friscoMad]

Google Drive also has a metadata API that it is offered for plugins using
the rest API. As a plug-in developer for several platforms I found it
really useful and feature complete.

I our case we used it to mark a folder as the plug in folder were we stored
every file shared with our app, that allowed the user to move or rename the
folder at their wish and we could still find it and use it.

Another use case is some internal metadata we need to open the file (our
app is about ciphering) that it is inside the file and we can only recover
sending the file to a web service, currently our owncloud app need to have
a database table to cache that data.
El 17/11/2014 11:37, "Lukas Reschke" notifications@github.com escribió:

I guess storing the metadata like exif/id3 could be used by apps (ex:
music app) when indexing files instead of providing their own metadata
facilities.

And how are they updated when the files are modified? - This requires then
some hooks to update them then.

—
Reply to this email directly or view it on GitHub
#1 (comment).

[butonic]

The app should not use write hooks. Extracting metadata with eg, getid3 is time consuming and will slow down file uploads significantly. Please consider using an async mechanism as discussed in owncloud/core#12024

Also content and metadata extraction overlaps a lot with search_lucene and music. The next thing to come up might be the mobile apps asking for "images taken in New York" ... having the metadata is useless without having a query language for it. Then we start reimplementing elasticsearch ...

[karlitschek]

At the moment it is ok to have only the metadata store so that backends or 3rdparty apps can write and read metadata. Automatic extraction and search/filtering is something to look into for a later release.

[karlitschek]

@butonic unfortunately elastic search will always stay an optional app because of its complex nature to host it. I'm also sure we only need a fraction of the feature for the metadata store here

[karlitschek]

@PVince81 let's use the tags in core for now and look into that later. I doubt that we have the time to do a user interface for tags anyways for 8

[PVince81]

For OC8 my goal is mostly to add the "favorites" feature which doesn't require any "tags list" UI as described here owncloud/core#3812 (comment).

I'll create a ticket in core for the favorites to have a more specific discussion.

[PVince81]

Seems there is already a ticket for the favorites, let's use that.

[PVince81]

One important thing is to clearly distinguish tags and attributes.
As for now in OC 7 and previous versions there's already a Tags class that is here only for tags (no key/value). If we want key/value in the future we'll need another way of storing them and a separate API.

[karlitschek]

Yes. Important to clearly distinguish the different usecases: Tags are created by users and should be visible in the UI. Attributes are more invisible to users. Examples are id3 tags, exif data or any other metadata that should be just passed through from the storage backend.

[Yetangitu]

Here is a potential use case:

https://mailman.owncloud.org/pipermail/devel/2014-November/000737.html

"...While the OPDS catalog part is relatively straightforward - it could be
based on the file app, serving atom+xml instead of JSON - there is more
to OPDS than just serving links. A fully fledged OPDS catalog entry
contains a lot of metadata related to the linked publication. OC
currently does not provide much in the way of metadata storage..."

The metadata for a single publication (or 'resource' in OPDS-terms) looks like this (in OPDS-format, it should probably not be stored like this):

<entry>
  <title>Bob, Son of Bob</title>
  <id>urn:uuid:6409a00b-7bf2-405e-826c-3fdff0fd0734</id>
  <updated>2010-01-10T10:01:11Z</updated>

  <author>
    <name>Bob the Recursive</name>
    <uri>http://opds-spec.org/authors/1285</uri>
  </author>
  <dc:language>en</dc:language>
  <dc:issued>1917</dc:issued>
  <category scheme="http://www.bisg.org/standards/bisac_subject/index.html"
            term="FIC020000"
            label="FICTION / Men's Adventure"/>

  <summary type="text">The story of the son of the Bob and the gallant part he played in
    the lives of a man and a woman.</summary>
  <content type="text">The story of the son of the Bob and the gallant part
    he played in the lives of a man and a woman. Bob begins his humble life
    under the wandering eye of his senile mother, but quickly learns how to
    escape into the wilder world. Follow Bob as he uncovers his father's past
    and uses those lessons to improve the lives of others.</content>

  <link rel="http://opds-spec.org/image"     
        href="/covers/4561.lrg.png"
        type="image/png"/> 
  <link rel="http://opds-spec.org/image/thumbnail" 
        href="/covers/4561.thmb.gif"
        type="image/gif"/>

  <link rel="self"
        href="/opds-catalogs/entries/4571.complete.xml"
        type="application/atom+xml;type=entry;profile=opds-catalog"/>

  <link rel="http://opds-spec.org/acquisition" 
        href="/content/free/4561.epub"
        type="application/epub+zip"/>
  <link rel="http://opds-spec.org/acquisition" 
        href="/content/free/4561.mobi"
        type="application/x-mobipocket-ebook"/>
</entry>

As you can see this is quite an extensive collection of metadata - authorship, publisher, summary, images, related links, etc. While it would be possible to store all this in a separate table in the database, that would IMnsHO be plain wrong. A more flexible way of storing attributes would really help in getting this thing off the drawing board.

[PVince81]

Note: I've moved the checklist to the top comment for better visibility.