Build RESTful API

[ctengel]

Based on #4 #9

1. I want to upload this thing I just downloaded off the internet
   • or i already did?
2. I want to discover stuff in object store
3. I want to upload stuff I found on a hard drive (noting that MC doesn't have a DB)
4. I want to get a cached version of a URL
5. I want to list files with a given metadata filters
6. I want to list a directory from a HD

POST /upload - this is a convenience function etc which basically if they know some stuff about a file it either sends them to existing or creates stuff to allow an actual upload

• mtime (from source) - datetime optional
• URL - for file on a host file:///; or mc:///; for ytdl it is the webpage - 2047 - mandatory
• direct T/F - i.e. if i do wget url will i get exactly this or do i need to finnegle this , optional assume true
• Metadata JSON
  • ytdl - partial (below) - just those things inherent to that URL or referrer
  • lm backend
  • referer info (optional) - URL_referer - ytdl URL or actual referrer or some ID of disk (i.e. HDD it was found on with mc)
    • i.e. ytdl 'webpage_url' - url of page itself - file referer
• metadata intrinsic
  • exif
• filename
• user - 16/32 - if no authentication
• ul_sw - 16/32
• ul_host -16/32
• partial i.e. this is just a partial rep of URL, etc
• Bucket name -varchar 63
• Size
• hash
• MIME - mime type 255

return -

• file - file object including object object
• upload info (or NONE if existing object)
  • path to upload to in obj store
  • path to post to to complete
• download info (or NONE if not yet)
  • easy link to obj store

POST /upload/objid

• complete=True
  (or just PUT to /object???)

POST /file (dynamic create object)
POST /object (discovered object)
GET files

search by metadata, prefix, bucket, etc
single item lookup by hash, name, etc to get real location
redirect to real location

note

• MC doesn't have long term storage, but we can put some info on the parent Tree object in file metadata
• originally we had seperate url and url_referer... however
  • this makes indexing etc complicated, two fields instead of one, so merging and adding boolean direct T/F field
  • ytdl use case URL is the 'webpage URL', direct is F... any info on actual DL can be in metadata
  • general web case, it is direct URL, direct T, any referer info in metadata
  • MC - we will likely concoct a url-style identifier for MC, or fall back on upload_host file:///; direct would be T; some MC data in metadata
  • twch - partial T direct F, see mtime, and metadata

[ctengel]

• bytes to text needs to be more elegant for the hash
• object mapping how does it do it?

[ctengel]

https://flask-restx.readthedocs.io/en/latest/marshalling.html#custom-fields is a big help