Roadmap

[dlucian]

This is an ambitions project. And engineering might not be the biggest challenge here. There are plenty of apps and services in the world. What I feel is of utmost importance is that we build the right thing. And that we build it right.

So a roadmap would look like something along the lines:

• APR, MAY, JUN 2023: discussions, sketches, idea generation, schematics, research (a.k.a. the drawing board)
• JUL, AUG, SEP 2023: build the simplest prototype ever (a.k.a. the silver bullet)
• OCT, NOV, DEC 2023: TBD

[michaeljfarr]

Hi Lucian. I've read your posts here, and we probably share some similarities in our ideas. I'll share some nebulous thoughts here, perhaps these can be re-organised into more tangible discussion areas.

There is a chance that the complexity of this topic would significantly exceed what we can exchange in written format within a reasonable timeframe. I have tried to explain some of this with my https://file-puddler.blogspot.com/ site, but even with that space, it is difficult to explain the engineering challenge space in a reliable fashion. With that in mind, perhaps we can aim to collaborate in workshops which might help us refine our shared understanding of the challenges and potential solutions.

Here are some things that I think might be true

• The ability to search data in situ is better than the ability to search it after it has been archived.
• Knowledge about the relationship between data elements is as good and sometimes better than a detailed understanding of specific items.
• Keeping things forever is great, but understanding the lifecycle of items and supporting that lifecycle is better.
• 'Typical' people dont like to pay for things they do not understand and cannot control.

Because of those things, I focus on solutions that:

• Focus on indexing data. So, include just the metadata on every file on every device and update that information automatically.
• Inspect files in situ and adapt and use existing archives as part of the solution.
• Special features of the archive (such as sharing, replication, websites etc) are used directly rather than trying to re-do them.
• Use metadata as an element to control redundancy and encryption (ie encrypt sensitive stuff with a different key etc)
• Avoid reliance on dedicated servers; instead, stream metadata into cloud drives so that new app instances can be quickly established from it. Done well this can get operational costs down to zero.
• Be smart about which files to inspect internally and to what degree. Use contextual information (ie is it in github already, who produced it etc)
• Support an amount of human workflow to tag files, but as a learning tool for AI. Allow the operator to take charge of where their data goes, but relieve them of that burden at any moment they chose.
• Understand the file hierarchy and/or other contextual information about the data item

Obviously, I have some raw solutions to these problems already. I've also found some specific situations where this could help small businesses.

[michaeljfarr]

I'd guess our scope is quite different, but the community of individuals engaged in archival systems is small enough to warrant another connection attempt. I'll provide you with a brief update on my progress and outline the areas where the approaches differ. If you believe there is potential for collaboration, please let me know.

I have successfully implemented an indexer capable of handling incremental updates also featuring an adaptable classification system. Each indexer run can gather various data segments, such as hashes, internal metadata, file system extension metadata, thumbnails, and other exports. I have incorporated initial UI features and continue to add new ones every few weeks. Currently, my focus is on developing a compound query system that will serve as the foundation for other archival rules, including encryption and redundancy.

Your system seems to resemble an open-source version of Google Photos, but with expanded support for diverse media types and metadata, and the advantage of being self-hosted. This would be great, as Google Photos' limitations make it impractical for many complex tasks and doesn't offer any extensibility capability.

In contrast, my system functions as an archive aggregator, designed to track the whereabouts of items and facilitate their movement across multiple archives. The aggregator concept allows users to combine free systems like OneDrive and Google Drive with cost-effective archives like B2, as well as self-hosting solutions.

Our respective systems are notably distinct and likely employ different technologies. Nevertheless, I anticipate that we both encounter similar maintenance challenges related to data extraction and indexing. There is potential for collaboration in areas such as metadata formats, metadata extraction techniques, classification ontologies formats and APIs, and strategies for ensuring data remains up to date.

For your reference, my UI is developed using React as a CRA (Create React App), with multi-platform hosting targeted through Tauri. Some screenshots of these features are available on the site linked above.

[dlucian]

That's wonderful news! You are making way more progress than I've been able to in the past weeks. You might want to write about your project on https://news.ycombinator.com/item?id=35729232, I have seen some very interesting things people built there.

Indeed there are and will be discrepancies between the systems that we're building and it does help to point them out.

I might have given the photos example too much, and made people think it's an open-source version of Google Photos, but this project is very far from that. Actually, I want to use PhotoPrism on the same device that's storing the photos, and one of my requirement is that it should work with a user's existing software, in harmony, so that it brings extra value to one's life without making them give up something.

I'm also really glad you're making progress with yours, I love to see driven people going out of their way to build amazing things and I do believe there are plenty of use-cases and users for everyone.

Our entire work will be open-source and public, so feel free to get inspired for your project and, of course, any contribution is greatly appreciated. I am also sure we'll have overlapping subjects on which we can collaborate.

Currently, I'm playing with different tools to get the foundation of the service strong. Looking into a combination of docker+roadrunner+postgresql as the foundation tools. I think the timeline I originally posted is a bit loose and I am aiming for a working prototype in July/August.