Project Notes (timeline)

[dlucian]

Whenever I get a chance to work on this, I'm writing down notes, or making small experiments. Since I've adopted the "build in public" mindset, and to maybe spark a conversation with the contributors, I'm going to post my notes in this thread.

[dlucian]

2023-05-16

• fswatch -r -x /path
  • deleting a file from Finder shows up as Renamed
  • deleting a folder from Finder shows up as Renamed and no contained files are mentioned
  • deleting a file or folder from command line (rm) shows all deleted files, folders and contained files as Removed
  • copying a folder shows all the files as Created AttributeModified
  • modifying the xattrs of a file shows as AttributeModified
• considering all the above, we cannot keep the index in sync with the filesystem by relying on the xattrs only, because deleting a folder only triggers a Renamed event on the folder and not on the contained files; this happens because the folder is actually moved into Recycle Bin
• the metadata can still be attached via xattrs, but we will still need a DB table listing all the files in all the folders so we can determine which files have been moved when a folder is moved;
• if we need a table with all the files, mind as well use it both for metadata, indexing and everything else; can be a single table with all the data, or multiple tables (filesystem table + metadata + index + embeddings etc.)

fswatch -r -x /Volumes/vault
/Volumes/vault/AND08journal.pdf IsFile Renamed
/Volumes/vault/fits-1.5.1.zip IsFile Renamed
/Volumes/vault/libmediainfo-23.04-x86_64.Arch_Extra.pkg.tar.zst IsFile Renamed
/Volumes/vault/.DS_Store AttributeModified IsFile Updated
/Volumes/vault/test_files IsDir Renamed
/Volumes/vault/.DS_Store AttributeModified IsFile Updated
/Volumes/vault/zettelkasten_v3.alfredworkflow IsFile Removed
/Volumes/vault/ocr2text Created AttributeModified IsDir AttributeModified
/Volumes/vault/ocr2text/.DS_Store IsFile
/Volumes/vault/ocr2text/.DS_Store OwnerModified Created PlatformSpecific AttributeModified IsFile AttributeModified
/Volumes/vault/ocr2text/LICENSE IsFile

2023-05-15

• exiftool has a recursive option, which works quite fast (4m35s for 134k files, see below).
• using RoadRunner might be a good idea to make processing as efficient as possible. It supports queues, docker, HTTP, Centrifugo, and seems simple enough.
• might need a simple monitoring docker container for Grafana or other performance metrics that we want to keep an eye on.
• vid2cleantxt is a very nice library to transcribe video and it has some very nice suggestions such as texthero for cleaning up the text afterwards. We have simpler alternatives as well.
• on docker composer up, postgres should init with the default DB and tables, a metadata container should randomly check X files if they have metadata and if not launch a complete metadata building.
• we should use a prefix for xattr metadata, such as pvp- to avoid conflicts. we should also store the checksum and version of the metadata, so updates in the future can trigger re-generation of metadata.

90502 directories scanned
134427 image files read
./exiftool -r -j ~/projects/  218.24s user 17.91s system 85% cpu 4:35.78 total

2023-05-14

• Neither FITS nor Apache Tika seem to do a very good job at extracting OCR from images and PDFs, they are very (Java software) slow and FITS isn't even returning the extracted OCR in the output XML.
• I am considering using Tesseract in a "manual" manner to extract text from images.
• For extracting OCR from PDFs, I can use some simple script like ocr2text.
• Also started to play with processing large sets of files, might be a good idea to use PHP for that¹.
• Thinking about creating a small PHP script that calculates checksums for files²

Footnotes

1. [[Parse large number of files with PHP]] ↩

2. [[Calculate file checksum in PHP]] ↩

[dlucian]

2023-06-07

• Could potentially add notes to a file, by hash. That means wherever that file ends up in and how it is organized, duplicated, copied, moved, it's still the same hash, so the same content, so the notes apply.
  Sample hash: 652f031c2aab46ef5e5c9d6f09e2e684f87a8ebf6736d7a343b35cba75cf1e96 (not the shortest, but the most unique)
• I need context.
  (Perhaps) a table that says which files are originals, because I would want to see original files in my timeline/map, but not randomly created files. For example I am storing a huge library of wallpapers, not really want to see those when I'm looking back on the same days in previous years, on in the timeline. I want to see my content. So perhaps it's "personal content" instead of "original content"? I thought about the "original content" because the first copy is mine, everyone else (might) have a copy of my file, but I have the original one.
• Address from GPS location. One thing the analyzer should do is convert a lat/lon to an actual place. Can be done via Google Maps API for example. This being an external API, might be subject to rate-limiting so the Analyzer should take this into consideration. Other APIs might be used for other purposes, so let's keep this in mind.
• Google Takeout exports emails in one big .mbox file. There are some scripts that convert it into distinct .eml files. This can be useful to blend an Google Takeout export with a continuous download/sync of emails.

[dlucian]

2023-06-10

• C2PA - https://c2pa.org
  • An open technical standard providing publishers, creators, and consumers the ability to trace the origin of different types of media.
  • https://opensource.contentauthenticity.org has a CLI tool for creating provenance
  • This falls right into my "context" problem, meaning when I do a search, I'd want to find my content before the random favicon.ico files.
• Originally I was thinking that the more complicated metadata (such as the one that Tensorflow or Google Maps API reverse geolocation) should go into a separate table, but now I'm not so sure anymore because the whole point of indexing the files is to do searches and searches with JOINS across big tables are going to be a big problem. I could split the tables later if needed. Also I'm considering that a lot of files will not be media.

[dlucian]

2023-07-17

• The Scan task should only ensure the database is up to date with the FS
• The Analyze task should run all the tools to analyze the data.
• Might be able to use spaCy to determine language.
• Need to find a way to add tools to the project.

2023-07-05

• We can use The Elements from Dublin Core to standardize the metadata.
• The Type should be an DCMI Type Vocabulary item.
• To determine the language of a document, we can use Polyglot (has good documentation).
• For places, we can use something standard as The Getty Thesaurus of Geographic Names (see Timisoara, has a web serviceand there are some libraries).
• Tools - composability
  • for example: exiftool, poliglot, ocrmypdf, pdftotext, embeddings, context (custom tool), etc.
  • these have requirements, and populate various information in the metadata system (maybe even other systems, like embeddings, full-text search etc.)

{
	"tools": {
		"exiftool": {
			"requires": [],
			"provides": ["date","format","coverage"]
		},
		"tesseract_ocr": {
			"requires": [
				"format:image/jpeg",
				"format:image/png",
				"format:image/gif"
			],
			"provides": ["description"]
		},
		"ocrmypdf": {
			"requires": ["format:application/pdf"],
			"provides": []
		},
		"pdftotext": {
			"tool": "pdftotext",
			"requires": ["format:application/pdf"],
			"provides": ["description"]
		},
		"pdftotext_postprocess": {
		    "tool": "pdftotext",
			"requires": ["format:application/pdf", "*ocrmypdf"],
			"provides": ["description"]
		},
		"context": {
			"requires": [],
			"provides": [
				"creator",
				"type",
				"contributor",
				"coverage",
				"publisher",
				"source"
			]
		},
		"garmin_connect": {
		    "requires": ["format:application/json","type","source"],
		    "provides": ["subject","title","description","%image"]
		},
		"poliglot": {
			"requires": ["title","description"],
			"provides": ["language"]
		},
		"video2text": {
			"requires": ["format:video/mp4"],
			"provides": ["subject"]
		},
		"video-transcription": {
			"requires": ["format:video/mp4"],
			"provides": ["description"]
		},
		"video-ocr": {
			"requires": ["format:video/mp4"],
			"provides": ["description"]
		}
	}
}

This way, tools that aren't dependent on others can run first, and then once we have the information, we can run those that are dependent.

Tools that don't provide anything are usually tools that are being used as input to another tool (ex. ocrmypdf -> pdftotext).

Once a document has been scanned, the analyze task would determine the order of the tools needed to be run, and would run each of them. Some tools can only run on certain file types, or if specific information is available. This should also be configurable.

The purpose is to add as much metadata as possible, in order to be able to make connections between documents.

2023-06-21

• Frank James hinted that it might be an idea to use Dublin Core and other RDF notation to describe what is connected to what. Not sure if AI cannot do something to aid that.

2023-06-14

• The OCR recognition via Tesseract should be in its own Docker container, but I don't think I have the time to properly do that now, so I will integrate it into the roadrunner container for now.
  • Will use it for images
  • Will use it for PDFs as well. To extract all text from a PDF, whether generated from OCR or otherwise, we might use a program like Poppler’s pdftotext or pdfgrep

2023-06-13

• One part of the system that I have yet to think about is pulling in data from external services. A few examples can be: Garmin Connect, Todoist and emails. These, although they can be in their proprietary format, might be converted to (1) plaintext for search indexing and (2) view mode to see them in a gallery of some sort.

2023-06-12

• More naming ideas:
  • words in other languages
    • Kukua = to grow / growth (swahili)
    • Kabla = before (swahili)
    • Juzi = the day before yesterday (swahili)