Add absolute path of the .csv in done.json file for Notion 'output profile'

[IAmVigneswaran]

In the current build, only the file name of the of the .csv is printed in the csv file.

In the next build, the absolute path of the .csv should be printed in the csv file.

[orchetect]

This is an easy fix.

While we are at it, I would suggest we make the done file a format that is easily parsable in case we want to add more information in it? It could be a JSON file. It's the most ubiquitous nested keyed data serialization format so pretty much anything can read it.

A few compelling reasons:

• Future-proofs clients that use the CLI in the event new data is added:
  • If the CLI gets additional export formats other than CSV, they may have additional metadata that might be wanted
  • Diagnostics or other output information could be included

[IAmVigneswaran]

I am open to other formats like JSON. But I would want .csv as the default output format.

The main reason is to utilise csv2notion. And CommandPost has already implemented csv2notion binary into the application. Furthermore, users of Airtable can easily utilise its native CSV importer.

You can add other format if you like for future proofing with other tools (Other than CommandPost). But, .csv format will always be default.

[orchetect]

I assume your scripts are reading the contents of the done file and feeding the csv path within it into csv2notion?

Parsing JSON through bash / command shell natively in macOS is actually not super intuitive but it's possible. Looking into some solutions.

[latenitefilms]

FWIW - we can easily work with JSON in CommandPost.

You can see how the "Shot Data" Toolbox works here:

https://github.com/CommandPost/CommandPost/blob/develop/src/plugins/finalcutpro/toolbox/shotdata/init.lua

We can use this Lua Extension for reading/writing JSON:

http://www.hammerspoon.org/docs/hs.json.html

[IAmVigneswaran]

@latenitefilms But currently csv2notion only supports .csv? For Shot Data are you converting from JSON to CSV?

I assume your scripts are reading the contents of the done file and feeding the csv path within it into csv2notion?

@latenitefilm For Marker Data Toolbox, that is the idea and workflow. But JSON could be added for any future workflows other toolbox?

@orchetect

I am extremely open to adding other formats like .JSON and .tsv. :) As mentioned. .csv should be the default for now.

An idea - This could add further complexities. What if there were different "Profiles" for output formats for different database platforms, NLEs and DAWs. For each profiles, the output data extension and structure would formatted in a different way.

Example:
For Notion - the current .csv format and structure.
For ProTools - it could be in a text file that ProTool could import.
For Resolve - it could be in a EDL file.

[orchetect]

What if there were different "Profiles" for output formats for different database platforms, NLEs and DAWs. For each profiles, the output data extension and structure would formatted in a different way.

That's precisely the idea I had in mind, and many of the codebase refactors done so far are in preparation to make this possible.

[IAmVigneswaran]

Let's continues the discussion on other formats here.

[orchetect]

The contents of the done file may still remain a stable format (ie: JSON). But its contents can vary depending on the "output profile".

What we want for the time being is for the "CSV to Notion" profile (aka, the only "profile" currently) to output the absolute path to the done file.

[IAmVigneswaran]

The main 2 purpose of done.txt file is to indicate the user that the export processing has successfully completed. And the absolute path of the .csv will be printed into the text file, for other applications or workflows to continue further. Right now we only have .csv.

If we have other output profiles, like .JSON, .EDL or .TSV and such, its path will be printed in the done file.

Are you suggesting that we use done.json instead of done.txt (plain text)?

[orchetect]

Are you suggesting that we use done.json instead of done.txt (plain text)?

Yes - To be clear, we're only talking about the done file in this thread. I'm suggesting its contents be JSON to make it more versatile and future-proof to additional features in the future. We may want to include more pieces of data other that just the path to a single file. Perhaps some output profile needs to know a few things. As a programmer, I always hate operating on narrow-minded spec that is hard to scale later.

In the mean time we can just leave it as a text file that contains the absolute path to the csv file.

[IAmVigneswaran]

Yes - To be clear, we're only talking about the done file in this thread. I'm suggesting its contents be JSON to make it more versatile and future-proof to additional features in the future. We may want to include more pieces of data other that just the path to a single file. Perhaps some output profile needs to know a few things. As a programmer, I always hate operating on narrow-minded spec that is hard to scale later.

Noted!

In the mean time we can just leave it as a text file that contains the absolute path to the csv file.

While we are at it, we can make it JSON as per your suggestion.

[orchetect]

Ok this is done - pushed to main.