Export as json(?) and then load

[michalsieron]

Problem Description

I would love to be able to load previously created script and change some of the settings or just see with nice GUI what does the script do

Proposed solution

I assume, that parsing script is not an option, so next best thing would be exporting to JSON or another format.

I am thinking about putting created JSON in a comment as first/last line and then the only thing to do is to get that line and parse it as JSON.

Alternatives considered

I can read script myself, but GUI is nice, especially when showing it to non-tech people.

[undergroundwires]

Hi @michalsieron ✋🏿 Thanks for your suggestion. Just to understand clearly, do you mean that we enable loading "custom script sets" or just loading a specific selection from the existing script set in privacy.sexy?

[michalsieron]

I meant loading scripts created from already existing script set.
For example someone chooses "Disable OS Services > Text Messaging" and "Remove Bloatware > Remove OneDrive > Uninstall OneDrive". Then JSON string with those settings is saved in created script so later they can paste that script into text area on privacy.sexy and it will be loaded and displayed with existing GUI.

[undergroundwires]

Then it's more doable 😊 It solves the same thing as in #33 but using JSON import/export instead of parsing files. I think we could combine both ideas. We could save a JSON variable as batch comment in the generated script file and just parse that comment with JSON to load the scripts. Something like:

@echo off
:: https://privacy.sexy — v0.9.2 — Sun, 21 Feb 2021 13:21:44 GMT

echo Do this
echo Do that
:: Scripts here...

pause
exit /b 0

:: {version:0.9.2, selectedScripts:[.., ..., ..]}

So in as in last line we could dump the name of those scripts. So the generated file becomes more "parsable". So one could import either the generated file or just its json. Importing should take account into points in my comment in #33 also.

I'd like to keep the UI minimal and wonder how the UI should look like 🤔 Where to put the JSON export button? Where to put the import button? Where in the UI to drag and drop the file for the import?

[michalsieron]

That's exactly what I was thinking here.

I am not sure if there's any need for additional buttons. I think, that simply automatically putting JSON there would be enough. User doesn't need to know there is a "hidden" JSON data. So no need for export button.

As for importing, I was thinking of making text area editable, so user could just paste their script there.
Alternatively you could go with button (next to "Select: None | Standard | Strict | All" buttons or below text area?). Then you would display simple modal (like with privacy) and there would be new text area.
Whichever of those you choose (or any other you come up with) you can always make text area into a drag and drop target.

[kdar]

This would be very helpful, especially if more features are added. Let's say in 2020 you generated your script and you're happy with it, then 2021 comes around and added 6 more blocks. It would be nice to load my old script, and then go check off those 6 new blocks to add to my previous script.

[undergroundwires]

@kdar thanks for your feedback. Great idea. And after you giving one more heads up I decide to prioritise this in next version 🥳

It sound very good. It would require for us to do some kind of diff of list of scripts between two application versions. I’m not sure how we’d do it 🤔

Here I'll share my thoughts for the first implementation, feedback is appreciated 😊

State JSON schema

{
  "versions": {
      "app": "0.11.0",
      "serializer": "0.1.0",
   },
  "collection": "macOs",
  "selectedScripts": [
     {
         "name":"Clear Safari last session history",
         "revert": false,
         "hash":  "26bf05939cb4d297c2ef6c500a3014b9"
       },
	  /// …
  ]
}

• versions
  • app: version of the app the script is generated from, would allow to implement the idea from @kdar
  • serializer: JSON serializer, would allow backwards compatible deserialization
• collection: Current selected collection
• selectedScripts: array of selected scripts, each script with following properties
  • name: so it can be reselected
  • revert: revert toggle on/off
  • hash: md5 hash of the generated code, it would allow automatically reselecting of scripts even though name is changed with a warning. Also can show report if the name is same but script content is changed.

Report modal

After each import we can report:

• Scripts that has their code changed
• Scripts that are renamed
• Scripts that can no longer be found (when both name and hash are changed)

[kdar]

I think the JSON schema looks sound to me. One question I have is, would it be worth it to assign an ID to each script? I understand it would take some work to update the collections to have an ID. That way, the name wouldn't matter.

[undergroundwires]

I think the JSON schema looks sound to me. One question I have is, would it be worth it to assign an ID to each script? I understand it would take some work to update the collections to have an ID. That way, the name wouldn't matter.

I'd like that, it would simplify code base as well. But I'm afraid of a manual ID generation would make contributions harder.

Now it's pretty easy to open-up a collection file and just add a new script. Adding an ID field would make it more complex when we are extending it. If it's an integer, it wouldn't be ordered. If it's a GUID it would be a bit exhausting to generate a new guid and copy paste it each time for a lazy person like myself, also it would be confusing for first time contributors.

I don't know how to assign IDs and still make it easy to add new scripts 🤔

[undergroundwires]

Just a quick update. I'm working on this. Not having IDs on scripts would cause a lot more instability so I'll go for your suggestion @kdar and have IDs. Guid seems overkill, we can maybe use first part of the GUIDs. Something like 3db4d2df instead of 3db4d2df-c5e9-420c-95bd-92548af6c085.

Btw code generated as JSON became too big. So switch to custom implementation of a format that looks like:

windows;Turn off Windows Location Provider;Disable Wi-Fi sense;Hide most used apps (tracks app launch);↩️Turn off sensors;↩️Disable cloud speech recognition

Semicolon separated parts. Starts with platform (windows), then IDs (those names will be replaced by IDs later), and uses emoji ↩️ for indicating it's revert.

It saves around 50% more characters than JSON. But it's still long, so I compress and base64 encode it, which in turn reduces the output size additional 50%. In the end I prepend only the version (0.0|) for import/export module and add PGP like markers around it (BEGIN EXPORT DATA).

So in the end it looks like this:

So we'll be able to import it using the button on picture or just drag and dropping files. Hopefully it will make it to next minor version if I find the time to test it and reaching the code/documentation quality I want.

Everything is very open and transparent in privacy.sexy. I don't know if I should skip compression, so we live with bigger files but people understand how export data would look like🤔 Let me know if you have suggestion on this. Otherwise I always appreciate your feedback, and this is the reason I'm implementing it and this way. So feel free to just tell how it feels.

[kdar]

Sure, saving any amount of KBs is fine especially if it's versioned and can be changed at a later date. For the case of transparency, I don't really see a gigantic reason that the export/import section needs to be unobfuscated (especially since the whole file that actually does the work is right there for review), and it's not like it's a difficult one to decipher if someone wanted to base64 decode it, decompress, and see what the raw data actually is.

If it still bothers you a bit, there could be a textbox on the site that shows what the raw imported data was when you go to import.

Thank you for tackling this! It's such a great quality of life improvement.

[neube3]

Hello. As for the unique IDs - I don't know how exactly do you internally keep the list of functions (not in the code, but just for yourself - or are you writing new scripts right in the code?), but if it would be possible - why not go with some simple half-automated numbers (or hexadecimals if you run out of 100 in a folder)? E.g.
Category / (folder / * n) script number
becomes
Privacy Cleanup / Clear Browser History / Clear IE History / Clear IE Recent URLs
becomes
00/01/00/01 (zero-indexed) or 01/02/01/02 (one-indexed, for regular people ;)) .

It is based on current structure, but you have to start with something. If you add new script to a folder, it - never looking at the name - just needs the new number, one bigger than previous max.

You'd have to keep the numbers for removed/defunct scripts, but the only case it would be an issue would be if they were the last one added to the current folder, but I think that, if implemented smartly, it would not be a problem (in my mind it would just be list of defunct scripts to remove and never display on the tree (i.e. left, "select scripts", pane).

[undergroundwires]

@neube3, thank you for the thoughts. It's very simple structure, but as you also mentioned, it tightly couples scripts with categories, but in reality some categories are bad and tend to change, so scripts should not be coupled to them. Changing IDs later will be very hard, if possible on later stages. So we need a good design.

I've implemented PoC for import/export but could not merge and the branch got too old. I need to rework on this. I can release start on this again once IDs are on-place. I will prioritize IDs as highest priority change for next minor release.

I suggest something based on GUID, I'd appreciate your feedback and review and it would be hard to change IDs later. I created a separate issue for ID discussion #262, let's take it there.