Enhancement: provide scripts in a machine-readable way

[brootware]

Problem description

Hello, thank you for the amazing tool. I've been using these scripts on my devices and sandbox machines with a cli-app version I made with standard recommendations. Currently I'm manually looking at the scripts and putting them into functions.

As you can tell, this is quite manual, error prone and time intensive.

Proposed solution

I'm just wondering if there is some for of API endpoint I could query to update the scripts.

Alternatives considered

Manually copy pasta scripts.

Additional information

https://github.com/brootware/privacy-sexy-lite

[femdiya]

@brootware make binaries pls

[brootware]

Hi @bardiarajabi , any reason for considering binaries ?

[femdiya]

@brootware Its hard for some people for compiling on windows.
I Personally have no problem without binaries.

[brootware]

oh there are no compilation process involved though @bardiarajabi . The CLI is just a simple power-shell script on windows that wraps the batch commands from privacy.sexy web app. The user can just invoke the cli just like this without compiling or building.

.\enforce_windows.ps1 help

You can check it out the project here. https://github.com/brootware/privacy-sexy-lite

[femdiya]

@brootware Oh yes true.
but posting it on issue tab isnt a good idea you can just e-mail the creator @undergroundwires

[undergroundwires]

Hi @brootware, what a great initiative! This is very useful, I'd want to use it myself. I'm ready to work with you to get rid of the manual work. This is important before gaining recognition because it would allow us to keep scripts updated and therefore as secure as possible.

How about renaming this issue to something like "provide scripts in a machine-readable way?

The challenge here is that building a REST API would require a back-end. But privacy.sexy is completely serverless. This is to not pay more money for keeping a back-end. The way privacy.sexy GUI works is that it parses these collection files on client and compiles them to generate the final scripts. Everything is done the client-side. You can read more about compilation here.

To provide you the compiled scripts we need to compile the scripts independently of the UI. I have some ideas around this but they all require mass-refactoring that I cannot complete in a short time. I will think more about this and come back.

P.S.: @bardiarajabi I also think binary distributions in future iterations would be pretty nice, especially on Windows. But this we should probably discuss in that repository.

[brootware]

Hi @brootware, what a great initiative! This is very useful, I'd want to use it myself. I'm ready to work with you to get rid of the manual work. This is important before gaining recognition because it would allow us to keep scripts updated and therefore as secure as possible.

How about renaming this issue to something like "provide scripts in a machine-readable way?

The challenge here is that building a REST API would require a back-end. But privacy.sexy is completely serverless. This is to not pay more money for keeping a back-end. The way privacy.sexy GUI works is that it parses these collection files on client and compiles them to generate the final scripts. Everything is done the client-side. You can read more about compilation here.

To provide you the compiled scripts we need to compile the scripts independently of the UI. I have some ideas around this but they all require mass-refactoring that I cannot complete in a short time. I will think more about this and come back.

P.S.: @bardiarajabi I also think binary distributions in future iterations would be pretty nice, especially on Windows. But this we should probably discuss in that repository.

Thank you @undergroundwires . Definitely agreed with you on keeping privacy.sexy server less to keep the costs to a minimum. Will be looking through the compilation process of the scripts and brainstorm some ideas around this too and come back.

[undergroundwires]

I think the easiest way would be to create a JSON file (application.json) with all necessary information that you can parse further. To do this, we need to move script compilation from client machine to compile-time. It would include all compiled scripts for every platform.

Implementation

Creation of application.json can be handled as a webpack step as part of compilation. Webpack way would integrate well with overall build process and provide Vue CLI advantages such as hot reloading when executing npm run serve. I did not work with extending webpack compilation on this level before. But I guess we could create a custom webpack plugin that would do create application.json.

The challenge with this is that Webpack plugins only support JavaScript meanwhile script parsing/compilation is written in TypeScript. We need to expose this TypeScript code to JavaScript (of Webpack) but we cannot do simple import from TypeScript to JavaScript. We need to first compile this TypeScript code and then we can import it or call it like CLI.

Changes needed to achieve this:

1. Create data-compiler. Move application data creation logic to: src/application/data-compiler/... It would read necessary data and return application.json.
2. Extend webpack to:
   • Compile and run src/application/data-compiler,
   • Save its JSON result as /application.json as static asset,
   • Inject its JSON object to runtime of GUI.
3. Change Vue GUI to use injected JSON instead of compiling from scratch.

This pre-compilation would make initial loading of GUI faster and give you two options:

1. Clone privacy.sexy with a specific version e.g. 0.11.0, compile and run data-compiler yourself and generate version-specific application.json.
   • Pros: Control which versions of scripts you have.
   • Cons: More complex, needs knowledge of internals of privacy.sexy. Harder to maintain for you. Needs frequent updates to match latest version.
2. Call https://privacy.sexy/application.json, parse it and use as JSON.
   • Pros: Easy, abstracts internals of how the file is generated. You always get the latest (which is desirable).
   • Cons: Potential issues if the data scheme changes.

A trade-off between more control vs simplicity.

Here I tried to visualize it:

Looking forward to your feedback.

[brootware]

Thank you for the detailed explanation of the implementation @undergroundwires . I really appreciate how it's mimicking an endpoint to be called by clients.

As I'm a bit rusty with web frameworks please bear with me as I try to recap this.

If I understand correctly, data-compiler will be a custom webpack plugin class that needs to be written to achieve the 2 options you have provided.

From a quick glance at the two options, No.2 sounds like a better approach personally as I'd like to make sure all the scripts are in sync with the web app. A few questions on this though.

Call https://privacy.sexy/application.json, parse it and use as JSON.
Pros: Easy, abstracts internals of how the file is generated. You always get the latest (which is desirable).
Cons: Potential issues if the data scheme changes.

• What counts as a data scheme change? Referring to the collections-files.md for example, if we were to add a category or a child of a category, does that count as a change? Or is the change at the object level like collection and category
• Would there be any data scheme changes in the foreseeable future, how often would it be?

From the visualization above, I'm assuming there will be refactoring involved in Application.ts?

[awhitepl]

I agree with underwires but specifically recommend also releasing the app on a reliable registry with your published images on say https://artifacthub.io/ and they can install the software to run wherever they like with with "docker/podman run -it imagename" and pass in any arguments via CLI flags making the app machine readable that was requested above in one go.

[undergroundwires]

Update: I work actively on this and very close to remove all the blockers. My goal is to release this on 0.13.0 empowering dependent projects with design we discussed here. If you plan to privacy-sexy-lite actively maintain I will add shout-out to related channels.

[brootware]

Hi @undergroundwires , sure I'm looking to refactor some parts of the scripts and test out which I will only be able to do it after I could get the machine readable endpoint. Until then please hold off on the shout-out first :)