Add middleware layer to support HTTP/OSC/Sockets/...

[benjaminbojko]

No description provided.

[pingevt]

So a couple of thoughts:

• Can we call this the "Launchpad Server"?
• Should this be a part of the launchpad project? (currently - yes, but not sure if we need more discussion around this)

Authentication

I think we should have some type of Authentication. Basic user/pass seems fine in my opinion. Also, utilizing Json Web tokens. Authentication should also be optional, but highly encouraged. And to note, not sure how to add authentication to OSC. Simply using well-defined or crazy addresses might help. Need to look into this more.

packages:

• bcryptjs
• jsonwebtoken

Authentication CLI

With that, I think we need a layer of User CRUD and Authentication.

• CLI tool to add, remove, list and change password of users.

Main Functions of the server:

Status
get current status of launchpad, I'm thinking

• apps running / not running
• last content download
• recent (5) log messages

startApps

stopApps

downloadContent

fullCycle
stop apps, download content, and restart apps

Suggested config at the moment...

"server": {
    "enabled": true,
    "port": 7676,
    "useAuth": true, // applies to all apis, but exposes the POST /login request.
    "httpApi": {
      "enabled": false // enable/disable API
    },
    "wsApi": {
      "enabled": true // enable/disable API
    },
    "oscApi": {
      "enabled": false, // enable/disable API
      "port": 7675,
      "addresses": [] // ability customize addresses
    }
  }

Current file structure from proof of concept:

index.js
/lib
+-- serverCore.js
+-- server-options.js
+-- authentication.js
+-- httpApi.js
+-- wsApi.js
+-- serverTransport.js (should rename to reflect that it is tied to the wsApi)
+-- oscApi.js

so the idea is that the server Core interacts directly with Launched. Each Api, interacts with the core server rather than LP itself. That way each API does the same thing and for example, "downloadContent" doesnt act differently in each API.

---

other thoughts and questions??

[benjaminbojko]

This is great, thanks for putting so much thought toward this. A few initial reactions below:

So a couple of thoughts:

• Can we call this the "Launchpad Server"?

Makes sense to me. The package name should be launchpad-server, and it should live in packages/server. The actual web client should be a separate package, which currently already exists as a blank package in packages/dashboard.

• Should this be a part of the launchpad project? (currently - yes, but not sure if we need more discussion around this)

Are you referring to the launchpad monorepo or the Github project for tasks/issues? Either way, my answer would be yes for both unless you have concerns.

Authentication

I think we should have some type of Authentication. Basic user/pass seems fine in my opinion. Also, utilizing Json Web tokens. Authentication should also be optional, but highly encouraged. And to note, not sure how to add authentication to OSC. Simply using well-defined or crazy addresses might help. Need to look into this more.

packages:

• bcryptjs
• jsonwebtoken

All of this makes sense to me. Eventually it might be nice to actually do that using a web-UI (similar to how VS code and other apps will open a browser link and then return back to the app using its own protocol like vscode://...). That can come later though and I think the bottom line of not storing credentials in a plain text file would be great. That being said, let's think about how to make that CLI API simple and easy to remember, and how to store the local database.

Do you have any ideas for that? How do node apps typically do that on Windows for credentials?

FWIW I've also been thinking of switching to a proper HTTP cache library to wrap into our MediaDownloader, which would actually benefit from some sort of local store itself, so let's keep that kind of cross-package usability in mind.

Authentication CLI

With that, I think we need a layer of User CRUD and Authentication.

• CLI tool to add, remove, list and change password of users.

Main Functions of the server:

Status get current status of launchpad, I'm thinking

This is great and lines up with #10. Could you post a draft JSON for what that might look like? Could you keep the naming consistent with all the existing commands, though?

• apps running / not running
• last content download
• recent (5) log messages

Could the # of logs be configurable in the launchpad config? I think 10-20 might be a better default here.

For the following commands, I think all of those should still go through the CommandCenter class, so the leaner and the more the server can just be a translation layer between protocols and commands, the better imho. As we use that class more, it might be good to also think about how to smartly centralize all the available commands. Currently those are just in the core index.js.

We should also plan for a heartbeat command that accepts an app id and possibly other metadata. Ideally this could be implemented across all protocols, but we can start with WebSockets.

startApps
stopApps
downloadContent
fullCycle stop apps, download content, and restart apps

A full cycle is what download content should probably do by default (and did in the legacy version), since we wouldn't want to change any files while the apps are running (could lead to locked files or apps crashing). It might still be good to have a nuclear restart-launchpad action, though.

Suggested config at the moment...

"server": {
    "enabled": true,
    "port": 7676,
    "useAuth": true, // applies to all apis, but exposes the POST /login request.
    "httpApi": {
      "enabled": false // enable/disable API
    },
    "wsApi": {
      "enabled": true // enable/disable API
    },
    "oscApi": {
      "enabled": false, // enable/disable API
      "port": 7675,
      "addresses": [] // ability customize addresses
    }
  }

A few thoughts on the config:

• I think each protocol would need its own port, right?
• It might be good to plan ahead and make auth its own object, jic we need to add more properties to that later
• Let's remove the API label from each individual entry and nest them all into a single object instead
• I'm leaning towards renaming API to either protocol or transport. What do you think works best, @pingevt ?
• Where possible, let's use the established short or long form for each protocol, so http, websockets and osc. AFAIK ws is a specific node library implementation of WebSockets

Here's a revised config:

"server": {
	"enabled": true,
	"auth": {
		"enabled": true
	},
	"protocols": {
		"http:": {
			"enabled": true,
			"port": 8080
		},
		"websockets": {
			"enabled": true,
			"port": 3000
		}
	}
}

Current file structure from proof of concept:

index.js /lib +-- serverCore.js +-- server-options.js +-- authentication.js +-- httpApi.js +-- wsApi.js +-- serverTransport.js (should rename to reflect that it is tied to the wsApi) +-- oscApi.js

Could you stick to the existing kebab-case naming convention for files & folders? I also think all the protocols or transports should be in a single folder (e.g. lib/transports) and serverCore should be launchpad-server for consistency. Let's mirror how the content package does that.

so the idea is that the server Core interacts directly with Launched. Each Api, interacts with the core server rather than LP itself. That way each API does the same thing and for example, "downloadContent" doesnt act differently in each API.

This makes sense to me. It's important that these all just feed directly to the CommandCenter class because that serializes each command (e.g. don't update content before the apps have actually stopped running). Ideally the Server class would be fairly lightweight and just connect and launch protocols in an elegant way. You can check out the core Launchpad class for reference. We may have to expose the CommandCenter `nstance in core to the server (e.g. as a constructor parameter).

One thing to consider: How can the server tie into Launchpad's startup/shutdown sequences? The core Launchpad class may just need to tell the Server instance to startup and shutdown accordingly in these functions.

other thoughts and questions??

Sorry for the many scattered thoughts and thanks again for digging into this!

[pingevt]

Ben thanks! this is exactly the feedback I needed. Most things are a yes/agree, but heres a few responses:

Are you referring to the launchpad monorepo or the Github project for tasks/issues? Either way, my answer would be yes for both unless you have concerns.

I'm referring to the monorepo. At first, in my head the server was a wrapper to Launchpad, and was more optional. As far as adding it to the code base now, I agree. Now, I'm a little more unsure how it is integrated. Should it be run, based on config, when you just run npx launchpad? Or a completely separate command.

Do you have any ideas for that? How do node apps typically do that on Windows for credentials?

I'm not entirely sure other than mongodb. That seems a little heavy handed for what we might need. I'm thinking we have a handful of users and a few other things. For the proof of concept, I am just encrypting the password, then saving it to a json file. We may be able to entirely encrypt the file.

I was going to ask in another ticket, how we should save the "last download time". So the idea of having some type of local store makes sense.

Could you keep the naming consistent with all the existing commands, though?

Yea - of course, I've been making them inconsistent because I've had questions... Also, one of the things I'm curious about for content, What about Live updates?

For the following commands, I think all of those should still go through the CommandCenter class...

Totally agree, but I'd love for you to show me exactly what your doing. Not sure I understand whats going on with the command class.

• I think each protocol would need its own port, right?

Not necessarily, websockets can use the http server. I think this makes sense, rather than 2 servers running. OSC though seems to need its own port.

• I'm leaning towards renaming API to either protocol or transport. What do you think works best, @pingevt ?

For what its worth, Winston names its logger plugins "transports". But it think Protocols. makes sense.

• Where possible, let's use the established short or long form for each protocol, so http, websockets and osc. AFAIK ws is a specific node library implementation of WebSockets

As always, I struggle with naming... good to get your feedback on it all.

One thing to consider: How can the server tie into Launchpad's startup/shutdown sequences? The core Launchpad class may just need to tell the Server instance to startup and shutdown accordingly in these functions.

I think this might be good to have an in person discussion. I'd love to hear your thoughts on how you built out core. Diving through the code, was intense.

Also, do we have the coding standard written down somewhere?

All this is great to start adapting the proof of concept though and a few investigations

[pingevt]

https://www.npmjs.com/package/lowdb - seems to be a simple solution, but it is simply writing a json file.

[benjaminbojko]

Should it be run, based on config, when you just run npx launchpad?

That's what I had hoped for. If we can keep it lightweight enough, it feels like it's going to be a central part to launchpad moving forward. We can leave plugins/extensions in the future to add more functionality (#15 ).

How to store credentials

Yeah I go back and forth too. I've seen sqlite used a lot for this kind of thing too. I'd say a few parameters to look for in options would be:

1. No plain text/unencrypted passwords
2. Doesn't require a huge load of npm dependencies
3. Established file format/protocol (mongodb, sqlite, json)
4. Works well with other libraries (e.g. Implement proper caching library that follows HTTP standards and stores asset age/etags #34)
5. Nice-to-have: human-readable file (either via plain text or a GUI)

Anything else that comes to mind?

What about Live updates?

Let's cross that bridge later. I think this might have to depend on each use-case, or we need some sort of "don't pull the rug from under our feet" option when updating :) My instinct says that this also requires a decent amount of front-end architecting for whatever app we're running, so I'm not sure if it would have to go through launchpad (i.e. app talks directly to CMS).

whats going on with the command class

For sure, let's chat tomorrow. TLDR is it's a central pub system (without the sub component as of yet), which serializes event execution and allows for pre/post-execution hooks to be added (e.g. post-content-update can be used to rebuild your static web app).

Ports

Ah interesting regarding web sockets. Just thinking of portability, maybe let's plan for each protocol allowing for its own port. Maybe some protocols, like WebSockets, can default to the HTTP server port.

Naming: API vs Transports vs Protocols

Ah I'm going back and forth on Transports vs Protocols. Per Wikipedia:

In computer networking, the transport layer is a conceptual division of methods in the layered architecture of protocols in the network stack in the Internet protocol suite and the OSI model. The protocols of this layer provide end-to-end communication services for applications

I was originally sure on Protocols, but now leaning towards Transports. What do you think?

[pingevt]

All sounds good...

Agreed with your points on the DB, I think my question is do we want to go with something as robust as an actual database like mongoDB? or something very lightweight. The lowdb package I mentioned, I'm pretty sure is just read/write json file(s). We get no perks of a real Database. The stuff I'm seeing at the moment, that seems fine, but do you think we'll need something more robust in the future? if not I'd lean towards lowdb.

No plain text/unencrypted passwords

Pretty sure this is up to us, no matter what we use. But I've been using bcryptjs so far.

Naming: API vs Transports vs Protocols

Ok - now I'm nerd-ing out a bit because I use these words and really have no clue what they mean. If I get it correctly, they are are all protocols... Cause thats really the most generic term. But they are part of the Transport layer. So they are Transport Protocols, right? so "transport" seems more defined.

Now this is also making me question the name httpApi. Is that an accurate name?

A couple of other links:

(https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview#apis_based_on_http)
(https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade)
(https://datatracker.ietf.org/doc/id/draft-ietf-hybi-thewebsocketprotocol-09.html#rfc.section.1.7)
(https://developer.mozilla.org/en-US/docs/Glossary/Protocol)

[benjaminbojko]

bcryptjs sounds great! So we'd just be storing the hash/salt and not the password?

Let's go with Transports! As you said, that's what Winston uses too and it feels applicable here since we're adding various inputs/outputs for existing data/signals.

[benjaminbojko]

Oh and for now let's go with one of the json file options for storing session info. Any thoughts on what that file could be named? What are people using out there?

[pingevt]

So we'd just be storing the hash/salt and not the password?

Yes, in the proof of concept the salt or Token Key is saved in a .env file... So, we might want to look at that. We might be able to add that to the credentials file if we want to reduce dependencies.

Transports it is!!

Oh and for now let's go with one of the json file options for storing session info. Any thoughts on what that file could be named? What are people using out there?

I'm not sure how lowdb handles it... so we may not need to worry about it until we start working on it. We probably name the Database, but I don't know where/how it stores the files.

I'm going to try and work on integrating the server UNDER launchpad and see how that goes.

[benjaminbojko]

Ok sounds good. Might be nice to have some way to auto-generate a local salt. I was also wondering if it would make sense to switch up our credentials format to work with whatever we're building here. E.g. putting API keys and sensitive info in the .env file feels more common practice 🤔

I'll make a new ticket for that one, so don't worry about it for now 😆

[pingevt]

New branch covering this... feature/server. Everything is at least scaffolded but theres's a ton of todos, discussions, options etc. before it is alpha ready. I didn't PR it yet, I'd like to get a little more done before that. Theres also 2 separate issues opened up: #55 and #56 to move discussions about those specifically.

[pingevt]

@benjaminbojko Koa is implemented now on this branch feature/server for this. Theres still a ton of todos, etc. but the basic scaffold is there. What do we want to get to in order to open this up for testing?

[pingevt]

Update video on slack: https://bluecadet.slack.com/archives/C03RY7MRA4V/p1662649473942579