FEATURE: Combine Template with Database by Name

[daveseah]

Currently, the runtime directory contains a netcreate.loki database file that has the netgraph data (nodes, edges). The template file is specified by name, hardcoded into nc-logic.js as const TEMPLATE_URL = "../templates/netcreate.json"; This is inconvenient; we'd like one way to specify a database with matching template.

[daveseah]

THIS IS NO LONGER VALID
KEEP READING THREAD

1215
Based on yesterday's conversation, we're thinking of implementing an argument for netcreate's startup.

To select a project:

npm run dev "project"

where project specifies the name of the database file to use AND the name of the template file.

The runtime directory would look like this:

runtime/
  _default.template
  project.loki
  project.template

If the project doesn't exist, the server will automatically create the files. The template file would be created from _default.template

To implement this, I'm anticipating:

• See if we can easily add argument support to our run scripts
• Change the client's data bootstrap to include the template data
• Add necessary support in the server
• Implement robust data validation in all new code
• Make sure it also works in STANDALONE mode, which uses a different data loading mechanism

1220

[daveseah]

Background on how NetCreate Loads Data

nc-logic.js implements NetCreate's data request routines by tapping into the UNISYS.LOAD_ASSETS hook, which is guaranteed to run and complete before REACT gets a chance to call any render() functions. For this to work with async data loads, your hook function must return a Promise; the current code is a good example of how to implement it. You can implement multiple LOAD_ASSETS hooks and UNISYS will make sure they all execute before proceeding to the next execution phase.

The asynchronous data load is handled through UNISYS' network messaging system so you don't have to worry about anything except receiving the data. The data is provided by the server, which implements its own UNISYS message handlers relevant to database operations.

Specific Points of Implementation

The server will deliver both D3DATA and TEMPLATE information in one call. This will make it easier to support standalone mode.
server-database.js function DB.WriteJSON()

unisys/server-database.js

In DB.InitializeDatabase() update data format

• accept an option specifying a PROJECT string
• make sure PROJECT string is properly validated (no spaces, no weird characters, etc)
• ensure code creates the new blank database PROJECT.loki and uses that
• add a TEMPLATE collection
• add code to read the JSON file named PROJECT.template out of the runtime directory and stuff it into the TEMPLATE collection (how to do this is maybe tricky...maybe store it as a key=PROJECT, value is content of json file)
• add code to read the default JSON file if can't find an existing one

In DB.PKT_GetDatabase() and DB.WriteJSON() handled updated data packet format:

• current format is { nodes, edges }
• change to { d3data: { nodes, edges }, template } where template is an object with the template properties

Also check other DB.PKT_* routines to ensure they are not affected.

- - REMEMBER YOU HAVE TO RESTART THE SERVER WHEN YOU MAKE CODE CHANGES - -

datastore.js

In DSTOR.PromiseD3Data():

• read the new data format

nc-logic.js

In LOADASSETS hook:

• handle the new data format for both D3DATA and TEMPLATE in the then() handler for PromiseD3Data()
• eliminate the obsolete PromiseJSONFile() call

1300

[daveseah]

Testing

I need to implement the ability to pass-in the project string from the command line, but you can test your implementation by hard-coding the option passed-in to DB.InitializeDatabase()

Also, this implementation should work for standalone mode, so test that too.

1300

[daveseah]

Changes to Configuration Approach

In Slack Feb 26 2019 conversations:

• Passing parameters via npm run script is a pain in the butt, because the tools in our toolchain consume arguments and don't make it all the way to our scripts. A workaround is to pass ENVIRONMENT variables, but this is cumbersome.
• The code now accomplishes dataset collection by writing a file that is read by web clients (both standalone and server-connected). This file sets a global dataset value that can be used by nc-logic.js to request the correct file.

THE NEW WAY

There's a new NC_LOGIC global called dataset, which is set by a script netcreate-config.js read-in by index.ejs(server version) and index.html (standalone version). The script file itself is really simple because it's designed to be written by a utility script that we'll write for configuration by the user.

The script isn't written yet, but you can currently manually edit nc-config.js to test.

[daveseah]

User Interface for Configuration of Template/Dataset

The script would work something like this. In Terminal

./nc dataset=tiberius

The nc script will use ShellJS to manipulate the runtime directory with the appropriate files, copying files back and forth. This is not really ideal, but it should work unless we want to rewrite the server database code to handle it inside. This means rewriting the way that server database to be aware of the currently-selected dataset. It can read nc-config.js, but it would have to be turned into a universal module that can be imported both by the web client and by nodeJS (this isn't difficult)

[benloh]

This has now been implemented in #67