Support paths to job files in project.yaml

[amberrignell]

For users who work locally or for certain projects in their GitHub repos, users want to keep their jobs in different .js files and then have the files referenced in their project.yaml files. When OpenFn deploy is called, we need to make sure we get the content of the files and add them to the right section on the project.yaml.
We should advise users to create a job.js file for every job such that for a workflow with 5 steps, we will need 5 jobs.js files. that are referenced in the .yaml file.

• When using cli deploy and detecting a file path in the yaml, you merge in the real content before sending it to the lightning server
• When using cli pull you get a new copy of the project.yaml and the projectState.json and compare it to the local copy of the project.yaml... you notice that the local project.yaml has file paths, and instead of committing the entirely new project.yaml to GitHub, you interpret the paths and write to those files.
• If someone adds a new workflow or new steps to lightning directly, then syncs to GitHub for a project that already has relative paths for other workflows or jobs, they will retain the relative paths for things that are already set up locally, but the new workflow or job will be returned inline. (they can move it out later by hand if they want.)
• on clicking export, keep it as a monolith for now. (out of scope to make export use either relative paths or monolith.)

Implementation Note

Instead of putting a path directly in the body key (which would also be a string value), we use this pattern instead:

  job-xyz:
    ...
    body:
      path: ../my-job.js

Note that this path is relative to the project.yaml, and not the current working directory.

This way we can tell that the body is not available directly under the key much more easily, and leaves room for other features.

For instance, a step that looks like this in the project.yaml

 Notify-CHW-upload-successful:
       name: Notify-CHW-upload-successful
       adaptor: @openfn/language-http@latest
       enabled: true
       credential:
       globals:
       body: |
         fn(state => state);

We'd expect users to configure their step this way and cli deploy still works.

 Notify-CHW-upload-successful:
       name: Notify-CHW-upload-successful
       adaptor: @openfn/language-http@latest
       enabled: true
       credential:
       globals:
       body: 
         path: ../my-job.js

This does not apply to exports at the moment. All project exports will be monolithic i.e. job code and other project settings are kept in the same file.

User acceptance criteria

• If the body is an object, and doesn't have a path key then we throw an error.
• Users can easily make a change to a job expression, and redeploy without errors if everything is structurally correct
• Running cli deploy for a project with relative file paths in the project.yaml should copy and paste the job code in the project.yaml when deploying to lightning.
• Running cli pull for a project on lightning whose local project.yaml file uses a relative path should automatically copy and paste the job code into the right files without errors.
• The CLI pull/deploy criteria above should also apply to GitHub sync.

[taylordowns2000]

hey @amberrignell , the language here looks a little funny. I'd love to make sure we're on the same page. monday AM?

[josephjclark]

@taylordowns2000 is this issue in the right place? This feels like a Lightning export thing?

In a workflow you can specify a path to an expression, so the CLI supports this. Although the CLI does not let you run a project.yaml fie. I don't know the expected relationship between an exported project and a Workflow object (Execution Plan, in runtime parlance).

I think this is something we ought to talk through.

[taylordowns2000]

Weirdly, I think this belongs in Kit... Hear me out!

1. When Kit "talks to Lightning" it's sending nothing but a JSON payload (that "reduced deployment payload" which knows nothing of file paths.)
2. When we "describe" a project from Lightning using the CLI, we should indicate the file structure we want.
3. When we "deploy" a project from the CLI to Lightning, Kit should look at the file structure indicated (see point above?) and then create the reduced deployment payload (a big JSON blob) then send it to Lightning.

In other words, the CLI allows you to take Lightning JSON and make some files, or it allows you to take some files and make some Lightning JSON.

I think the file path stuff should live here. The Lightning "project-as-code" interface is (beautifully) dumb by design and the "wrangling" happens in CLI.

cc @zacck @stuartc

[taylordowns2000]

over on #561 we talked about this:

note @stuartc ... we discussed an implementation path where:

1. if you run openfn pull <projectId> with no existing project.yaml, you get a single shot by default.
2. if you run openfn pull <projectId> with no existing project.yaml but you have a separated: true(better name please?) attribute in yourconfig.jsonyou get a set of files nested in./jobs`. (our opinionated way.)
3. if you run `openfn pull with an existing project.yaml, we will read that project.yaml before updating it to determine if we need to place any resources in their own files/directories.
4. if you run openfn deploy with an existing project.yaml, we will read references files from filepaths (if any) to build a the "reduced deployment payload" (JSON blob) before sending it to the Lightning API.

I think this is still the simplest way to do it. @aleksa-krolls mentioned that this is fairly high priority recently and @mtuchi will probably agree.

The open question is whether we want to try to support more flexibility. It will be more complex/expensive to deliver, so the question for Aleksa and Mtuchi might be: "would you run into any major problems if you had two options: (a) monolith project.yaml OR (b) relative-path project.yaml where each job is a .js file inside a ./jobs folder?"

If there would be major problems, can you describe how that rigid option B would fail?