Inventing deploy manifest

[rugwirobaker]

Change Summary

What and Why:

We need a way to start a deployment in one session on one machine and finish it in a different session or even another machine.
This will form the basis for our remote deployers that will help us monitor and recover from failed deployments without user intervention.

How:

To do that I'm investing what I will call a "deploy manifest" is basically a json blob/file that contains a json version of the fly.toml config file
the different flags we've passed to fly deploy and an image, the result of the build process. We're basically hijacking the deploy process just after
we've figured out all the deployment options and exporting them with the --export-manifest flag. To resume the rest of the deploy process we will
pass it to fly deploy with a --manifest flag.

Related to:
N/A

Documentation

• Fresh Produce
• In superfly/docs, or asked for help from docs team
• n/a

[KTanAug21]

Hi @rugwirobaker ! Here's trying my first pass on code reviewing a flyctl-deployments-related PR( let me know if this kind of review is useful or not! ):

Topic: I want to point out, in case this "remote-deploy" flag can be commented out, implemented, or of course, left as-is for future use

1. Is the flag "remote-deploy" already used when running the deploy command? The only code section I can find it mentioned in is this commented out line. Which points to the use of the function deployRemotely which I can't find declared anywhere just yet

2. If there's no implementation yet for its use, does this flag need to be functional before this PR is merged? It's not right, as this PR will just "provide the basis for remote deployers..."?

3. If it's not yet needed, can this be commented out for now? I think this can be added later on a separate PR, with an the implementation for its use.

4. But! Of course, it can also be left alone, as it is now, for future use.

[rugwirobaker]

Yeah, I commented it out that part but forgot to remove the flag. Thanks for pointing it out.

[KTanAug21]

I've also manually tested the two commands above!

• deploy --export-manifest manifest.json creates a manifest.json file:

  Running flyctl.exe deploy --export-manifest manifest.json, provides the following relevant results:
  > The "Deploy manifest saved to manifest.json" log was printed
  > Created a manifest.json file in the root directory of my Laravel Fly App project

• deploy --manifest manifest.json, will deploy using the manifest.json file, and reflect changes to the machine

  Now the fun part, to see if the next command works( which, I think can modify the deployment process right?), I added a new environment variable in manifest.json:

    "env": {
         "APP_ENV": "production",
        "DB_CONNECTION": "pgsql",
        "LOG_CHANNEL": "stderr",
        "LOG_LEVEL": "info",
        "LOG_STDERR_FORMATTER": "Monolog\\Formatter\\JsonFormatter",
        "SESSION_DRIVER": "cookie",
        "SESSION_SECURE_COOKIE": "true",
   +   "test": "test"
    },
  

  Then I run the command flyctl.exe deploy --manifest manifest.json. Afterwards, I connected with the machine with fly ssh console, checked if that env variable was reflected, and it was!

[rugwirobaker]

@KTanAug21 Yeah, that's exactly it. The manifest can be sent to a remote machine that can resume the deployment without uploading the whole codebase. I hadn't thought that you could modify the manifest post-generating it. This makes me wonder if that's desirable or if we'd want to make it immutable by generating some signature that will tell us if it was modified after it was generated.

[KTanAug21]

I see! Is this behavior of the manifest "being modifiable" the reason, why you've added that case "-" for getting the manifest from io.In?.

Will getting the manifest stream from io.In make sure that the manifest content wasn't modified?

[billyb2]

this is excellent work btw. this is a really great solution for remote deploys

[KTanAug21]

Is it also worth mentioning in the Description the use of "-" for getting manifest from io.In, as can be seen in this case "-"?

Also! I'm curious! I can't imagine how the scenario for using "-" actually works( I want to test it out). Is it like this?

1. run fly deploy --export-manifest="-", will print manifest details to stdout
2. run fly deploy --manifest="-" from ( somewhere else(a different console or machine?) OR the same console/machien? ) to get the manifest stream from running step Command to generate manifest #1 above, so that it can catch its stdout stream?

[KTanAug21]

Or is it something like run fly deploy --export-manifest="-" as input to fly deploy --manifest="-"?
If that's it, then how can I run it from my local console?

[rugwirobaker]

Heads up I changed --manifest to --from-manifest. I think fly deploy --from-manifest is better equipped to convey that we are deploying using a manifest.

And the right format is fly deploy --from-manifest - and fly deploy --export-manifest. This is mostly because flag.GetString always returns "" even when the flag is not specified. The if flag.GetString(ctx, "from-manifest")=="" would always be true and take over everything if we decided that an absence of the manifest filename means we will be reading from StdIn. So I decided to use this pattern cause I saw it used inf kubectl`

[rugwirobaker]

We can still improve the docs though