prometheus on gokrazy: how to include web UI assets?

[stapelberg]

          Some things are really straight forward and worked like a breeze. Basically just `gok add` these:

• github.com/prometheus/alertmanager/cmd/alertmanager
• github.com/prometheus/node_exporter
• github.com/prometheus/blackbox_exporter

And add options to the config.json for WaitForClock, ExtraFilePaths, and CommandLineFlags for each of them to use a provided config file, e.g. --config.file=/etc/alertmanager/alertmanager.yml.

Unfortunately, prometheus server is the hard part, if you want the GUI to work. This requires the assets to be built first and I could not find a way to do this with the upstream module, only with a local copy.

git clone git@github.com:prometheus/prometheus.git
cd prometheus
make build  # this is more than necessary (assets, assets-compress) but the easiest for a PoC
rm -rf web/ui/node_modules/  # not sure why these make trouble on go update
gok add ./cmd/prometheus
cat <<EOF # add build flag to config.json to include (this command only shows what to add)
"github.com/prometheus/prometheus/cmd/prometheus": {
    "GoBuildFlags": [
        "-a",
         "-tags=netgo,builtinassets,stringlabels"
     ],
}
EOF
gok update

Some notes:

• -a might not be necessary, but I did not check. It is what is used by the promu build tool.
• options for mapping config files need to be added

So this works for now, but I would prefer to not need to clone and update the prometheus repository manually.

If you have an idea how I could reliably get the assets for the build so I can use gok add github.com/prometheus/prometheus/cmd/prometheus, please let me know!

Originally posted by @Syphdias in #51 (comment)

[stapelberg]

Hey @Syphdias 👋

I split this question into a separate issue because it’s not really related to the userguide.

Does prometheus not use go:embed for its assets? Could/should it?

[Syphdias]

I split this question into a separate issue because it’s not really related to the userguide.

Hopefully it will be, if I write it down after I understood it.

Does prometheus not use go:embed for its assets? Could/should it?

I’m quite the beginner on go, so this will be a learning opportunity.

I figured since you explicitly mentioned prometheus in your recent GPN talk, you already run it on gokrazy and just didn’t get to writing it down. Sounds like you didn’t.

It took me quite a while to figure out why I got an error when just adding upstream and why the precompiled binary through breakglass worked. When using make build in the prometheus repository it uses npm first to generate some assets (make assets stage) and then compressed them and echos the list of files to the embed.go which is probably what you are referring to (script is here). This is why I went through make build to get all the necessary files first.

I have no idea if it should use go:embed. Yes, it already does but can I trigger this from the go build command somehow?

PS: Something of note might be that the go module version is different from the Prometheus version. Not sure if that matters in any way.

[stapelberg]

I figured since you explicitly mentioned prometheus in your recent GPN talk, you already run it on gokrazy and just didn’t get to writing it down. Sounds like you didn’t.

I run the node exporter and blackbox exporter regularly. I ran the prometheus server (built using make) for a few days for an event network, but I don’t do that on a permanent basis.

Sorry that the nuance of “how to deploy” wasn’t included in my talk for the prometheus example. I mainly wanted to point out that it’s possible, i.e. that prometheus runs in a gokrazy environment once you get it installed. I’ll be more careful regarding phrasing in the next iteration.

When using make build in the prometheus repository it uses npm first to generate some assets (make assets stage) and then compressed them and echos the list of files to the embed.go which is probably what you are referring to (script is here). This is why I went through make build to get all the necessary files first.

I have no idea if it should use go:embed. Yes, it already does but can I trigger this from the go build command somehow?

I see. In Go, generation of any sort (typically with the go generate command) is supposed to be done by the developers, with the result committed to source control. Apparently prometheus doesn’t commit the web ui assets to source control, so that’s what would need changing to make prometheus buildable with the go tool.

I think this would need a discussion upstream in prometheus. Haven’t checked the issues if buildability with the go tool was already discussed, though, so that would be the next step.

[Syphdias]

Sorry that the nuance of “how to deploy” wasn’t included in my talk for the prometheus example. I mainly wanted to point out that it’s possible, i.e. that prometheus runs in a gokrazy environment once you get it installed. I’ll be more careful regarding phrasing in the next iteration.

I really enjoyed your talk. How to deploy in detail was not the focus of the talk of the talk after all. And to be fair. I did get prometheus to run on gokrazy in two different ways even. One was to use the precompiled binary, the other was what you did prior to your talk. I was wondering how it should be done though.

I found prometheus/prometheus#7517 which refers to the mailing list post from 2020 where someone explains that "the new React UI requires some separate build steps that are not Go-based, and the result of that is not checked in (because it's hard to make completely reproducible builds that wouldn't create diffs every time the React UI is built)."
It then continues to link to where the magic happens: a dead link and lines in the Makefile but on master instead of a commit.
I also found prometheus/prometheus#548 which suggests that in 2015 the needed assets were available.

I guess it is unlikely that they will change their build process for our convenience. If the could use go generate, they probably would have.

So I see two approaches to deal with prometheus (and possibly other go written tools that require extra stuff:

1. Use the compiled binary the project provides. It only needs to be copied over and executed. It would be nice to be able to provide a URL where to get it from but since often releases come as archives. So that would require some sort of selecting the binary, e.g. http://some/prometheus/release.tar:prometheus.
2. Have a PreBuildCommand or somthing like that to do arbitrary things like make build before running go build.

1 could be accomplished with a small go programs to wrap the logic for getting the binary either from disk (ExtraFilePaths) or remote URL (potentially extracting tar). If done like this I think only one program can run like this.
I could also imagine this as gokrazy feature itself something like a dummy module using PreBuildBinary (not a existing feature). This might be a direction you do not want the project to go in, though.

2 would need to be implemented on gokrazy level unless there is a go build argument to run something arbitrary before doing the actual work,e.g. fake command: go build -prebuild "cd prometheus-path && make build" -tags=netgo,builtinassets,stringlabels

Do you have any ideas how to work around the asset generation problem?

[BrunoTeixeira1996]

Hi guys, I think a cool feature to add here (when the UI for the prometheus is working) is to add the Grafana stack.

[Syphdias]

Hi guys, I think a cool feature to add here (when the UI for the prometheus is working) is to add the Grafana stack.

@BrunoTeixeira1996, to be clear. Prometheus is absolutely working. I just didn't find a "main stream" way to install it. Check out the first post in this thread. The best way would be to make prometheus (the upstream project) include the assets when building. I think if go install github.com/prometheus/cmd/prometheus would work and provide a "asset filled" binary, then it would work for gokrazy out of the box.

There is already a guide to Grafana you can find here: https://gokrazy.org/development/non-go/

[stapelberg]

Thanks for digging through the history.

I guess it is unlikely that they will change their build process for our convenience. If the could use go generate, they probably would have.

Well, it can’t hurt to ask again. The discussions you linked to happened in 2015 and 2020, so maybe enough time has passed that they might reconsider.

Regarding the problem of reproducibility: Whether the submitted assets will be used could be controlled by a build tag. There’s already the builtinassets tag, so maybe we could have one more. That said, I’m not entirely sure what aspect of reproducibility is problematic. The Go build will be reproducible, even if the NPM part isn’t. If that’s problematic, I don’t see how submitting files changes the equation, as users won’t be able to reproduce the full prometheus install either way.

So I see two approaches to deal with prometheus (and possibly other go written tools that require extra stuff:

The general approach I follow with such special cases is that the working copy is managed outside of gokrazy, to the point that when the gokrazy image is built, a go build is all that’s needed. (I use the replace directive to point gokrazy to my working copy.)

For example, to include plugins in the Caddy webserver, the recommended approach is to underscore import the plugins and run the Caddy logic. So, instead of installing caddy directly in gokrazy, I instead have the following main.go:

package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"
	_ "github.com/caddyserver/caddy/v2/modules/standard"
	_ "github.com/kirsch33/realip"
	_ "github.com/caddyserver/transform-encoder"
)

func main() {
	caddycmd.Main()
}

In terms of updating this setup, I need to run go get github.com/caddyserver/caddy/v2 myself, either on-demand, or as part of my update script.

We should probably add this more explicitly to https://gokrazy.org/userguide/automation/, but that page already lists a few recipes for how to accomplish things outside the scope of gokrazy (for example: configuration templating).

So, in other words, I think custom installation, update or build steps, or otherwise massaging a piece of software into a consumable state for gokrazy, should be done before running the gok CLI. That clearly separates the responsibility and concerns, and keeps gokrazy small.

[Syphdias]

Well, it can’t hurt to ask again. The discussions you linked to happened in 2015 and 2020, so maybe enough time has passed that they might reconsider.

Done.

For example, to include plugins in the Caddy webserver, the recommended approach is to underscore import the plugins and run the Caddy logic. So, instead of installing caddy directly in gokrazy, I instead have the following main.go:

So, in other words, I think custom installation, update or build steps, or otherwise massaging a piece of software into a consumable state for gokrazy, should be done before running the gok CLI. That clearly separates the responsibility and concerns, and keeps gokrazy small.

You could use upstream prometheus and map any assets via ExtraFilePaths since prometheus does having them in the file system — shoot, I just realised you cannot map them to /perm/home/prometheus, but it could still be copied to it. As long as the react app stays compatible with the versions the assets do not need to change. Update path would then be to generate assets and copy them via breakglass.

[andig]

@stapelberg coming here from #287. Seems this issue is about including additional assets during build, with prometheus as example (update issue title to reflect this?).

For https://github.com/evcc-io/evcc we have a similar issue, but with a twist. Our UI assets are included in the build (go:embed), but need be generated from the cloned repo before build (compile vue into js). We do this via Makefile.
Since generation requires nodejs, we cannot simply do that on the fly using go:generate (I guess we actually could, but it feels wrong).
The root cause of the issue here seems to be that during gok overwrite, the build does not seem to use the current folder where we can pre-create the assets and hence the generated assets are missing during the embed step and our binary fails.

Can we influence the build process or the build folder somehow?

[stapelberg]

@stapelberg coming here from #287. Seems this issue is about including additional assets during build, with prometheus as example (update issue title to reflect this?).

Updated the title.

For https://github.com/evcc-io/evcc we have a similar issue, but with a twist. Our UI assets are included in the build (go:embed), but need be generated from the cloned repo before build (compile vue into js). We do this via Makefile. Since generation requires nodejs, we cannot simply do that on the fly using go:generate (I guess we actually could, but it feels wrong).

To be clear, go:generate is not run as part of the build, it’s a command that code authors run and commit the results to git.

For example, in another project of mine, I’m using this setup:

• https://github.com/gokrazy/bull/blob/main/regenerate.sh is called by go generate through this gen.go file’s //go:generate directive.
• regenerate.sh uses esbuild to bundle/minify JavaScript code
  • Note: Go 1.24 (not yet released) introduces tool management, so I am using go tool esbuild to version the esbuild tool instead of using the system’s version. Not totally relevant for your issue, but a nice feature.
• I commit the result of go generate to git (example commit).
• Building my program works with go install or go build, no generation step is needed.
• If you want to modify the JavaScript code, you still need to npm install as usual, then re-generate and submit to git. But for running the code, you don’t need to do anything.

If you can adopt this model of submitting the generated assets to git, you won’t need any extra steps.

The root cause of the issue here seems to be that during gok overwrite, the build does not seem to use the current folder where we can pre-create the assets and hence the generated assets are missing during the embed step and our binary fails. Can we influence the build process or the build folder somehow?

My previous answer from July 2023 is still current: #208 (comment), specifically:

So I see two approaches to deal with prometheus (and possibly other go written tools that require extra stuff:

The general approach I follow with such special cases is that the working copy is managed outside of gokrazy, to the point that when the gokrazy image is built, a go build is all that’s needed. (I use the replace directive to point gokrazy to my working copy.)

For the evcc case, I have already provided the exact recipe for such a setup over in #287:

# Create a new, empty gokrazy instance called evcc:
% gok -i evcc new

# Clone evcc into a directory somewhere on disk:
% git clone https://github.com/evcc-io/evcc
% cd evcc

# Setup the gokrazy instance to include this working directory
% gok -i evcc add .

# Work with gokrazy as usual…
% gok -i evcc overwrite …

Note the gok add call which sets up the replace directive accordingly. After that point, the result of running make (or whichever other command) will be picked up by gokrazy.

[andig]

For the evcc case, I have already provided the exact recipe for such a setup over in #287.

Thank you for your patience. It seems I'm missing two pieces of the puzzle: where does gok create the go.mod files for compile (containing the replace) and which config.json does it use? When I looked into /Users/andig/gokrazy/evcc/config.json it contained the evcc-io/evcc (which is expected) and I didn't realise that the actual magic was done via replace.
The other piece: https://gokrazy.org/userguide/automation/ suggests to create a config.json from template (in cwd?). gok apparently uses the one from /Users/andig/gokrazy/evcc/config.json.

[stapelberg]

When I looked into /Users/andig/gokrazy/evcc/config.json it contained the evcc-io/evcc (which is expected) and I didn't realise that the actual magic was done via replace.

The config.json file specifies which packages to include (by name, without a specific version), the go.mod file specifies which version of each package should be built.

where does gok create the go.mod files for compile (containing the replace) and which config.json does it use?

As explained in the quickstart section, gok new creates instances in the --parent_dir, which defaults to ~/gokrazy.

The gok new command also prints the full path to the config.json file it creates:

% gok -i evcc new
gokrazy instance configuration created in /home/michael/gokrazy/evcc/config.json
(Use 'gok -i evcc edit' to edit the configuration interactively.)
Use 'gok -i evcc add' to add packages to this instance
To deploy this gokrazy instance, see 'gok help overwrite'

If you’re ever in doubt where the file is located, you can run gok -i evcc edit to open the config.json file in your editor and see which path got opened.

The builddir is located in the instance directory, i.e. next to config.json. In my case, the builddir is in /home/michael/gokrazy/evcc/builddir, and I can find all go.mod files for all packages in there.

The other piece: https://gokrazy.org/userguide/automation/ suggests to create a config.json from template (in cwd?). gok apparently uses the one from /Users/andig/gokrazy/evcc/config.json.

The Makefiles described in that section live in the instance directory, too, i.e. next to config.json.