🐛 Bug Report: Unable to build and deploy Backstage using provided documentation

[cliffAtOsaic]

📜 Description

I have created and configured a Backstage application which builds and runs with yarn start-backend and yarn start.

I am now attempting to deploy our Backstage app to a QA server. The documentation recommends using docker for this. I have followed the steps at https://backstage.io/docs/deployment/docker to build the app with yarn and build the image with Docker.

👍 Expected behavior

When I execute the command docker run -it -p 7007:7007 backstage --env-file env.sh I expect the app to run in a Docker container on port 7007.

👎 Actual Behavior with Screenshots

When I run the docker app, I get a Node.js command prompt. I cannot access the website using port 7007 (or any port).

$ sudo docker run -it -p 7007:7007 backstage --env-file env.sh
Welcome to Node.js v20.18.0.
Type ".help" for more information.
>                                       

👟 Reproduction steps

Follow the instructions at https://backstage.io/docs/deployment/docker

📃 Provide the context for the Bug.

I am unable to deploy the app using Docker containers.

🖥️ Your Environment

$ yarn backstage-cli info
OS:   Linux 6.8.0-45-generic - linux/x64
node: v20.18.0
yarn: 4.4.1
cli:  0.27.1 (installed)
backstage:  1.31.1

Dependencies:
  @backstage/app-defaults                                          1.5.11
  @backstage/backend-app-api                                       1.0.0
  @backstage/backend-common                                        0.24.1, 0.25.0
  @backstage/backend-defaults                                      0.5.0
  @backstage/backend-dev-utils                                     0.1.5
  @backstage/backend-openapi-utils                                 0.1.18
  @backstage/backend-plugin-api                                    0.8.1, 1.0.0
  @backstage/catalog-client                                        1.7.0
  @backstage/catalog-model                                         1.7.0
  @backstage/cli-common                                            0.1.14
  @backstage/cli-node                                              0.2.8
  @backstage/cli                                                   0.27.1
  @backstage/config-loader                                         1.9.1
  @backstage/config                                                1.2.0
  @backstage/core-app-api                                          1.15.0
  @backstage/core-compat-api                                       0.2.8, 0.3.0
  @backstage/core-components                                       0.14.10, 0.15.0
  @backstage/core-plugin-api                                       1.9.4
  @backstage/e2e-test-utils                                        0.1.1
  @backstage/errors                                                1.2.4
  @backstage/eslint-plugin                                         0.1.9
  @backstage/frontend-app-api                                      0.8.0
  @backstage/frontend-plugin-api                                   0.7.0, 0.8.0
  @backstage/integration-aws-node                                  0.1.12
  @backstage/integration-react                                     1.1.31
  @backstage/integration                                           1.15.0
  @backstage/plugin-api-docs                                       0.11.9
  @backstage/plugin-app-backend                                    0.3.74
  @backstage/plugin-app-node                                       0.1.25
  @backstage/plugin-auth-backend-module-atlassian-provider         0.3.0
  @backstage/plugin-auth-backend-module-auth0-provider             0.1.0
  @backstage/plugin-auth-backend-module-aws-alb-provider           0.2.0
  @backstage/plugin-auth-backend-module-azure-easyauth-provider    0.2.0
  @backstage/plugin-auth-backend-module-bitbucket-provider         0.2.0
  @backstage/plugin-auth-backend-module-bitbucket-server-provider  0.1.0
  @backstage/plugin-auth-backend-module-cloudflare-access-provider 0.3.0
  @backstage/plugin-auth-backend-module-gcp-iap-provider           0.3.0
  @backstage/plugin-auth-backend-module-github-provider            0.2.0
  @backstage/plugin-auth-backend-module-gitlab-provider            0.2.0
  @backstage/plugin-auth-backend-module-google-provider            0.2.0
  @backstage/plugin-auth-backend-module-guest-provider             0.2.0
  @backstage/plugin-auth-backend-module-microsoft-provider         0.2.0
  @backstage/plugin-auth-backend-module-oauth2-provider            0.3.0
  @backstage/plugin-auth-backend-module-oauth2-proxy-provider      0.2.0
  @backstage/plugin-auth-backend-module-oidc-provider              0.3.0
  @backstage/plugin-auth-backend-module-okta-provider              0.1.0
  @backstage/plugin-auth-backend-module-onelogin-provider          0.2.0
  @backstage/plugin-auth-backend                                   0.23.0
  @backstage/plugin-auth-node                                      0.5.2
  @backstage/plugin-auth-react                                     0.1.6
  @backstage/plugin-bitbucket-cloud-common                         0.2.23
  @backstage/plugin-catalog-backend-module-logs                    0.1.0
  @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.2.0
  @backstage/plugin-catalog-backend                                1.26.0
  @backstage/plugin-catalog-common                                 1.1.0
  @backstage/plugin-catalog-graph                                  0.4.9
  @backstage/plugin-catalog-import                                 0.12.3
  @backstage/plugin-catalog-node                                   1.13.0
  @backstage/plugin-catalog-react                                  1.13.0
  @backstage/plugin-catalog                                        1.23.0
  @backstage/plugin-events-node                                    0.4.0
  @backstage/plugin-kubernetes-backend                             0.18.6
  @backstage/plugin-kubernetes-common                              0.8.3
  @backstage/plugin-kubernetes-node                                0.1.19
  @backstage/plugin-kubernetes-react                               0.4.3
  @backstage/plugin-kubernetes                                     0.11.14
  @backstage/plugin-org                                            0.6.29
  @backstage/plugin-permission-backend-module-allow-all-policy     0.2.0
  @backstage/plugin-permission-backend                             0.5.49
  @backstage/plugin-permission-common                              0.8.1
  @backstage/plugin-permission-node                                0.8.3
  @backstage/plugin-permission-react                               0.4.26
  @backstage/plugin-proxy-backend                                  0.5.6
  @backstage/plugin-scaffolder-backend-module-azure                0.2.0
  @backstage/plugin-scaffolder-backend-module-bitbucket-cloud      0.2.0
  @backstage/plugin-scaffolder-backend-module-bitbucket-server     0.2.0
  @backstage/plugin-scaffolder-backend-module-bitbucket            0.3.0
  @backstage/plugin-scaffolder-backend-module-gerrit               0.2.0
  @backstage/plugin-scaffolder-backend-module-gitea                0.2.0
  @backstage/plugin-scaffolder-backend-module-github               0.5.0
  @backstage/plugin-scaffolder-backend-module-gitlab               0.5.0
  @backstage/plugin-scaffolder-backend                             1.25.0
  @backstage/plugin-scaffolder-common                              1.5.6
  @backstage/plugin-scaffolder-node                                0.4.11
  @backstage/plugin-scaffolder-react                               1.12.0
  @backstage/plugin-scaffolder                                     1.25.0
  @backstage/plugin-search-backend-module-catalog                  0.2.2
  @backstage/plugin-search-backend-module-pg                       0.5.35
  @backstage/plugin-search-backend-module-techdocs                 0.2.2
  @backstage/plugin-search-backend-node                            1.3.2
  @backstage/plugin-search-backend                                 1.5.17
  @backstage/plugin-search-common                                  1.2.14
  @backstage/plugin-search-react                                   1.8.0
  @backstage/plugin-search                                         1.4.16
  @backstage/plugin-signals-react                                  0.0.5
  @backstage/plugin-techdocs-backend                               1.10.13
  @backstage/plugin-techdocs-common                                0.1.0
  @backstage/plugin-techdocs-module-addons-contrib                 1.1.14
  @backstage/plugin-techdocs-node                                  1.12.11
  @backstage/plugin-techdocs-react                                 1.2.8
  @backstage/plugin-techdocs                                       1.10.9
  @backstage/plugin-user-settings-common                           0.0.1
  @backstage/plugin-user-settings                                  0.8.12
  @backstage/release-manifests                                     0.0.11
  @backstage/test-utils                                            1.6.0
  @backstage/theme                                                 0.5.7
  @backstage/types                                                 1.1.1
  @backstage/version-bridge                                        1.0.9

### 👀 Have you spent some time to check if this bug has been raised before?

- [X] I checked and didn't find similar issue

### 🏢 Have you read the Code of Conduct?

- [X] I have read the [Code of Conduct](https://github.com/backstage/backstage/blob/master/CODE_OF_CONDUCT.md)

### Are you willing to submit PR?

None

[awanlin]

Hi @cliffAtOsaic, sorry to hear you are having issues, does it work when you run it exactly as the documentation shows - docker run -it -p 7007:7007 backstage

What is env.sh can you share that here? This isn't in the documentation so can't tell it's impact on this not working for you.

[cliffAtOsaic]

No, it doesn't work when I run it exactly as the documentation shows either. The env.sh contains our secrets so there isn't much I can share. It contains environment variables that are read in app-config.yaml.

The only change I made was to the Dockerfile to copy my env.sh into the image. When I run without the env-file parameter, it throws because it can't find the path to the https certificate:

Error: Failed to read config file at "/[path]/app-config.yaml", error at .backend.https.certificate.cert, $file substitution value was undefined

[awanlin]

Thanks for the follow up @cliffAtOsaic, this is odd as this works for me. I've been helping a few people with different issues around this subject with no clear insight into why it works for me and in our CI but people like yourself are having issues. There are the steps I take:

1. Run npx @backstage/create-app@latest using "backstage" as the name
2. cd into the backstage directory
3. Run yarn install --immutable
4. Run yarn tsc
5. Run yarn build:backend --config ../../app-config.yaml --config ../../app-config.production.yaml
6. Run docker image build . -f packages/backend/Dockerfile --tag backstage
7. Run docker run -it -p 7007:7007 backstage

Is that exactly what you did?

[cliffAtOsaic]

Hi @awanlin, yes, that's exactly what I did originally, only I had to add sudo to the docker commands. That gave me the error about the missing certificate path in the config file. I then modified the Dockerfile to copy the "env.sh" script and added the --env-file env.sh paramter to the last command (#7 on your list). The new sequence then became:

yarn install --immuatable
yarn tsc
yarn build:backend --config ../../app-config.yaml --config ../../app-config.production.yaml
sudo docker image build . -f packages/backend/Dockerfile --tag backstage
sudo docker run -it -p 7007:7007 backstage --env-file ./env.sh

What I find strange is that I get a Node.js command prompt with no output.

[awanlin]

Agree, it's very odd. I get this as my initial output based on the steps I posted:

Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config.yaml"}, FileConfigSource{path="/app/app-config.production.yaml"}, EnvConfigSource{count=0}}
{"level":"info","message":"Found 0 new secrets in config that will be redacted","service":"backstage"}
{"level":"info","message":"Listening on :7007","service":"rootHttpRouter"}
{"level":"info","message":"Plugin initialization started: 'app', 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'permission', 'search', 'kubernetes'","service":"backstage","type":"initialization"}

Can you share your full Dockerfile so I can try it locally, please?

[cliffAtOsaic]

Yes, thank you, please see below.

Reading through it, I also changed some directories from ~/ to /opt/backstage/ because running as sudo changed the home directory (to /home/root/ instead of /home/<user>/.

# This dockerfile builds an image for the backend package.
# It should be executed with the root of the repo as docker context.
#
# Before building this image, be sure to have run the following commands in the repo root:
#
# yarn install --immutable
# yarn tsc
# yarn build:backend
#
# Once the commands have been run, you can build the image using `yarn build-image`

FROM node:20-bookworm-slim

# Set Python interpreter for `node-gyp` to use
ENV PYTHON=/usr/bin/python3

# Install isolate-vm dependencies, these are needed by the @backstage/plugin-scaffolder-backend.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends python3 g++ build-essential && \
    rm -rf /var/lib/apt/lists/*

# Install sqlite3 dependencies. You can skip this if you don't use sqlite3 in the image,
# in which case you should also move better-sqlite3 to "devDependencies" in package.json.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends libsqlite3-dev && \
    rm -rf /var/lib/apt/lists/*

# From here on we use the least-privileged `node` user to run the backend.
USER node

# This should create the app dir as `node`.
# If it is instead created as `root` then the `tar` command below will fail: `can't create directory 'packages/': Permission denied`.
# If this occurs, then ensure BuildKit is enabled (`DOCKER_BUILDKIT=1`) so the app dir is correctly created as `node`.
WORKDIR /opt/backstage/docker/tmp

# Copy files needed by Yarn
COPY --chown=node:node .yarn ./.yarn
COPY --chown=node:node .yarnrc.yml ./

# This switches many Node.js dependencies to production mode.
ENV NODE_ENV=production

# This disables node snapshot for Node 20 to work with the Scaffolder
ENV NODE_OPTIONS="--no-node-snapshot"

# Copy repo skeleton first, to avoid unnecessary docker cache invalidation.
# The skeleton contains the package.json of each package in the monorepo,
# and along with yarn.lock and the root package.json, that's enough to run yarn install.
COPY --chown=node:node yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./
RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz

RUN --mount=type=cache,target=/opt/backstage/.cache/yarn,sharing=locked,uid=1000,gid=1000 \
    yarn workspaces focus --all --production && rm -rf "$(yarn cache clean)"

# This will include the examples, if you don't need these simply remove this line
COPY --chown=node:node examples ./examples

# Copy dev environment scripts
COPY --chown=node:node env.sh ./env.sh

# Then copy the rest of the backend bundle, along with any other files we might want.
COPY --chown=node:node packages/backend/dist/bundle.tar.gz app-config*.yaml ./
RUN tar xzf bundle.tar.gz && rm bundle.tar.gz

CMD ["node", "packages/backend", "--config", "app-config.yaml", "--config", "app-config.production.yaml"]

[cliffAtOsaic]

I may have discovered something interesting. I did some reading and it seems like Docker is exiting early because the Backstage app is exiting early. When the container starts, the command under CMD gets executed.

I decided to execute it outside of docker and here's what happened:

$ node packages/backend --config app-config.yaml --config app-config.production.yaml
node:internal/modules/cjs/loader:495
      throw err;
      ^

Error: Cannot find module '~/git/devops-backstage/backstage/packages/backend/dist/index.cjs.js'. Please verify that the package.json has a valid "main" entry

This runs normally using yarn start-backend and yarn start... should I instead be starting the app with Yarn and not Node?

[cliffAtOsaic]

This appears to be a repro of #24808

[awanlin]

The Docker command:

CMD ["node", "packages/backend", "--config", "app-config.yaml", "--config", "app-config.production.yaml"]

Is running the compiled code, the results of running - yarn build:backend --config ../../app-config.yaml --config ../../app-config.production.yaml.

Where as yarn dev or yarn start-backend and yarn start are not. I would expect the Docker command used at the command line to fail.

[cliffAtOsaic]

The Docker command:

CMD ["node", "packages/backend", "--config", "app-config.yaml", "--config", "app-config.production.yaml"]

Is running the compiled code, the results of running - yarn build:backend --config ../../app-config.yaml --config ../../app-config.production.yaml.

Where as yarn dev or yarn start-backend and yarn start are not. I would expect the Docker command used at the command line to fail.

I compiled the code with yarn install, yarn tsc, and yarn build:backend before running the CMD used in the Docker command. Is this the wrong CMD to be running in Docker?

[awanlin]

When you run yarn build:backend this compiles the code into the packages/backend/dist folder as a *.tar.gz file. The Dockerfile then copies that into the proper location and then extracts the files from the *.tar.gz files, this is done in the bottom half. You are not doing this which is why it's not working.

The Dockerfile command is correct and it's working for myself and others. What we need to figure out is what's the difference in your environment and mine. I'm running on macOS but you are on Linux 6.8.0-45-generic - linux/x64, could there be something there that we need to take into account? Are you using WSL?

I haven't had time to try out your Dockerfile yet, it's been a very busy week as I'm heading off to BackstageCon/KubeCon next week. I very much would like to though.

[cliffAtOsaic]

The problem with running in Docker is I get no output. Let my try extracting the files like it does in the Dockerfile and see if I can get the 'node' command to run locally.

I.e., I want to get a repro outside of Docker so I can see why Docker appears to be no-op.

I'm running Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-45-generic x86_64) on a virtual or physical corporate server, not WSL.

[cliffAtOsaic]

Not sure if this is relevant, but I got his error after extracting the skeleton.tar.gz package and running the yarn workspace command:

$ yarn workspaces focus --all --production && rm -rf "$(yarn cache clean)"
error This project's package.json defines "packageManager": "yarn@4.4.1". However the current global version of Yarn is 1.22.22.

Presence of the "packageManager" field indicates that the project is meant to be used with Corepack, a tool included by default with all official Node.js distributions starting from 16.9 and 14.19.
Corepack must currently be enabled by running corepack enable in your terminal. For more information, check out https://yarnpkg.com/corepack.

Could the issue be that Corepack is missing from the Dockerfile install? Are you using Yarn 1.x or Yarn 4.x?

[cliffAtOsaic]

Replacing yarn with corepack yarn allowed me to specify Yarn 4.x over Yarn 1.x.

After extracting the skeleton tarball and getting running corepack yarn workspaces focus --all --production && rm -rf "$(corepack yarn cache clean)" I still get the same error:

$ node packages/backend --config app-config.yaml --config app-config.production.yaml
node:internal/modules/cjs/loader:495
      throw err;
      ^

Error: Cannot find module '~/git/devops-backstage/test/packages/backend/dist/index.cjs.js'. Please verify that the package.json has a valid "main" entry

When I look at ./packages/backend there is no 'dist' folder. My guess is the tarball wasn't packed correctly?

packages/backend$ dir
package.json

packages/app$ ls
package.json

[darylgraham]

@cliffAtOsaic the skeleton tarball is just the package.json file so that makes sense. It’s the bundle that contains the actual files.

Take it back to first principles and do the build without Docker:

yarn install --immuatable
yarn tsc
yarn build:backend --config ../../app-config.yaml --config ../../app-config.production.yaml
tar tvf bundle.tar.gz

That should have the dist folder and index.cjs.js files that you expect.

[lalit774]

@awanlin I am updating yarn 1 to yarn 4. I am getting this issue.

As mention in Dockerfile. if i use followings.

backstage/packages/backend/Dockerfile

Line 51 in 0d5d421

yarn workspaces focus --all --production

i am getting this error becuase of dev dependencies.
yarn tsc
command not found

yarn build:all
command not found: backstage-cli

yarn dev
[backend] command not found: backstage-cli

It seems to me '--production' only download prodcution dependencies. so how should we mange this for production. curently i am using following commands on production.

yarn tsc
yarn build:all
yarn dev

[darylgraham]

@lalit774 you would normally run yarn tsc and yarn build:all before building the docker image. That is why the command you mentioned does not install dev dependencies like backstage-cli.

yarn dev is a tool used when developing locally, you should not use it in a deployed container.

[awanlin]

Thanks for helping out @darylgraham, I'm going to leave this with you 👍

[cliffAtOsaic]

Take it back to first principles and do the build without Docker:

So why does the Dockerfile only copy the skeleton and not the bundle?

[cliffAtOsaic]

tar tvf bundle.tar.gz

Never mind me, I see it copies the bundle tarball a little later in the Dockerfile.

I extracted bundle.tar.gz and ran the node command from CMD and now i have a new error:

$ node packages/backend --config app-config.yaml
node:internal/modules/cjs/loader:1228
  throw err;
  ^

Error: Cannot find module '@backstage/backend-defaults'

I confirmed that it is listed in package.json:

  "dependencies": {
    "@backstage/backend-common": "^0.25.0"
  } 

I ran yarn install --immutable and even tried yarn install and got the error both times.

I then tried yarn add @backstage/backend-defaults in the directory where I extracted the tarball... it added 12 packages (which is odd) but then still says it Cannot find module '@backstage/backend-defaults'.

[vinzscam]

I confirmed that it is listed in package.json:

  "dependencies": {
    "@backstage/backend-common": "^0.25.0"
  } 

I ran yarn install --immutable and even tried yarn install and got the error both times.

I then tried yarn add @backstage/backend-defaults in the directory where I extracted the tarball... it added 12 packages (which is odd) but then still says it Cannot find module '@backstage/backend-defaults'.

it seems you have @backstage/backend-common instead of @backstage/backend-defaults. Can you confirm you have @backstage/backend-defaults in your package.json?