Support running an app from a subdirectory

[jmorrell]

There is currently no supported way to run a node app from a directory within your git repo. You can work around it most of the time by using git subtree push but that wouldn't work if you have shared code outside of your directory.

A super common situation is to have the following repo structure:

├── shared-code
├── server
└── frontend-app

Being able to direct the buildpack to look in /server instead of / would be helpful for cases like this.

[Arrow7000]

Exactly what I need. Thanks @jmorrell!

[jmike]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add package.json under server and frontend-app. List dependencies (as you normally do) and specify a start script for the server.

2. Create a root package.json file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

3. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

[jmike]

OK, I just realized you are working for Heroku, so this is probably not a real question, but rather a feature request. Sorry for the misunderstanding.

In any case, I will leave my answer above in case someone finds it useful.

[Arrow7000]

@jmike thanks for weighing in. Is this something you've tried yourself and has worked? Because in the absence of an official solution I'll give that a try.

[jmike]

@Arrow7000 yeap, works like a charm. The only problem is caching - see issue #387 even though it's somewhat unrelated to the application structure. See, I am using local files as npm dependencies.

In any case if you face a similar issue you can always disable caching according to heroku instructions https://devcenter.heroku.com/articles/nodejs-support#cache-behavior.

[Arrow7000]

Thanks @jmike that worked like a treat! It's not the cleanest solution, because it requires an extra package.json with important options duplicated - eg engines.node - but until we get an official solution this seems like the best and simplest way to get it done!

[gablg1]

We had been using git subtree push for a while but started facing the exact issue described above, so we implemented the following custom buildpack https://github.com/Pagedraw/heroku-buildpack-select-subdir

which allows us to deploy multiple apps from the same Heroku repo. Then we just make each of the frontend-app and server require the shared-code as an npm local dependency.

One caveat is that we have to explicitly add the node_modules folder installed to the NODE_PATH so npm knows where to look for requires within shared-code.

It works for us. Let me know if it also works for you!

[Jan0707]

@jmike I tried your approach and it sadly does not work for me.
Here is my package.json

{
  "name": "App",
  "engines": {
    "node": "8.1.x"
  },
  "scripts": {
    "postinstall": "npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js"
  }
}

It fails though when trying to run gulp:

remote: -----> Building dependencies
remote:        Installing node modules (package.json)
remote:
remote:        > App@ postinstall /tmp/build_98b4546541f7050670cac9ef04e8ace1
remote:        > npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js
remote:
remote:        added 14 packages in 2.164s
remote:        sh: 1: app/Resources/node_modules/.bin/gulp: not found
remote:        npm ERR! file sh
remote:        npm ERR! code ELIFECYCLE
remote:        npm ERR! errno ENOENT
remote:        npm ERR! syscall spawn
remote:        npm ERR! App@ postinstall: `npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js`
remote:        npm ERR! spawn ENOENT
remote:        npm ERR!
remote:        npm ERR! Failed at the App@ postinstall script.
remote:        npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
remote:
remote:        npm ERR! A complete log of this run can be found in:
remote:        npm ERR!     /app/.npm/_logs/2017-07-14T10_49_30_508Z-debug.log

Any idea on what I am doing wrong here would be highly appreciated :)

[atomkirk]

It appears that if you run the buildpack from within a subdirectory, the commands it's suppose to install are not available after release. My buildpack runs the heroku/php buildpack in a sub directory, but when I heroku run bash, which php returns nothing. Same goes if I cd into the directory.

[jmorrell]

@atomkirk The buildpacks install their dependencies globally, and I'm not sure what you mean by running the "php buildpack in a sub directory". Could you open a support ticket at https://help.heroku.com/ so we can sort out what's going wrong?

[atomkirk]

So it looks like they don't install them globally. When I deploy a root directory php app with heroku/php buildpack, then heroku run bash, and which php, its found in /app/.heroku/php/bin/php and echo $PATH is /app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/usr/local/bin:/usr/bin:/bin:/app/vendor/bin

So this is actually an easy fix, just need to fix PATH so it points to /app/subdir/… instead of /app/…

Would be nice if the buildpack system provided an option so we didn't have to make our own buildpacks to work around it

[jmorrell]

@atomkirk Sounds like you're trying to do something that's not supported cc @dzuelke

[jmike]

@Jan0707 have you tried running gulp on postinstall from within the internal package.json file, i.e. app/Resources/package.json in your case?

Another idea might be to install gulp with root package.json and update your configuration as such:

{
  "name": "App",
  "engines": {
    "node": "8.1.x"
  },
  "dependencies": {
    "gulp": "^3.9.1"
  },
  "scripts": {
    "postinstall": "npm install --prefix app/Resources && gulp --gulpfile app/Resources/gulpfile.js"
  }
}

Please note that you don't really need to provide the qualified file path for gulp - npm knows what to do.

[dzuelke]

Why a subdirectory, @atomkirk? Or, at least, why does the PHP runtime have to be in a subdirectory, and not just your application? A lot of moving parts are set up to look at $HOME/.heroku/… for buildpacks, and that includes how PHP is built and where it looks for its config files, where binaries look for other .sos, and so forth. It's not something that's easily changed, and so far I don't understand the use case for it.

[atomkirk]

Well, thanks for the response, but @jmorrell is right, I'm trying to do something unsupported. I kind of took us off topic. Sounds like if you want to support the original issue, the buildpack will probably have to run in the subdirectory and then PATH point at where it put the .heroku folder (or if the buildpack puts the .heroku folder in the repo root, then change nothing)

[gkatsanos]

This feature is what I was looking for. I have a client and a server which have two independent package files, and I want them to be sibling directories and not nested. Therefore I need a way to instruct heroku to start from a specific package file which is not located at the root of my repository.

[ivanovivelin]

Hi,
I seem to face the same issue when running under sub directory =>

"postinstall": "npm install --prefix server && npm install --prefix ./server/client",

The entire project is inside server folder , including the client ... any help

[jmorrell]

@IIvanov8888 could you open a support ticket at help.heroku.com?

[wmaciejak]

@gablg1 for me your solution doesn't work.

After valid deploy I see in console

app[client.1]: bash: npm: command not found. 

The same situation with node.
Even If I change $PATH variable, the issue still occurs.

[ClayShentrup]

I have a similar issue. I want to run an Ember Fastboot app without having to make a separate repo for my API backend (which is Rails in my case). Ideally I'd be able to specify multiple web processes for a single app. Since I can't do that, I have to make two separate apps for the frontend and backend. But I don't want to create multiple repos for this. I just want to be able to tell one app to use one subdirectory and tell the other app to use the other subdirectory from the git project root.

[WeishiZeng]

As mentioned in the ticket, one can use git subtree push --prefix {app_subfolder} heroku master to push a subfolder only. It would only work if the subfolder is self-contained

[Bnaya]

@WeishiZeng can you please elaborate? :)

[WeishiZeng]

@Bnaya The git command is used to push a subfolder to remote. If the subfolder happens to be a self-contained app recognized by heroku, then it'll be deployed.

[javidjamae]

One note to add to the conversation is that the git subtree push solutions only work when you're directly pushing your app to Heroku, but it doesn't solve the problem when you're setting up a Heroku CI Pipeline. In that case, the filesystem is readonly and (AFAIK) there is no way to control what subdirectory of the code is used for the test execution, deployment, etc.

I encountered this issue when trying to use a monorepo with a node.js app as I describe here: https://stackoverflow.com/questions/51449750/how-do-i-get-heroku-ci-to-run-tests-when-using-a-monorepo

[kris-campos]

has there not been an official solution for this yet? is there a ticket somewhere that we can follow?

[jmorrell]

@kris-campos I haven't had a chance to loop back on this yet, so there is no official solution. In the meantime I would look at Yarn workspaces: https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/ which I believe npm is also in the process of implementing.

This will be the first thing I explore as a potential solution. If you try them please report back with how it worked for you.

[danielmahon]

@jmorrell I just starting switching to yarn workspaces & lerna. So far using https://github.com/timanovsky/subdir-heroku-buildpack I can get the package to run properly but I'm unable to access a shared module because it bascially wipes everything outside the sub directory so shared-code doesnt exist and is a local module, not published. Did you figure out a good way to solve this?

├── shared-code
├── server
└── frontend-app

[danielmahon]

@jmorrell

UPDATE: Well I tried to avoid it but looks like I need to use https://www.npmjs.com/package/yalc instead of yarn link. Since the symlinks don't seem to always work properly inside Heroku. I could drop yarn workspaces completely for yalc and it would definitely simplify/remove most of these steps but the point of this was to try to use yarn workspaces so I'll leave it for now and just use yalc inside Heroku.

This is my current working deployment setup using a local lerna / yarn workspaces monorepo:

root
├── buildpack-run.sh
├── lerna.json
├── package.json
├── packages
│   ├── app
│   │   ├── package.json
│   ├── graphql
│   │   ├── package.json
│   ├── native
│   │   └── package.json
│   ├── shared
│   │   ├── package.json
│   └── site
│       ├── package.json
└── yarn.lock

First, I'm using these 3 buildpacks:

https://github.com/weibeld/heroku-buildpack-run
https://github.com/timanovsky/subdir-heroku-buildpack
https://github.com/danielmahon/create-react-app-buildpack

I use heroku-buildpack-run to run this script to copy the "sibling" shared package into a lib folder within the main package to be deployed, as well as the root level yarn.lock which while it contains ALL the package dependencies, yarn install should still only match the dependencies from package.json.

#!/bin/bash

echo "       Copying shared modules"

mkdir -p packages/app/lib/@myscope
cp -R packages/shared packages/app/lib/@myscope
cp yarn.lock packages/app
# repeat for other simultaneous deployments

I use subdir-heroku-buildpack which just pulls out and replaces your root build folder with one specified at PROJECT_PATH. This is why I needed to copy the shared package from the first step into this one.

I use a forked create-react-app-buildpack because I needed to update it's own dependancy of heroku-buildpack-nodejs with a forked version of my own https://github.com/danielmahon/heroku-buildpack-nodejs that simply adds the --ignore-optional flag to the default yarn install command.

Maybe we can set --ignore-optional with an env variable so the buildpacks don't need forked?

In my package.json for the deployed module, I have the following npm scripts to yalc add the shared package that's now in the lib folder.

  "scripts": {
    "heroku-postbuild": "npm-run-all -s yalc:publish yalc:add ",
    "yalc:publish": "cd lib/@myscope/shared && yalc publish",
    "yalc:add": "yalc add @myscope/shared"
    ...
  },

I also needed to set the shared package as an optional dependency since it is unpublished.

  "optionalDependancies": {
    "@myscope/shared": "*"
  },

As of now, running lerna version --exact --force-publish in the root of the monorepo, properly updates all the package versions, and performs a git push which triggers a new build on Heroku for two apps which share the same local sibling package.

This works but obviously requires much more setup than I would like, let me know if anyone sees anything that can be simplified (I suppose a dedicated buildpack would help), until there is a better option.

[chrissyast]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add package.json under server and frontend-app. List dependencies (as you normally do) and specify a start script for the server.
2. Create a root package.json file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

1. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

Why do you have the server and frontend-app in the postinstall but only the server in the procfile?

[danielleadams]

@chrissyast The postinstall script will occur during the build after npm install - it's running installs on the front end and back end code.

The Procfile specifies the "start" command for the server. There's only one server per Heroku dyno, and npm start --prefix server will start the server. Users host their assets (front end files) a few different ways... maybe via a CDN, or through their web server, or a separate web server, but that's why there's only one script for Procfile.

[gregory]

removing the yarn.lock did the trick for me :)

[emil14]

I don't really like the suggested solution. I have a Go app and now I have to add useless package.json just to run nodejs in subdirectory. In would be great if we could use Procfile or something like that to point to where package.json is located

[emil14]

BTW I'm also think that many people here are actually trying to deploy their build artifacts generated by grunt or webpack. But we don't keep our generated stuff in git - we have to build it on Heroku or worse - create new deploy-branches and modify .gitignore file to commit a crime. Should be an easy way to deploy stuff ignored by git

[LawJolla]

Heroku losing more ground. 😢

https://vercel.com/blog/monorepos

[ghost]

Why not let the official buildpacks accept an environment variable specifying the root directory? What's wrong with the blunt approach? An official reply from Heroku would be nice as to why this issue is difficult or taking so long to fix.

[danielleadams]

The reason we are waiting is because npm 7 will support workspaces (https://github.com/npm/rfcs/blob/latest/accepted/0026-workspaces.md), which is a solution for subdirectories in Node projects. Yarn already has subdirectory support, and npm will in the near future.

I helped review the npm workspaces RFC, and it seemed that this would be a good solution for Heroku developers. If we find that npm workspaces is not a fit for Heroku users, we can reexamine. We need to make this decision knowing how npm workspaces work with Heroku users so that the buildpack doesn't step on what npm is offering.

[LawJolla]

@danielleadams thanks for your reply and attention. But for a quick vent, neither Netlify, Vercel, nor AWS Amplify needed NPM 7 to do an incredible job supporting monorepos.

I’ve been reworking my servers to work with the server less architecture so I can, with a heavy heart, migrate away from Heroku.

[danielleadams]

@LawJolla I hear you and appreciate your support of Heroku. I'm fully aware of what the other cloud providers have done, but unfortunately, that is not the direction we opted to go.

[ghost]

Huh? What about the other Buildpacks then? Cause this is really an issue that ends up affecting all of them. Perhaps Heroku needs to admit that buildpacks are not suited for this problem and a new approach needs to be found. Maybe some sort of switchboard that lets us map the directories to their runtimes.

[danielleadams]

@StephenCarboni I hear you. You're welcome to open a support ticket to give that feedback, but the topic being discussed on this Issue is only for subdirectories within Node projects, hence the multiple comments of waiting for NPM's version of workspaces to be released.

[metal450]

I've spent the entire work day trying to get a simple monorepo deployed. I'm solely trying to deploy the frontend at this point. I've tried dozens of solutions, so far to no avail. The 2 apparently simplest from this thread seemed to be:

#385 (comment)
and
#385 (comment)

However in both cases, although it purports to have pushed successfully, accessing the app in a browser yields 404 not found. My project layout is simply:
/my-project
...../shared
...../frontend <-create-react-app

I tried accessing the heroku url directly, as well as accessing /frontend. Both cases, 404.

After reading this entire thread and probably 20 other blog posts about this issue, I'm honestly a bit baffled that such a seemingly common setup is so difficult. I'm not using yarn workspaces nor Lerna, just a simple react app that includes code from a shared folder. If anyone could give me some pointers as to specifically why the 2 comments mentioned above are yielding 404s, I'd be appreciated. I figured this final deployment step would be the simplest/quickest, but seems to be the biggest hangup of this whole project so far :(

[metal450]

Second full work day on this. I gave up on Heroku for the frontend, & deployed it to Netlify, which took about 2 seconds & worked immediately. Now I'm stuck with the backend.

I tried @alexpetralia 's idea of putting release: cd backend && npm install && npm run build. When I push, I see it running the commands, so it appears to work. However, it does not. If I login interactively (heroku run bash), none of the packages have been installed in "backend." If I manually run the exact same commands: cd backend && npm install && npm run build, then it works fine. It just doesn't work when they're in the procfile.

I still cannot comprehend that 3 years later this is so hacky & difficult.

[chrissyast]

@metal450 Would Netlify not work for the backend too?

[joserocha3]

@metal450 Can you try to use this build pack?
https://github.com/lstoll/heroku-buildpack-monorepo

Make sure to add the monorepo buildback, then add the node js buildpack below it. Also note that you do not use a Procfile with this buildpack.

I have it working with this monorepo setup like yours. Check the server directory. Notice the heroku-postbuild in package.json.
https://github.com/joserocha3/startup-super-pack

[metal450]

@chrissyast: Netlify is for static sites (frontends) only. I think you can maybe get backends to work with some modification ("serverless functions"), which I haven't started looking into. I guess I may just have to abandon Heroku entirely. It just still feels...ridiculous that it could possibly be this inflexible.

[metal450]

@joserocha3: Tried it - build failed, it couldn't find source files in /shared. I think this only works if your subprojects are well and truly completely separate, but if you give it a subdirectory that accesses any shared resources in another, it fails. i.e.:

 /project
 |--- /frontend
 |--- /backend   <-Build fails w/ monorepo buildpack if this accesses anything in /shared
 |--- /shared

[metal450]

Finally got my own hacky solution that works. It's similar to some of the others above, but or some reason using the Procfile didn't work at all (I tried using its release:, web:, and a combination of the both - all to no avail). With no Procfile, the root package.json contains only:

 {
     "scripts": {
       "postinstall": "npm install --prefix backend && npm run build --prefix backend",
       "start": "node backend/dist/app.js"
     }
   }

Make sure the backend's package.json's script tag has "build": "tsc" (it's a typescript project).

This works for the backend. I gave up on Heroku for the frontend, which is on Netlify. So between using Netlify for frontend & this for backend, I have a hacked-together working stack, if or until Heroku finally gets around to supporting something as basic as an official way to give it a subdirectory to run.

[umbe1987]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add `package.json` under _server_ and _frontend-app_. List dependencies (as you normally do) and specify a _start_ script for the server.

2. Create a root `package.json` file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

1. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

Thanks @jmike for the suggestion, it helped me a lot finding a solution for my case!

DISCLAIMER: It's been difficult!

I have a root folder with a server subfolder (that was created with express-generator-typescript), and a client subfolder (created with angular-CLI).

Instead of using postinstall I used heroku-prebuild, which is specific to Heroku and will not install anything when I build locally (I guess).

The other trick was to specify --production=false when I install with npm install, in order to install also the devDependencies.

Finally, heroku-postbuild is added to build client and server every time the app is updated (pushed to github, hopefully this makes sense...).

I managed to deploy to Heroku adding this package.json to my root folder:

root package.json (verbose because I originated it with npm init defaults)

{
  "name": "tips4family",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "cd ./server && node -r module-alias/register ./dist --env=production",
    "build": "cd ./server && npm run build --prod && cd ../client && npm run build --prod",
    "heroku-prebuild": "npm install --prefix server --production=false && npm install --prefix client --production=false",
    "heroku-postbuild": "cd ./server && npm run build --prod && cd ../client && npm run build --prod",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/umbe1987/tips4family.git"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/umbe1987/tips4family/issues"
  },
  "homepage": "https://github.com/umbe1987/tips4family#readme",
  "engines": {
    "node": "13.11.0",
    "npm": "6.13.7"
  },
  "dependencies": {}
}

Side note: to make the angular compile in the correct server dist folder, change client's angular.json to (only relevant part):

"options": {
            "outputPath": "../server/dist/public",
            ...

Note that if your server is not using typescript, you might want to use this instead (omit dist):

"options": {
            "outputPath": "../server/public",
            ...

[kaikuchn]

I wanted to replace an outdated custom buildpack that just installs node & npm for asset compilation in a subfolder (/assets/). And now I'm flabbergasted that I can't use the official buildpack because I can't tell it where to find my package.json

This is annoying :/ I don't know what workspaces are/will be but I'm not even sure if that would fix my problem...

[oswaldoacauan]

For future reference and for anyone that is running a monorepo using NPM workspaces.

You can workaround this issue by using the workspace option inside .npmrc file. This will make all npm commands to respect your workspace option, and that includes the npm ci that is automatically run in Heroku.

If anyone is deploying via Github Actions and using heroku-deploy action you can do something similar to this:

- name: ⬇️ checkout repo
  uses: actions/checkout@v4

- name: 🥁 update .npmrc with workspace option
  run: |
    printf "\nworkspace=\"your-app\"" >> .npmrc
    cat .npmrc
    git config --global user.email "${{secrets.HEROKU_DEPLOYMENT_EMAIL}}"
    git config --global user.name "github-actions[bot]"
    git add -A && git commit -m "updated .npmrc with workspace option"

- name: ♊️ deploy to heroku
  uses: akhileshns/heroku-deploy@v3.12.14
  ...