Topics/output dir

[Calderis]

Based on issue #1878

What ?

I suggest adding option --output-dir html to setup the output directory of build and develop command.

How ?

The path is set to environment GATSBY_OUTPUT_DIR and replace the hardcoded public value every where.

Since the env GATSBY_OUTPUT_DIR is set, it means we can configure it in a .env file with dotenv or start child process with env option.

Why ?

In that way we'll be able to run multiple instances of the same Gastby in parallel (if --output-dir is set dynamically) without conflicts in the output folder ☺️

Examples

Option 1
In your site folder, do

gatsby build --output-dir mywebsite

And you will have everything statically compiled below /mywebsite folder

Option 2
Install node package dotenv then create a .env file. Still in your site folder at the root

gatsby build

and your .env file

GATSBY_OUTPUT_DIR=mywebsite

Option 3
Managed by a distant node process (with a node child_process for example) you start your program by specifying your environment

let environment = process.env;
environment.GATSBY_OUTPUT_DIR = 'mywebsite';

let child = spawn('gatsby', ['build'], {
  cwd: gatsbyFolder,
  shell: true,
  env: environment
});

[gatsbybot]

Deploy preview for using-drupal ready!

Built with commit 5090a7b

https://deploy-preview-4756--using-drupal.netlify.com

[gatsbybot]

Deploy preview for gatsbygram ready!

Built with commit 5090a7b

https://deploy-preview-4756--gatsbygram.netlify.com

[Calderis]

When I pulled master my code remove code done in previous commit (10a1faf)
So I set it back 😶

[KyleAMathews]

Hey the code looks good! I do have a slightly annoying request — I really don't like calling the folder we build to the "output" folder. Could we use GATSBY_BUILD_DIR and buildDirectory, etc. for the names?

[Calderis]

Alright! I'm on it !

[Calderis]

Done. Just one test that seems not to pass

[KyleAMathews]

Yeah, appveyor is failing when building gatsbygram for some reason (on all PRs recently). The tests are passing though.

[KyleAMathews]

So one problem with this is that there could be community plugins that also hard-code public. I guess people who care about using a folder other than public can deal with those.

Also, output is still being used in a number of places including the cli flag name.

[Calderis]

Also, output is still being used in a number of places including the cli flag name.

you mean replacing --output-dir --build-dir ?

[KyleAMathews]

Yes and other places that say output

[Calderis]

Ready to make change if required :)

[pieh]

@Calderis It seems like there is conflict in gatsby-plugin-manifest right now

[Calderis]

Solved !

[m-allanson]

Should this line be removed? As people can override GATSBY_BUILD_DIR to set their preferred build directory.

[Calderis]

Yes you right, but in an other hand they could override node_env as well. In this case it mean that we use GATSBY_BUILD_DIR in gatsby and so it would create conflict to use it in an other way.

[m-allanson]

Thanks @Calderis! This is looking good, I've added one question. And have one more request:

Gatsby recently added a requirement that contributors sign-off commits. Could you do git rebase --signoff and force push the amended branch?

Check out the new Developer Certificate of Origin docs or comment here if you've any questions.

Thanks for seeing through these changes :)

[Calderis]

@m-allanson I think it should be good (hope I did the sign stuff in the right way)

[Calderis]

💪

[m-allanson]

Great stuff, thanks @Calderis!

[KyleAMathews]

We should validate that the build directory is a valid one. Someone could do ../../../../../ or whatever. Perhaps we should insist that the build directory is a simple directory name at the same level as public would be? Or would people want to do nested build directories e.g. build1/test-1 or whatever?

[Calderis]

I think we should give the possibility to people to d nested build directories

[Calderis]

If the folder doesn't exist, we create it first

[m-allanson]

I wonder if we should add more checks to the path of the build directory. e.g. if I set the path to / is that going to cause problems? Maybe limit it to being inside the current Gatsby project?

[Calderis]

I think we shouldn't be too involved in what user want to do. If we setup constraint it will be a pain for developer to do it their own way. If for some reason the guy want to point a folder out of gatsby, he should be able to do it (imo).

[m-allanson]

👍 🎉 🙏

[m-allanson]

Merged! I've added an issue for creating docs for this feature: #5623

[Calderis]

💪👨‍💻✌️

[KyleAMathews]

Awesome! Just published this PR so feel free to give it a go for realz.

[KyleAMathews]

This PR broke both develop and builds so reverting... :-(

#5636

Just a note — webpack requires have to be static — you can't do string interpolation.

[rdela]

Coming over from #5636, see also #5639 & #5640

One can find a more verbose version of bug report below at
hygraph/gatsby-graphcms-example#72 (comment)
in case it helps diagnose breakage, mentioned in #5636 (comment)

The
gatsby-plugin-manifest (dependencies): from 1.0.22 to 1.0.23
gatsby-plugin-offline (dependencies): from 1.0.16 to 1.0.17
updates were fine
rdela/gatsby-graphcms-rd@97007c6
https://app.netlify.com/sites/kokodokai/deploys/5b0fed5611b73b405d8fa4f1

…but in combination with the
gatsby (dependencies): from 1.9.262 to 1.9.264 1.9.265
update, broke build on node/yarn current
rdela/gatsby-graphcms-rd@af1270c
rdela/gatsby-graphcms-rd@9bf65aa
https://app.netlify.com/sites/kokodokai/deploys/5b101af21f12b716302675d7
https://app.netlify.com/sites/kokodokai/deploys/5b103e590733d565e88f65d9

Same error, this is probably a result of the string interpolated webpack require Kyle mentions above. Maybe it will help to have an example, let me know if I can help test fix please, and thanks for all the work everyone.

11:32:03 AM: error Building static HTML for pages failed
11:32:03 AM: See our docs page on debugging HTML builds for help https://goo.gl/yL9lND
11:32:03 AM:    6 | };
11:32:03 AM:    7 | function webpackContextResolve(req) {
11:32:03 AM: >  8 | 	return map[req] || (function() { throw new Error("Cannot find module '" + req + "'.") }());
11:32:03 AM:      | 	                                ^
11:32:03 AM:    9 | };
11:32:03 AM:   10 | webpackContext.keys = function webpackContextKeys() {
11:32:03 AM:   11 | 	return Object.keys(map);
11:32:03 AM: 
11:32:03 AM:   WebpackError: Cannot find module './/opt/build/repo/public/chunk-manifest.json  '.
11:32:03 AM:   
11:32:03 AM:   - .json$:8 
11:32:03 AM:     ^/.//.*//chunk/-manifest/.json$:8:34
11:32:03 AM:   
11:32:03 AM:   - .json$:8 webpackContextResolve
11:32:03 AM:     ^/.//.*//chunk/-manifest/.json$:8:89
11:32:03 AM:   
11:32:03 AM:   - .json$:5 webpackContext
11:32:04 AM:     ^/.//.*//chunk/-manifest/.json$:5:1
11:32:04 AM:   
11:32:04 AM:   - static-entry.js:188 module.exports
11:32:04 AM:     .cache/static-entry.js:188:25
11:32:04 AM:   
11:32:04 AM:   - static-entry.js:99 Function.Promise.fromNode.Promise.fromCallback
11:32:04 AM:     .cache/static-entry.js:99:27

Update
Upgrading to Gatsby 1.9.266 with this reverted solved the issue for me. rdela/gatsby-graphcms-rd#60 (comment)

[Calderis]

is there someone to guide me a little to resolve the webpack stuff ?

[digitalkaoz]

@KyleAMathews a configurable output path (public + .cache) would be useful in readonly filesystems (lambda) to. so whats the state of this?

[m-allanson]

@Calderis I'm actually not too sure, maybe we can figure it out during a pairing session? If you sign up for a time slot at https://calendly.com/gatsbyjs, that'll hook you up with a Gatsby team member.

[Calderis]

@m-allanson Gatsby 2 is coming, Is it worth to ?

[digitalkaoz]

It even doesnt work in gatsby2 Wetteren Rémi <notifications@github.com> schrieb am Fr., 3. Aug. 2018, 22:45:

…

@m-allanson <https://github.com/m-allanson> Gatsby 2 is coming, Is it worth to ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAR613vaFi8KD0afw8hzedcfvj1EFemHks5uNGIEgaJpZM4TAQ-N> .

[jlengstorf]

Hiya @Calderis! 👋

This is definitely late, but on behalf of the entire Gatsby community, I wanted to say thank you for being here.

Gatsby is built by awesome people like you. Let us say “thanks” in two ways:

1. We’d like to send you some Gatsby swag. As a token of our appreciation, you can go to the Gatsby Swag Store and log in with your GitHub account to get a coupon code good for one free piece of swag. (We’ve got t-shirts and hats, plus some socks that are really razzing our berries right now.)
2. If you’re not already part of it, we just invited you to join the Gatsby organization on GitHub. This will add you to our team of maintainers. You’ll receive an email shortly asking you to confirm. By joining the team, you’ll be able to label issues, review pull requests, and merge approved pull requests.

If you have questions, please don’t hesitate to reach out to us: tweet at @gatsbyjs and we’ll come a-runnin’.

Thanks again! 💪💜

[coltonbh]

I'd like to echo what @digitalkaoz mentioned above. The configurable output path is necessary for building static sites using AWS Lambda. Here's the use case I'd like to have:

We current use gatsby to build medical reports for clinic members. Our input to the build process is a directory containing files with arrays of JSON objects and images/movies. We use gatsby-source-filesystem to ingest these files and then build a unique static site for each member who visits our facility that summarizes their medical data from a visit. Gatsby is great for this!

I'm starting to build a more serious pipeline for automating gatsby site generation. Using AWS Lambda to build gatsby sites would be ideal (ingest this asset package from S3, build report on Lambda, publish to a new bucket that serves the static site securely to members); however, since AWS Lambda only allows writes to the /tmp/ directory, I can't build our reports on this service. Lambda seems like an ideal infrastructure piece for building and publishing sites in this situation where I'd need to build thousands of sites each from a unique data input package. Additionally, the solution of writing files to /public and then copying them over using some other tool isn't as option since AWS Lambda's filesystem is read-only except for the /tmp/ directory.

In short--using AWS Lambda for building gatsby sites is not an option. Being able to build to a specific directory would make this a possibility and it would be a great way for us to build thousands of gatsby sites programatically.

[MpMeetPatel]

"postbuild": "rm -rf build/* && cp -r public/* build/", --> you can have any name intead of build folder ofcourse.

make sure directory build exists before you run the command

I use this command (The main reason of needing this , i put public folder into .gitignore and i only push build code on github/gitlab)

[coltonbh]

Hi @MpMeetPatel. That would not work for the use case I've specified. AWS lambda is a read-only file system with the only allowed writes being to the /tmp/ directory. So if you want to use AWS lambda for building gatsby sites--in our case thousands of them--we'd need the ability to build directly to the /tmp/ directory. Building to ./public and then copying the files to another directory is not possible on Lambda.

[davidrhoderick]

Sorry everyone to revive this very old thread but @coltonbh, was there ever a solution to deploying via an AWS Lambda using /tmp? I've been doing some searching online and I can't seem to figure out how to gatsby build to the /tmp folder...

[coltonbh]

Hi @davidrhoderick. I haven't been working Gatsby since my original post. However it looks like #4756 added this feature to Gatsby maybe?