x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

bradfitz · 2019-10-16T16:22:02Z

Following the instructions at https://github.com/golang/build/tree/master/app/appengine#deploying I can no longer deploy:

bradfitz@go:~/src/golang.org/x/build/app/appengine$ GO111MODULE=on gcloud app --account=bradfitz@google.com --project=golang-org deploy app.yaml
Services to deploy:
        
descriptor:      [/home/bradfitz/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/bradfitz/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t151730]
target url:      [https://build-dot-golang-org.appspot.com]


Do you want to continue (Y/n)?

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 263 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: Filename cannot contain '.', '..', '\r', start with '-', '_ah/', or '\n': cmd/coordinator/coordinator.

What changed?

Docs old?

/cc @toothrot @dmitshur @bcmills @andybons

The text was updated successfully, but these errors were encountered:

dmitshur · 2019-10-16T16:33:53Z

Docs worked when I deployed last, on Oct 8. I use --no-promote flag explicitly and then promote after making sure the new version looks okay, but I don't expect that to cause the difference above.

Is your git status clean and are you on latest master commit of x/build?

I'm trying a --no-promote deploy now and not seeing any issues so far:

appengine $ GO111MODULE=on gcloud --project=golang-org app deploy --no-promote app.yaml
Services to deploy:

descriptor:      [/Users/dmitshur/go/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/Users/dmitshur/go/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t122939]
target url:      [https://20191016t122939-dot-build-dot-golang-org.appspot.com]


     (add --promote if you also want to make this service available from
     [https://build-dot-golang-org.appspot.com])

Do you want to continue (Y/n)?  Y

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 1 file to Google Cloud Storage                 ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...done.                                               
Deployed service [build] to [https://20191016t122939-dot-build-dot-golang-org.appspot.com]

You can stream logs from the command line by running:
  $ gcloud app logs tail -s build

To view your application in the web browser run:
  $ gcloud app browse -s build --project=golang-org


Updates are available for some Cloud SDK components.  To install them,
please run:
  $ gcloud components update

gopherbot · 2019-10-16T16:45:52Z

Change https://golang.org/cl/201477 mentions this issue: app/appengine: update deploy instructions

bradfitz · 2019-10-16T17:08:53Z

I did have an empty file cmd/coordinator/coordinator. (final period). Now that that's deleted, it makes it further but fails with Cloud build did not succeed within 10m:

^Cbradfitz@go:~/src/golang.org/x/build/app/appengine$ GO111MODULE=on gcloud app --account=bradfitz@google.com --project=golang-org deploy app.yaml
Services to deploy:

descriptor:      [/home/bradfitz/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/bradfitz/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t165030]
target url:      [https://build-dot-golang-org.appspot.com]


Do you want to continue (Y/n)?

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 1 file to Google Cloud Storage                 ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...failed.                                                                                                                                                      
ERROR: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m.
Build error details: Build error details not available..
Check the build log for errors: https://console.cloud.google.com/gcr/builds/4d8f8f8a-bf65-4e0d-9cc6-ebc8053c8422?project=397748307997

Versions:

$ gcloud version
Google Cloud SDK 246.0.0
alpha 2019.05.10
app-engine-go 
app-engine-python 1.9.85
beta 2019.05.10
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.05.10
gsutil 4.38
kubectl 2019.05.10

dmitshur · 2019-10-16T17:19:59Z

That's very strange. The deploy took 2~ minutes for me when I tried it just now.

The error is similar to an issue we had when deploying the playground was taking too long, and hitting that time limit. See #28046. You can try increasing the cloud build timeout with gcloud config set app/cloud_build_timeout 900, but I think we need to fix the build taking more than 10 minutes in the first place.

Looking at the cloud build log mentioned in your output, it just says:

[...]
Finished Step #1 - "builder"
TIMEOUT
ERROR: context deadline exceeded

Compare with my earlier successful build (https://console.cloud.google.com/cloud-build/builds/65043963-4de2-4a3d-bf06-9862eace8b0b?project=golang-org):

[...]
Finished Step #1 - "builder"
PUSH
DONE

Not a lot of relevant detail here. Maybe it was a flake, so try again?

bradfitz · 2019-10-16T17:24:16Z

I can't wait for #34744 to delete all this.

bradfitz · 2019-10-16T17:39:36Z

Ah, I did have some extra files in git status. It was an unknown directory I thought was harmless but it had some huge iso/disk images in it.

That didn't matter prior to the GO111MODULE=on addition. Now I guess we upload everything to The Cloud, rather than just the relevant files.

Maybe I'll add a sanity check step in there to look for big unknown files.

dmitshur · 2019-10-16T18:04:40Z

Yeah, GO111MODULE=on makes gcloud app deploy use the module root as the root, rather than the directory containing app.yaml. It was added to make deploys consistent regardless of who was deploying and their local environment’s GO111MODULE value.

I think there are ways to ignore certain files or directories from being included, for example via .gcloudignore file. See https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore.

dmitshur · 2019-11-06T17:31:34Z

@bradfitz Is there more to do here, or did CL 205597 fix this by adding the ~~.cloudignore~~ .gcloudignore file?

Edit: Fixed typo.

bradfitz · 2019-11-06T21:47:54Z

@dmitshur, the .gcloudignore (with a g) file fixes the worst of it. That could be enough probably.

dmitshur · 2020-01-14T18:29:50Z

I ran into this today too. I couldn't deploy x/build/app/appengine because it was failing to build due to a "MemoryError":

dmitshur@dev-dmitshur:~/go/src/golang.org/x/build$ cd app/appengine/
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ go test 
PASS
ok  	golang.org/x/build/app/appengine	0.108s
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ make deploy-test
go install golang.org/x/build/cmd/xb
GO111MODULE=on gcloud app --account=$(xb google-email) --project=golang-org deploy --no-promote app.yaml
Services to deploy:

descriptor:      [/home/dmitshur/go/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/dmitshur/go/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20200114t160755]
target url:      [https://20200114t160755-dot-build-dot-golang-org.appspot.com]


     (add --promote if you also want to make this service available from
     [https://build-dot-golang-org.appspot.com])

Do you want to continue (Y/n)?  Y

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 560 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...failed.                                             
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 9610bd65-62a1-4ba6-bd2e-8f60981f0d64 status: FAILURE.
Error ID: E479B424.
Error type: BuildError.
Error message: 2020/01/14 16:10:38 Building /tmp/staging/srv, with main package at ./app/appengine, saving to /tmp/staging/usr/local/bin/start
2020/01/14 16:10:38 Running /usr/local/go/bin/go build -o /tmp/staging/usr/local/bin/start ./app/appengine
2020/01/14 16:11:13 Wrote build output to /builder/outputs/output
2020/01/14 16:11:13 Wrote output to /tmp/staging/usr/local/bin/start
2020/01/14 16:11:13 Making tarball /tmp/tar-out666215249/app.tar.gz
2020/01/14 16:12:21 Creating app image on GCR: us.gcr.io/golang-org/app-engine-tmp/app/ttl-2h:232fb67d-0b88-4b64-80ac-8bc887b981e5
2020/01/14 16:13:06 Failed to create app image on GCR: [/bin/appender.par --src-image us.gcr.io/gae-runtimes/go113:go113_20191215_1_13_5_RC00 --tarball /tmp/tar-out666215249/app.tar.gz --dst-image us.gcr.io/golang-org/app-engine-tmp/app/ttl-2h:232fb67d-0b88-4b64-80ac-8bc887b981e5]: Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/bin/appender.par/__main__.py", line 90, in <module>
  File "/bin/appender.par/__main__.py", line 76, in main
  File "/bin/appender.par/containerregistry/client/v2_2/append_.py", line 74, in __init__
  File "/bin/appender.par/containerregistry/client/v2_2/docker_image_.py", line 110, in uncompressed_blob
  File "/usr/lib/python2.7/gzip.py", line 260, in read
    self._read(readsize)
  File "/usr/lib/python2.7/gzip.py", line 319, in _read
    self._add_read_data( uncompress )
  File "/usr/lib/python2.7/gzip.py", line 337, in _add_read_data
    self.extrabuf = self.extrabuf[offset:] + data
MemoryError.
Makefile:10: recipe for target 'deploy-test' failed
make: *** [deploy-test] Error 1
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$

I had a "clean working tree" as reported by git status, but the problem was that there are 16 .gitignore files in the x/build repository, and they mask some files from being easily visible while the current .gcloudignore doesn't exclude them from being uploaded:

dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git-branches 
| Branch               | Base   | Behind | Ahead |
|----------------------|--------|-------:|:------|
| **deploy-cl-214433** | master |      0 | 1     |
| master               | master |      0 | 0     |

| Branch               | Remote        | Behind | Ahead |
|----------------------|---------------|-------:|:------|
| **deploy-cl-214433** |               |        |       |
| master               | origin/master |      0 | 0     |
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ find ../.. -name '.gitignore'
../../cmd/buildlet/.gitignore
../../cmd/coordinator/.gitignore
../../cmd/pubsubhelper/.gitignore
../../cmd/docker2boot/.gitignore
../../cmd/scaleway/.gitignore
../../cmd/gitmirror/.gitignore
../../env/darwin/macstadium/10_8/.gitignore
../../env/windows/.gitignore
../../env/openbsd-386/.gitignore
../../env/plan9/.gitignore
../../env/netbsd-amd64/.gitignore
../../env/freebsd-amd64/.gitignore
../../env/linux-ppc64le/osuosl/.gitignore
../../env/netbsd-386/.gitignore
../../env/openbsd-amd64/.gitignore
../../maintner/maintnerd/.gitignore
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ rm ../../cmd/buildlet/.gitignore ../../cmd/coordinator/.gitignore ../../cmd/pubsubhelper/.gitignore ../../cmd/docker2boot/.gitignore ../../cmd/scaleway/.gitignore ../../cmd/gitmirror/.gitignore ../../env/darwin/macstadium/10_8/.gitignore ../../env/windows/.gitignore ../../env/openbsd-386/.gitignore ../../env/plan9/.gitignore ../../env/netbsd-amd64/.gitignore ../../env/freebsd-amd64/.gitignore ../../env/linux-ppc64le/osuosl/.gitignore ../../env/netbsd-386/.gitignore ../../env/openbsd-amd64/.gitignore ../../maintner/maintnerd/.gitignore
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ find ../.. -name '.gitignore'
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    ../../cmd/buildlet/.gitignore
	deleted:    ../../cmd/coordinator/.gitignore
	deleted:    ../../cmd/docker2boot/.gitignore
	deleted:    ../../cmd/gitmirror/.gitignore
	deleted:    ../../cmd/pubsubhelper/.gitignore
	deleted:    ../../cmd/scaleway/.gitignore
	deleted:    ../../env/darwin/macstadium/10_8/.gitignore
	deleted:    ../../env/freebsd-amd64/.gitignore
	deleted:    ../../env/linux-ppc64le/osuosl/.gitignore
	deleted:    ../../env/netbsd-386/.gitignore
	deleted:    ../../env/netbsd-amd64/.gitignore
	deleted:    ../../env/openbsd-386/.gitignore
	deleted:    ../../env/openbsd-amd64/.gitignore
	deleted:    ../../env/plan9/.gitignore
	deleted:    ../../env/windows/.gitignore
	deleted:    ../../maintner/maintnerd/.gitignore

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	../../cmd/buildlet/buildlet.openbsd-386-64
	../../cmd/buildlet/buildlet.openbsd-amd64-64
	../../cmd/coordinator/coordinator
	../../cmd/gitmirror/gitmirror
	../../env/openbsd-386/install64-i386.iso
	../../env/openbsd-386/openbsd-i386-gce.tar.gz
	../../env/openbsd-amd64/install64-amd64.iso
	../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz
	../../maintner/maintnerd/ca-certificates.crt
	../../maintner/maintnerd/maintnerd

no changes added to commit (use "git add" and/or "git commit -a")
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ ls -lah  ../../cmd/buildlet/buildlet.openbsd-386-64 ../../cmd/buildlet/buildlet.openbsd-amd64-64 ../../cmd/coordinator/coordinator ../../cmd/gitmirror/gitmirror ../../env/openbsd-386/install64-i386.iso ../../env/openbsd-386/openbsd-i386-gce.tar.gz ../../env/openbsd-amd64/install64-amd64.iso ../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz ../../maintner/maintnerd/ca-certificates.crt ../../maintner/maintnerd/maintnerd
-rwxr-xr-x 1 dmitshur dmitshur 6.9M Oct  9  2018 ../../cmd/buildlet/buildlet.openbsd-386-64
-rwxr-xr-x 1 dmitshur dmitshur 7.9M Oct  9  2018 ../../cmd/buildlet/buildlet.openbsd-amd64-64
-rwxr-xr-x 1 dmitshur dmitshur  33M Feb 27  2019 ../../cmd/coordinator/coordinator
-rwxr-xr-x 1 dmitshur dmitshur  13M Jan  4  2019 ../../cmd/gitmirror/gitmirror
-rw-r--r-- 1 dmitshur dmitshur 324M Oct 19  2018 ../../env/openbsd-386/install64-i386.iso
-rw-r--r-- 1 dmitshur dmitshur 326M Oct 26  2018 ../../env/openbsd-386/openbsd-i386-gce.tar.gz
-rw-r--r-- 1 dmitshur dmitshur 344M Oct 19  2018 ../../env/openbsd-amd64/install64-amd64.iso
-rw-r--r-- 1 dmitshur dmitshur 372M Oct 26  2018 ../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz
-rw-r--r-- 1 dmitshur dmitshur 230K Jul 17  2018 ../../maintner/maintnerd/ca-certificates.crt
-rwxr-xr-x 1 dmitshur dmitshur  19M Jan 25  2019 ../../maintner/maintnerd/maintnerd
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$

In my case, it was trying to deploy more than 1 GB of unwanted binary data.

I've opened #36553 to either remove .gitignore files that we can get by without (so that files aren't invisible when you do git status), or include them into .gcloudignore.

xyzdevops · 2020-01-17T22:21:53Z

having a shared go.mod file causes unpredictable deploys. best to have a separate .mod file in each app engine app.yaml folder

gopherbot added this to the Unreleased milestone Oct 16, 2019

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Oct 16, 2019

dmitshur added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 16, 2019

dmitshur changed the title ~~x/build/app/appengine: can't deploy anymore~~ x/build/app/appengine: deploying with dirty working tree affects the deploy Oct 31, 2019

bradfitz closed this as completed Nov 6, 2019

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 6, 2019

dmitshur mentioned this issue Jan 14, 2020

x/build/app/appengine: deploying with clean working tree affects the deploy because certain files are in .gitignore but not .gcloudignore #36553

Closed

golang locked and limited conversation to collaborators Jan 16, 2021

gopherbot added the FrozenDueToAge label Jan 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 •

edited

Loading

gopherbot commented Oct 16, 2019

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 •

edited

Loading

bradfitz commented Oct 16, 2019

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 •

edited

Loading

dmitshur commented Nov 6, 2019 •

edited

Loading

bradfitz commented Nov 6, 2019

dmitshur commented Jan 14, 2020 •

edited

Loading

xyzdevops commented Jan 17, 2020

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

Comments

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 • edited Loading

gopherbot commented Oct 16, 2019

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 • edited Loading

bradfitz commented Oct 16, 2019

bradfitz commented Oct 16, 2019

dmitshur commented Oct 16, 2019 • edited Loading

dmitshur commented Nov 6, 2019 • edited Loading

bradfitz commented Nov 6, 2019

dmitshur commented Jan 14, 2020 • edited Loading

xyzdevops commented Jan 17, 2020

dmitshur commented Oct 16, 2019 •

edited

Loading

dmitshur commented Oct 16, 2019 •

edited

Loading

dmitshur commented Oct 16, 2019 •

edited

Loading

dmitshur commented Nov 6, 2019 •

edited

Loading

dmitshur commented Jan 14, 2020 •

edited

Loading