x/tools/godoc: Dockerize and convert to run on GAE Flex #27205

andybons · 2018-08-24T21:48:24Z

godoc relies on https://golang.org/doc/go1.10#goroot but GAE Classic is still running Go 1.9.

Convert to use GAE Flex so that we can rid ourselves of these limitations (and deploy godoc from the release-branch.go1.11)

/cc @bradfitz @dmitshur @katiehockman

The text was updated successfully, but these errors were encountered:

bradfitz · 2018-08-24T21:56:00Z

Milestone: Go 1.6 :)

bradfitz · 2018-08-24T23:01:35Z

Also, because we're using a Go 1.10 x/tools on golang.org now, certain links don't work from the release notes & blog post, like:

https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more

Because the godoc heading rules. This should be a heading:

Modules, module versions, and more

But it renders as a plain paragraph at the moment.

dmitshur · 2018-08-24T23:32:37Z

Similarly, this statement from https://golang.org/doc/go1.11#godoc is not yet true on https://golang.org/pkg/:

The godoc web server now shows which version of Go introduced new API features.

As a workaround, in the meantime, you can refer to https://tip.golang.org/pkg/. E.g.:

https://tip.golang.org/pkg/os/#UserCacheDir

steren · 2018-08-30T20:29:12Z

FYI, Go 1.11 on the App Engine standard environment is coming.

bradfitz · 2018-08-30T20:35:29Z

@steren, that doesn't help us. App Engine's Go runtime by definition will always be behind what we need, because we have to release Go before App Engine can release a Go runtime for that version. So we can't use App Engine's provided runtimes. We can only do a custom runtime (BYODockerfile).

So we're moving off App Engine classic.

steren · 2018-08-30T21:12:05Z

I see, yes in that case, a Dockerfile approach seems to be appropriate.

broady · 2018-08-30T23:06:10Z

Checklist of things to do.

Create Dockerfile
Update deployment process/script to use env: flex
Consider introducing another build flag to include "golang.org" stuff (short, dl). Currently the "appengine" build tag is used for this (see appinit.go). An environment variable might be even better.

Done already: networking/routing. We did this work already about a year ago.

Cannot use "google.golang.org/appengine" packages in Flex, so these must be migrated. Significant packages/code that needs to be updated:

cmd/godoc/appinit.go (main entrypoint for golang.org)
godoc/dl (downloads page uses datastore/memcache)
godoc/short (short URL facility, uses datastore/memcache)

I guess the main questions are what we use to replace the memcache bits in dl and short. It's probably right to use golang.org/x/playground's cache.

Relevant deps of cmd/godoc, which will all need to be checked for any appengine stuff:

golang.org/x/tools/blog
golang.org/x/tools/blog/atom
golang.org/x/tools/godoc
golang.org/x/tools/godoc/analysis
golang.org/x/tools/godoc/dl
golang.org/x/tools/godoc/proxy
golang.org/x/tools/godoc/redirect
golang.org/x/tools/godoc/short
golang.org/x/tools/godoc/static
golang.org/x/tools/godoc/util
golang.org/x/tools/godoc/vfs
golang.org/x/tools/godoc/vfs/httpfs
golang.org/x/tools/godoc/vfs/mapfs
golang.org/x/tools/godoc/vfs/zipfs
golang.org/x/tools/present

gopherbot · 2018-09-04T23:57:41Z

Change https://golang.org/cl/133355 mentions this issue: godoc: migrate to App Engine flexible

broady · 2018-09-04T23:59:54Z

One more thing we need to do: move the shortlink admin service to another GAE app or just a local command.

It uses appengine/user to check the viewer has admin access. There's no simple migration for that (need to set up OAuth/Firebase Auth).

bradfitz · 2018-09-05T09:58:36Z

How about we migrate the shortlinks to a plain text file stored in git?

The admin service is broken even for classic App Engine on custom domains anyway and when I want to add a new link, I spend more time remembering/finding the ugly workaround URL than I would if I just TBR'd a CL to a data file.

We can cache the data file in memory and on cache miss (for new links) fault to Gerrit for fetching the file at HEAD, rate limited to once every 5 minutes or so.

broady · 2018-09-05T13:41:27Z

I like that, it also gives non-admins a list of available shortlinks.

Where should the list live? x/tools/godoc/short/links.tsv?

bradfitz · 2018-09-05T14:13:01Z

That works. But slight preference for links.txt(or links.dat) instead and not requiring tabs. I'd just use any whitespace. Ignore comment lines starting with '#' otherwise just use strings.Fields and get first element is short key and second is full URL.

bradfitz · 2018-09-25T12:56:02Z

It's now been over a month for this "Soon" bug.

What remains?

andybons · 2018-09-26T18:08:34Z

Ping @broady, who last I heard had a working version locally and was working on the deploy scripts. He's in Sydney at the moment so there may be some delay, though.

broady · 2018-09-28T00:53:46Z

broady · 2018-09-28T07:56:37Z

Update: well, I sank several hours into figuring out why the package listing was empty.

It looks like there have been numerous changes to x/tools/godoc/vfs, my best guess is CL 101295.

Precisely, this change to zipfs:
golang/tools@16d1af8#diff-6b09edd71f61887ea342b82eb207149b

/cc @agnivade

dmitshur · 2018-11-01T20:16:57Z

It seems this issue is mostly resolved, with a minor cleanup followup task left. We should either remove the Soon label, since I don't think it applies now, or close this and open a new issue for the remaining cleanup task.

(/cc @andybons since you added the Soon label.)

andybons · 2018-11-01T20:25:25Z

Removed soon label

gopherbot · 2018-11-20T21:10:34Z

Change https://golang.org/cl/150599 mentions this issue: [release-branch.go1.11] godoc: migrate to App Engine flexible

gopherbot · 2018-11-20T22:03:24Z

Change https://golang.org/cl/150678 mentions this issue: [release-branch.go1.11] cmd/godoc: improve deployment scripts, add buildinfo

gopherbot · 2018-11-20T22:03:26Z

Change https://golang.org/cl/150679 mentions this issue: [release-branch.go1.11] cmd/godoc: re-enable host checking, allow test versions

gopherbot · 2018-11-20T22:03:29Z

Change https://golang.org/cl/150682 mentions this issue: [release-branch.go1.11] cmd/godoc: add cloud build config

gopherbot · 2018-11-20T22:03:30Z

Change https://golang.org/cl/150681 mentions this issue: [release-branch.go1.11] cmd/godoc: add make publish to migrate traffic

gopherbot · 2018-11-20T22:03:32Z

Change https://golang.org/cl/150680 mentions this issue: [release-branch.go1.11] cmd/godoc: move regression tests to a go test

See bug for more details on exactly what was migrated. Notably: * No more Google-internal deployment scripts; see README.godoc-app and the Makefile for details. * Build tag "golangorg" is used for the godoc configuration used for golang.org. * Use of App Engine libraries replaced with GCP client libraries. * Redis is used to replace App Engine memcache. * Google analytics is controlled by an environment variable. * Regression tests have been migrated from Google-internal. * hg -> git hash map is moved from Google-internal. Updates golang/go#28893 Updates golang/go#27205 Change-Id: Ia0a983f239c50eda8be2363494c8b784f60c2c6d Reviewed-on: https://go-review.googlesource.com/133355 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150599

…ildinfo * Build Go from a given version (make.bash) * Add a /buildinfo file that describes the inputs of the build/deployment. * Use Makefile/environment variables to override Go version and Docker tag. Updates golang/go#28893 Updates golang/go#27205 Change-Id: Ia7a88b75f9d5b2319d2381e56bc963eb53e889c7 Reviewed-on: https://go-review.googlesource.com/c/138978 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150678

…t versions test.golang.org is no longer -- instead allow access to version-specific App Engine URLs (like 20181002t1342-dot-golang-org.appspot.com). App Engine Flex uses the X-Forwarded-Proto to signify the proto used by the originating request (it always uses h1 on 8080 when proxying the request). Updates golang/go#28893 Updates golang/go#27205 Change-Id: I423ffe65df325500a2fa04c7b655797ecc6ad037 Reviewed-on: https://go-review.googlesource.com/c/139237 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150679

Run them separately from the other tests in godoc_test by requiring a regtest.host flag and by filtering on the test name. Updates golang/go#28893 Updates golang/go#27205 Change-Id: I166d2278a3f6954307f7c935567a81e73f78e7bb Reviewed-on: https://go-review.googlesource.com/c/139238 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150680

Also rename `make build` and `make push` to `make docker-build` and `make docker-push` in preparation to introduce Cloud Build (removing the dependency on Docker). Updates golang/go#28893 Updates golang/go#27205 Change-Id: Iae19b9a6f77d09246a1332c7ec9eceec449cdba8 Reviewed-on: https://go-review.googlesource.com/c/139239 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150681

Deploys no longer depend on Docker. With only Make and gcloud installed, the following should deploy a new version: $ git clone https://go.googlesource.com/tools $ cd tools $ cd cmd/godoc $ make cloud-build deploy Updates golang/go#28893 Updates golang/go#27205 Change-Id: I5cc1142e02dc288450d55dbd4da4b30c0a080bd5 Reviewed-on: https://go-review.googlesource.com/c/139240 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/c/150682

dmitshur · 2019-01-30T05:51:56Z

The task described in the original issue has long been complete. This issue has remained opened because there was one remaining clean up task (see #27205 (comment)):

remove build tag from main.go (i.e., merge appinit.go and main.go)

The work happening in #29206 obsoletes the need for this to be done. /cc @cnoellekb After the golang.org website is deployed from x/website, x/tools/cmd/godoc will be reduced in scope to just serve documentation locally, and its appinit.go will be simply deleted.

That work is better tracked by other open issues, so I'll close this as done.

agnivade · 2019-01-30T06:14:05Z

There were a few others also which weren't done.

remove zipping from build process (generate-index.bash)
ability to add shortlinks (old app had a login, GAE Flex doesn't have authed handlers)

Not sure if they would be obsoleted too.

dmitshur · 2019-01-30T06:24:30Z

Thanks for pointing that out, I didn't see those two.

remove zipping from build process (generate-index.bash)

I think this one will either be obsolete or will naturally come up as part of the post-move cleanup process. ~~I don't see a need to track it via a dedicated issue, but if someone wants to, feel free to open one.~~ Edit: This opportunity is now captured in an open issue at #29206 (comment).

ability to add shortlinks (old app had a login, GAE Flex doesn't have authed handlers)

I was going to open a new issue for that. I saw that https://golang.org/s pointed to this issue, but didn't find that task which related for it. Opened #29988 to track this.

gopherbot · 2019-01-30T06:41:41Z

Change https://golang.org/cl/160377 mentions this issue: godoc/short: point to new tracking issue for shortlink creation

There's a new dedicated issue tracking this task, point to it instead of the previous large issue which has been closed by now. Updates golang/go#29988 Updates golang/go#27205 Change-Id: Ib1443d14a6369322b36cdf8305344a35c421a2e5 Reviewed-on: https://go-review.googlesource.com/c/160377 Reviewed-by: Agniva De Sarker <agniva.quicksilver@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot · 2019-01-30T17:45:08Z

Change https://golang.org/cl/160477 mentions this issue: [release-branch.go1.11] godoc/short: point to new tracking issue for shortlink creation

…shortlink creation There's a new dedicated issue tracking this task, point to it instead of the previous large issue which has been closed by now. Updates golang/go#29988 Updates golang/go#27205 Change-Id: Ib1443d14a6369322b36cdf8305344a35c421a2e5 Reviewed-on: https://go-review.googlesource.com/c/160377 Reviewed-by: Agniva De Sarker <agniva.quicksilver@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> (cherry picked from commit 1c35819) Reviewed-on: https://go-review.googlesource.com/c/160477 Reviewed-by: Andrew Bonventre <andybons@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org>

andybons added NeedsFix The path to resolution is known, but the work has not been done. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) labels Aug 24, 2018

andybons added this to the Unreleased milestone Aug 24, 2018

andybons self-assigned this Aug 24, 2018

This was referenced Aug 25, 2018

doc: link to main entry point for modules documentation is not working and various module sections are not recognized as headers #27209

Closed

doc: website not showing Go1.11 release notes link #27206

Closed

bradfitz mentioned this issue Aug 25, 2018

doc: downloads page claims that Go 1.11 supports unsupported operating systems #27213

Closed

FiloSottile mentioned this issue Aug 27, 2018

cmd/godoc: golang.org out of date #27284

Closed

tklauser mentioned this issue Aug 28, 2018

Invalid links in https://golang.org/doc/go1.11#modules #27307

Closed

agnivade mentioned this issue Aug 29, 2018

Section markup missing for gofmt in https://golang.org/cmd/go/ #27334

Closed

andybons mentioned this issue Aug 30, 2018

website: long TTFB wait for /pkg/* #27057

Closed

broady self-assigned this Sep 5, 2018

dmitshur mentioned this issue Sep 10, 2018

tour: Dead link at that page golang/tour#597

Closed

agnivade mentioned this issue Sep 11, 2018

cmd/go: documentation for 'fmt' subcommand is not detected as a section #27604

Closed

andybons removed the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Nov 1, 2018

dmitshur mentioned this issue Nov 20, 2018

x/tools/godoc: backport Flex changes onto release-branch.go1.11 #28893

Closed

dmitshur closed this as completed Jan 30, 2019

dmitshur mentioned this issue Jan 30, 2019

website: ability to add shortlinks #29988

Closed

1 task

dmitshur mentioned this issue Feb 21, 2019

website: move to App Engine flexible #21628

Closed

golang locked and limited conversation to collaborators Jan 30, 2020

gopherbot added the FrozenDueToAge label Jan 30, 2020

rsc unassigned broady and andybons Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/godoc: Dockerize and convert to run on GAE Flex #27205

x/tools/godoc: Dockerize and convert to run on GAE Flex #27205

andybons commented Aug 24, 2018

bradfitz commented Aug 24, 2018

bradfitz commented Aug 24, 2018

dmitshur commented Aug 24, 2018

steren commented Aug 30, 2018

bradfitz commented Aug 30, 2018

steren commented Aug 30, 2018

broady commented Aug 30, 2018 •

edited

Loading

gopherbot commented Sep 4, 2018

broady commented Sep 4, 2018

bradfitz commented Sep 5, 2018

broady commented Sep 5, 2018 •

edited

Loading

bradfitz commented Sep 5, 2018

bradfitz commented Sep 25, 2018

andybons commented Sep 26, 2018 •

edited

Loading

broady commented Sep 28, 2018 •

edited

Loading

broady commented Sep 28, 2018

dmitshur commented Nov 1, 2018

andybons commented Nov 1, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

dmitshur commented Jan 30, 2019 •

edited

Loading

agnivade commented Jan 30, 2019 •

edited

Loading

dmitshur commented Jan 30, 2019 •

edited

Loading

gopherbot commented Jan 30, 2019

gopherbot commented Jan 30, 2019

x/tools/godoc: Dockerize and convert to run on GAE Flex #27205

x/tools/godoc: Dockerize and convert to run on GAE Flex #27205

Comments

andybons commented Aug 24, 2018

bradfitz commented Aug 24, 2018

bradfitz commented Aug 24, 2018

dmitshur commented Aug 24, 2018

steren commented Aug 30, 2018

bradfitz commented Aug 30, 2018

steren commented Aug 30, 2018

broady commented Aug 30, 2018 • edited Loading

gopherbot commented Sep 4, 2018

broady commented Sep 4, 2018

bradfitz commented Sep 5, 2018

broady commented Sep 5, 2018 • edited Loading

bradfitz commented Sep 5, 2018

bradfitz commented Sep 25, 2018

andybons commented Sep 26, 2018 • edited Loading

broady commented Sep 28, 2018 • edited Loading

broady commented Sep 28, 2018

dmitshur commented Nov 1, 2018

andybons commented Nov 1, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

gopherbot commented Nov 20, 2018

dmitshur commented Jan 30, 2019 • edited Loading

agnivade commented Jan 30, 2019 • edited Loading

dmitshur commented Jan 30, 2019 • edited Loading

gopherbot commented Jan 30, 2019

gopherbot commented Jan 30, 2019

broady commented Aug 30, 2018 •

edited

Loading

broady commented Sep 5, 2018 •

edited

Loading

andybons commented Sep 26, 2018 •

edited

Loading

broady commented Sep 28, 2018 •

edited

Loading

dmitshur commented Jan 30, 2019 •

edited

Loading

agnivade commented Jan 30, 2019 •

edited

Loading

dmitshur commented Jan 30, 2019 •

edited

Loading