bundle: new docs, command-line interface

Before, "bundle -help" printed only one not particularly useful line:

	Usage of bundle:

and "bundle" printed a bit more but not a standard usage message:

	bundle: Usage: bundle package dest prefix

	Arguments:
	 package is the import path of the package to concatenate.
	 dest is the import path of the package in which the resulting file will reside.
	 prefix is the string to attach to all renamed identifiers.

And most of that output belonged in (but was missing from) the doc comment,
which leaned heavily on the reader inferring usage from one example.

While fixing all this, I ended up dropping "dest" and "prefix" as required
arguments: perfectly good defaults can be inferred in the common case
where the bundle is intended for the current directory. The defaults can be
overridden with the new -dst and -prefix option.

I corrected a TODO about not assuming the package name could be
derived from the destination import path. Like -dst and -prefix, the
default package name can be inferred from the current directory,
but there is also a new -pkg option to override the default if needed.

I added a -o option to specify the destination output file (standard output
is still the default output). One benefit of -o is that it makes the bundle
commands self-contained in the sense of not needing a shell.
That in turn makes them suitable for use with "go generate", so when -o
is specified the bundle output now includes a "//go:generate" comment
for updating the bundle mechanically.

To keep bundle working for net/http's use case, I added a -importmap
option to specify additional import rewrites (net/http needs the http2
import of golang.org/x/net/http2/hpack rewritten to begin with internal/golang.org).

The net effect of all of this is that the first two lines at the top of
net/http/h2_bundle.go will change from:

	// Code generated by golang.org/x/tools/cmd/bundle command:
	//   $ bundle golang.org/x/net/http2 net/http http2

to

	// Code generated by golang.org/x/tools/cmd/bundle.
	//go:generate bundle -o h2_bundle.go -prefix http2 -import golang.org/x/net/http2/hpack=internal/golang.org/x/net/http2/hpack golang.org/x/net/http2

and net/http's copy of http2 can now be updated by "go generate"
instead of by copying and pasting commands from source code to shell.

For an experiment I am doing with cmd/dist bundling archive/zip,
the header is even shorter and makes a better demonstration of the
power of the defaults:

	// Code generated by golang.org/x/tools/cmd/bundle.
	//go:generate bundle -o zip.go archive/zip

New usage message, printed by "bundle -help" and "bundle":

	Usage: bundle [options] <src>
	  -dst path
	    	set destination import path (default taken from current directory)
	  -import map
	    	rewrite import using map, of form old=new (can be repeated)
	  -o file
	    	write output to file (default standard output)
	  -pkg name
	    	set destination package name (default taken from current directory)
	  -prefix p
	    	set bundled identifier prefix to p (default source package name + "_")

See CL for new doc comment.

Following this CL I will send a CL updating the two-line header in net/http/h2_bundle.go
to match the new usage of the bundle, whatever that ends up being.

Change-Id: I83a65b6a500897854027cefdefb8f62f1d4448b4
Reviewed-on: https://go-review.googlesource.com/19428
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>

cmd/bundle/main.go

@@ -4,34 +4,81 @@
 
 // +build go1.5
 
-// The bundle command concatenates the source files of a package,
-// renaming package-level names by adding a prefix and renaming
-// identifiers as needed to preserve referential integrity.
-//
-// Example:
-// 	$ bundle golang.org/x/net/http2 net/http http2
-//
-// The command above prints a single file containing the code of
-// golang.org/x/net/http2, suitable for inclusion in package net/http,
-// in which toplevel names have been prefixed with "http2".
-//
-// Assumptions:
-// - no file in the package imports "C", that is, uses cgo.
-// - no file in the package has GOOS or GOARCH build tags or file names.
-// - comments associated with the package or import declarations,
-//   may be discarded, as may comments associated with no top-level
-//   declaration at all.
-// - neither the original package nor the destination package contains
-//   any identifiers starting with the designated prefix.
-//   (This allows us to avoid various conflict checks.)
-// - there are no renaming imports.
-// - test files are ignored.
-// - none of the renamed identifiers is significant
-//   to reflection-based logic.
-//
-// Only package-level var, func, const, and type objects are renamed,
-// and embedded fields of renamed types.  No methods are renamed, so we
-// needn't worry about preserving interface satisfaction.
+// Bundle creates a single-source-file version of a source package
+// suitable for inclusion in a particular target package.
+//
+// Usage:
+//
+//	bundle [-o file] [-dst path] [-pkg name] [-prefix p] [-import old=new] <src>
+//
+// The src argument specifies the import path of the package to bundle.
+// The bundling of a directory of source files into a single source file
+// necessarily imposes a number of constraints.
+// The package being bundled must not use cgo; must not use conditional
+// file compilation, whether with build tags or system-specific file names
+// like code_amd64.go; must not depend on any special comments, which
+// may not be preserved; must not use any assembly sources;
+// must not use renaming imports; and must not use reflection-based APIs
+// that depend on the specific names of types or struct fields.
+//
+// By default, bundle writes the bundled code to standard output.
+// If the -o argument is given, bundle writes to the named file
+// and also includes a ``//go:generate'' comment giving the exact
+// command line used, for regenerating the file with ``go generate.''
+//
+// Bundle customizes its output for inclusion in a particular package, the destination package.
+// By default bundle assumes the destination is the package in the current directory,
+// but the destination package can be specified explicitly using the -dst option,
+// which takes an import path as its argument.
+// If the source package imports the destination package, bundle will remove
+// those imports and rewrite any references to use direct references to the
+// corresponding symbols.
+// Bundle also must write a package declaration in the output and must
+// choose a name to use in that declaration.
+// If the -package option is given, bundle uses that name.
+// Otherwise, if the -dst option is given, bundle uses the last
+// element of the destination import path.
+// Otherwise, by default bundle uses the package name found in the
+// package sources in the current directory.
+//
+// To avoid collisions, bundle inserts a prefix at the beginning of
+// every package-level const, func, type, and var identifier in src's code,
+// updating references accordingly. The default prefix is the package name
+// of the source package followed by an underscore. The -prefix option
+// specifies an alternate prefix.
+//
+// Occasionally it is necessary to rewrite imports during the bundling
+// process. The -import option, which may be repeated, specifies that
+// an import of "old" should be rewritten to import "new" instead.
+//
+// Example
+//
+// Bundle archive/zip for inclusion in cmd/dist:
+//
+//	cd $GOROOT/src/cmd/dist
+//	bundle -o zip.go archive/zip
+//
+// Bundle golang.org/x/net/http2 for inclusion in net/http,
+// prefixing all identifiers by "http2" instead of "http2_",
+// and rewriting the import "golang.org/x/net/http2/hpack"
+// to "internal/golang.org/x/net/http2/hpack":
+//
+//	cd $GOROOT/src/net/http
+//	bundle -o h2_bundle.go \
+//		-prefix http2 \
+//		-import golang.org/x/net/http2/hpack=internal/golang.org/x/net/http2/hpack \
+//		golang.org/x/net/http2
+//
+// Two ways to update the http2 bundle:
+//
+//	go generate net/http
+//
+//	cd $GOROOT/src/net/http
+//	go generate
+//
+// Update both bundles, restricting ``go generate'' to running bundle commands:
+//
+//	go generate -run bundle cmd/dist net/http
 //
 package main
 
@@ -45,51 +92,105 @@ import (
 	"go/parser"
 	"go/token"
 	"go/types"
-	"io"
+	"io/ioutil"
 	"log"
 	"os"
-	"path/filepath"
+	"path"
 	"strings"
 
 	"golang.org/x/tools/go/loader"
 )
 
+var (
+	outputFile = flag.String("o", "", "write output to `file` (default standard output)")
+	dstPath    = flag.String("dst", "", "set destination import `path` (default taken from current directory)")
+	pkgName    = flag.String("pkg", "", "set destination package `name` (default taken from current directory)")
+	prefix     = flag.String("prefix", "", "set bundled identifier prefix to `p` (default source package name + \"_\")")
+
+	importMap = map[string]string{}
+)
+
+func init() {
+	flag.Var(flagFunc(addImportMap), "import", "rewrite import using `map`, of form old=new (can be repeated)")
+}
+
+func addImportMap(s string) {
+	if strings.Count(s, "=") != 1 {
+		log.Fatal("-import argument must be of the form old=new")
+	}
+	i := strings.Index(s, "=")
+	old, new := s[:i], s[i+1:]
+	if old == "" || new == "" {
+		log.Fatal("-import argument must be of the form old=new; old and new must be non-empty")
+	}
+	importMap[old] = new
+}
+
+func usage() {
+	fmt.Fprintf(os.Stderr, "Usage: bundle [options] <src>\n")
+	flag.PrintDefaults()
+	os.Exit(2)
+}
+
 func main() {
 	log.SetPrefix("bundle: ")
 	log.SetFlags(0)
 
+	flag.Usage = usage
 	flag.Parse()
 	args := flag.Args()
-	if len(args) != 3 {
-		log.Fatal(`Usage: bundle package dest prefix
-
-Arguments:
- package is the import path of the package to concatenate.
- dest is the import path of the package in which the resulting file will reside.
- prefix is the string to attach to all renamed identifiers.
-`)
+	if len(args) != 1 {
+		usage()
 	}
-	initialPkg, dest, prefix := args[0], args[1], args[2]
 
-	if err := bundle(os.Stdout, initialPkg, dest, prefix); err != nil {
-		log.Fatal(err)
+	if *dstPath != "" {
+		if *pkgName == "" {
+			*pkgName = path.Base(*dstPath)
+		}
+	} else {
+		wd, _ := os.Getwd()
+		pkg, err := build.ImportDir(wd, 0)
+		if err != nil {
+			log.Fatal("cannot find package in current directory: %v", err)
+		}
+		*dstPath = pkg.ImportPath
+		if *pkgName == "" {
+			*pkgName = pkg.Name
+		}
+	}
+
+	code := bundle(args[0], *dstPath, *pkgName, *prefix)
+	if *outputFile != "" {
+		err := ioutil.WriteFile(*outputFile, code, 0666)
+		if err != nil {
+			log.Fatal(err)
+		}
+	} else {
+		_, err := os.Stdout.Write(code)
+		if err != nil {
+			log.Fatal(err)
+		}
 	}
 }
 
 var ctxt = &build.Default
 
-func bundle(w io.Writer, initialPkg, dest, prefix string) error {
+func bundle(src, dst, dstpkg, prefix string) []byte {
 	// Load the initial package.
 	conf := loader.Config{ParserMode: parser.ParseComments, Build: ctxt}
-	conf.TypeCheckFuncBodies = func(p string) bool { return p == initialPkg }
-	conf.Import(initialPkg)
+	conf.TypeCheckFuncBodies = func(p string) bool { return p == src }
+	conf.Import(src)
 
 	lprog, err := conf.Load()
 	if err != nil {
 		log.Fatal(err)
 	}
 
-	info := lprog.Package(initialPkg)
+	info := lprog.Package(src)
+	if prefix == "" {
+		pkgName := info.Files[0].Name.Name
+		prefix = pkgName + "_"
+	}
 
 	objsToUpdate := make(map[types.Object]bool)
 	var rename func(from types.Object)
@@ -122,8 +223,13 @@ func bundle(w io.Writer, initialPkg, dest, prefix string) error {
 
 	var out bytes.Buffer
 
-	fmt.Fprintf(&out, "// Code generated by golang.org/x/tools/cmd/bundle command:\n")
-	fmt.Fprintf(&out, "//   $ bundle %s %s %s\n\n", initialPkg, dest, prefix)
+	fmt.Fprintf(&out, "// Code generated by golang.org/x/tools/cmd/bundle.\n")
+	if *outputFile != "" {
+		fmt.Fprintf(&out, "//go:generate bundle %s\n", strings.Join(os.Args[1:], " "))
+	} else {
+		fmt.Fprintf(&out, "//   $ bundle %s\n", strings.Join(os.Args[1:], " "))
+	}
+	fmt.Fprintf(&out, "\n")
 
 	// Concatenate package comments from all files...
 	for _, f := range info.Files {
@@ -136,8 +242,7 @@ func bundle(w io.Writer, initialPkg, dest, prefix string) error {
 	// ...but don't let them become the actual package comment.
 	fmt.Fprintln(&out)
 
-	// TODO(adonovan): don't assume pkg.name == basename(pkg.path).
-	fmt.Fprintf(&out, "package %s\n\n", filepath.Base(dest))
+	fmt.Fprintf(&out, "package %s\n\n", dstpkg)
 
 	// Print a single declaration that imports all necessary packages.
 	// TODO(adonovan):
@@ -153,7 +258,11 @@ func bundle(w io.Writer, initialPkg, dest, prefix string) error {
 	}
 	fmt.Fprintln(&out, "import (")
 	for _, p := range info.Pkg.Imports() {
-		if p.Path() == dest {
+		if p.Path() == dst {
+			continue
+		}
+		if x, ok := importMap[p.Path()]; ok {
+			fmt.Fprintf(&out, "\t%q\n", x)
 			continue
 		}
 		fmt.Fprintf(&out, "\t%q\n", p.Path())
@@ -181,7 +290,7 @@ func bundle(w io.Writer, initialPkg, dest, prefix string) error {
 			if sel, ok := n.(*ast.SelectorExpr); ok {
 				if id, ok := sel.X.(*ast.Ident); ok {
 					if obj, ok := info.Uses[id].(*types.PkgName); ok {
-						if obj.Imported().Path() == dest {
+						if obj.Imported().Path() == dst {
 							id.Name = "@@@"
 						}
 					}
@@ -217,6 +326,14 @@ func bundle(w io.Writer, initialPkg, dest, prefix string) error {
 		log.Fatalf("formatting failed: %v", err)
 	}
 
-	_, err = w.Write(result)
-	return err
+	return result
 }
+
+type flagFunc func(string)
+
+func (f flagFunc) Set(s string) error {
+	f(s)
+	return nil
+}
+
+func (f flagFunc) String() string { return "" }