File Embedding in Go

Talk at Golang Leipzig November Meetup

What is File Embedding?

• packing arbitrary resources into a binary/executable
• a resource can be anything, e.g. HTML templates, CSS or Javascript files, a favicon, translations...
• example layout of a typical web application:

├── assets
│   ├── base.css
│   ├── base.js
│   ├── ...
│   └── favicon.svg
├── migrations
│   ├── 01_initial_setup.down.sql
│   └── ...
├── notes.go
├── storage.go
├── storage_test.go
└── views
    ├── index.gohtml
    ├── ...
    └── layouts
        └── base.gohtml

Advantages 👍

• only a single file needs to be distributed/deployed
• especially useful for desktop applications (no need for $XDG_HOME, %APPDATA% or ~/Library)
• embedded data is "immutable" (at least from outside the process)

Disadvantages 👎

• binary size grows with each resource that gets embedded → increased memory usage
• space efficiency depends on the encoding of the embedded content
  • base64 wastes about 1/3 for encoding
• large files are not suitable for embedding

If the resources are compressable---e.g. most plain text files are---then a binary packer like upx can reduce the file size dramatically:

$ upx -o notes.upx notes
$ du -sh notes notes.upx 
24M     notes
9.7M    notes.upx

Available Implementations

• go-bindata most popular but now deprecated, recommends pkger
• pkger
  • requires a Go modules project
  • API simulates os.File
  • 👎 stores a lot of redundant metadata, e.g. absolute paths to files 👮‍♀️ or imported packages
• statik
  • simulates an http.FileSystem---ok for some use cases but pretty otherwise inconvenient
  • 👎 can only include a single directory
• go.rice
  • 👎 complicated to use, scans your source code for rice.FindBox("/path") calls and includes them
  • 👎 weird limitations, e.g. paths cannot be constant strings or undefined behaviour when called in init()
  • failed to set it up for my example application, I really wonder why this got popular 🤷‍♂️

It's Time for Yet Another File Embedding Tool

• embed treats Not Invented Here syndrome and enforces dogfooding
• tiny prototype implementation (still a single Go file)

Design

• very simple API that provides methods for:
  • listing embedded files
  • get file as bytes
  • get file as string
• no error return values
  • if a file is not found the default value is returned ([]byte{} or "")
  • it is very easy to test if the expected files were actually embedded
• embed generates a single embeds.go file that implements the API (example)
• filenames and contents are stored as base64 encoded strings
  • compression adds a lot of overhead (additional dependency, runtime cost, etc.)
  • use a binary packer if in doubt
• provides a golang-migrate driver

Usage

• check klingtnet/embed/examples or klingtnet/notes for a real world example

NAME:
   embed - A new cli application

USAGE:
   embed [global options] command [command options] [arguments...]

VERSION:
   unset

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --package value, -p value                    name of the package the generated Go file is associated to (default: "main")
   --destination value, --dest value, -d value  where to store the generated Go file (default: "embeds.go")
   --include value, -i value                    paths to embed, directories are stored recursively (can be used multiple times)
   --help, -h                                   show help (default: false)
   --version, -v                                print the version (default: false)

Draft for Embedded Static Files

The draft---published by Russ Cox and Brad Fitzpatrick in July---identified the following problems with the current situation:

• too many existing tools 😅
• all depend on a manual generation step
• generated file bloats git history (with a second slightly larger copy of each file)
• not go-gettable if generated file is not checked into source control

Adding support for file embedding to the go command will eliminate those problems.

Proposed changes:

• new //go:embed directive
• embed package containing embed.Files that implements fs.FS ^{file system interface draft} which makes it directly usable with net/http and hmtl/template

Here's an example for the embed directive:

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.Files

The draft also contains various considerations regarding:

• directory traversal, e.g. ../../../etc/passwd should be prevented by disallowing paths above the module root
• content compression is discussed but left open as an implementation detail

Still curious?

• watch the draft presentation video, proposal pull request and its follow up adoption pull request
• there was also the idea to just have a magic directory called static
• should embed include hidden files?