[MVP] Watch mode

[evanw]

I want esbuild to demonstrate that it's possible for a normal web development workflow to have high-performance tools. I consider watch mode a part of my initial MVP feature set for esbuild since I believe development builds should be virtually instant. This issue tracks the watch mode part of my MVP.

Watch mode involves automatically rebuilding when files on disk have been changed. While esbuild is much faster than other bundlers, very large code bases may still take around a second to build. I consider that too long for a development workflow. Watch mode will keep the previous build in memory and only rebuild the changed files between builds.

This may or may not involve disabling certain cross-file optimizations while in watch mode depending on what the performance looks like. I may also implement a local HTTP server that can be queried for the latest build, which is a nice way to avoid a page reload accidentally picking up a stale build.

[zmitry]

What about persistent cache and tool like this https://github.com/cortesi/modd ?

[progrium]

I was going to pull esbuild into my hotweb project which is basically this. Not sure how you'd like to separate concerns or if you want to just borrow each others code or what: https://github.com/progrium/hotweb

[gerardo-junior]

I do to do this very simply with browsersync with live reload

bs.watch('src/**/*.js', function (event, file) {
    require('esbuild').build({
         stdio: 'inherit',
         entryPoints: ['./src/scripts/index.js'],
         outfile: `${dist}/assets/scripts${!('development' === process.env.NODE_ENV) ? '.min' : ''}.js`,
         minify: !('development' === process.env.NODE_ENV),
         bundle: true,
         sourcemap: 'development' === process.env.NODE_ENV
    }).then(() => bs.reload())
      .catch(() => process.exit(1))
})

[zandaqo]

Watch mode involves automatically rebuilding when files on disk have been changed. While esbuild is much faster than other bundlers, very large code bases may still take around a second to build. I consider that too long for a development workflow. Watch mode will keep the previous build in memory and only rebuild the changed files between builds.

I understand the motivation and support it in principle, but I'd question prioritizing the watch mode over, say, tree shaking, aliases, or other vital features for a modern bundler.

With ~100x performance boost over existing bundlers, using esbuild without watch mode is already faster (e.g. 50 ms for me) than other bundlers operating in watch mode (750ms for me with Webpack). As it stands, though, without tree shaking esbuild produces bundles multiple times bigger than webpack, hence, I can only use it in development. But if speeding up my dev builds was my only goal, I would have gone with Snowpack... and it would make my already complex webpack setup a nightmare come update time.

That's where I think esbuild can help--reduce the complexity of our Snowpack+Webpack+Babel setups if it manages to give reasonable dev build speeds and feature coverage. Personally, I target modern browsers, use ES2019 and native web components with lit-element, thus, I require no compilation and ask my bundler only to, well, bundle all my code and minimize it at least by removing dead code I'm getting from third-party libraries.

[evanw]

I understand the motivation and support it in principle, but I'd question prioritizing the watch mode over, say, tree shaking, aliases, or other vital features for a modern bundler.

I’m totally with you. I’m actually already actively working on tree shaking, code splitting, and es6 export (they kind of have to all be done together since they are all related). Don’t worry!

[retronav]

A workaround for watch mode:
https://dev.to/obnoxiousnerd/watch-and-build-code-with-esbuild-2h6k

[tonyhb]

I'd rely on watchexec here - it's a really good utility to listen to file changes built with rust. Very lightweight, very fast, very simple.

[koenbok]

We have a fairly large code base that takes 1-2s to build, which is already amazing. But we're definitely interested to bring this down even more with a great watch mode.

[zmitry]

Any updates on this? I want to integrate esbuild into my development workflow (use esbuild for development and storybooks) but without watch it requires some extra effort to get it working and with 2s build time it's almost the same as with webpack.
As for plugins, I found that I don't really need that feature as well as css output, I could handle that externally with my existing setup. The only deal breaker is watch mod at the moment.

[koenbok]

Our initial results here are cold build (~60s), warm build (~20s) and rebuild (~0.5s) all became ~2s. This alone is enough for us to make it worth the switch, even in watch mode, because it seems like the cold build savings (after a branch switch) make it a net win. That said, having a real watch mode would be amazing.

[rtsao]

I suppose the first step here is figuring out the plan for file watching. Unfortunately, it looks like fsnotnify, the most popular Golang file watcher (from what I can tell) doesn't yet support FSEvents on macOS. I think without this, a tool like esbuild is almost certainly going to run into issues with file descriptor limits.

Personally, I think it'd be fine to just use watchman, but I can see how imposing a hard external dependency on users might be unpalatable. FWIW, Parcel implemented their own C++ file watcher to avoid a hard dependency on watchman.

[evanw]

I've heard a lot about how flaky file watching is, which makes me hesitate to integrate file watching into esbuild itself. The primary mechanism that should be in esbuild core is incremental builds and a way to trigger them.

Personally I've found that it's more reliable to trigger incremental builds for development by running a local server that does a build when a HTTP request comes in instead of using a file watcher. Then the file is never out of date due to a race between the build tool and the loading of the file. That's what I'm hoping to build first as far as watch mode. Perhaps something like esbuild --serve=localhost:8000? It should be easy to do watch mode correctly and reliably using a local server without the issues people normally encounter with file system watchers.

The hard part is doing incremental builds. Once that is in place, it should be relatively straightforward to allow external code to trigger an incremental build (e.g. with a signal) and then use that to hook an existing external file watching library to esbuild without having to build the file watcher into esbuild itself.

[haggholm]

Maybe a balance could be struck between “an actual project build MVP needs file watching” and “esbuild should not contain all the logic for file watching” by maintaining, say, a lightly curated set of recipes for things like “esbuild+watchman”, “esbuild+chokidar”, and other (not necessary watch related) things that should not be part of esbuild but are the kinds of questions that everyone using it is likely to ask?

[rtsao]

The simplest possible form of incremental builds I can think of would be to merely skip re-bundling chunks that should not be changed at all. At least in cases where the dependency graph itself is unchanged, it should be straightforward to identify which chunks should be ultimately unaffected.

This a rather coarse form of incremental builds, but in scenarios where the majority of the bundle is from node_modules, the amount of skipped work could be substantial (provided node_modules are in a separate chunk from the modified code). Given esbuild is already extremely fast, I wonder if it might be "good enough" in practice until something more granular could be implemented. I'm not too familiar with the bundler logic, but I wonder if this more limited form of incremental builds would be easier to implement without making too many changes.

[laurentpayot]

For now I'm watching my TS files with Chokidar CLI to trigger full esbuild builds at changes and it is so fast that it does the trick for me (I'm bundling Firebase with a web worker and a few files).

[mattdesl]

I'm noticing this as an issue with my project, bundle time once I include ThreeJS as a module is around ~500-700ms, which is pretty substantial during development compared to my old browserify tools (due to incremental loading they bundle in < 150 ms).

A watcher and server could be convenient to some users, but IMHO it would be better offloaded to another module/tool, and esbuild should just be a bundler. For example: my current use case wouldn't use esbuild's server or file watcher as I require a few custom things, and also I plan to make a pure client-side web version.

I'd rather an interface (in plugins?), a way to skip loading and transforming files, and instead use the already-transformed contents from my own program memory or from disk, i.e. a .cache directory that I would have set up in my custom tool atop esbuild.

[evanw]

Heads up that an incremental build API is coming soon (probably in the next release). Parsed ASTs are cached between builds and are not re-parsed if the contents don't change. Initial build time improvements are modest (~1.3x-1.5x faster) since linking still always happens, but further improvements are possible in the future.

[evanw]

The incremental build API has been released. Documentation is here: https://esbuild.github.io/api/#incremental.

[mattdesl]

Thanks Evan, in my simple ThreeJS test this drops the re-bundle time from ~100ms to ~40ms. 🎉

I now realize the reason my earlier builds were in the 500ms range was because of { sourcemap: true }, which doesn't seem to improve much with incremental bundling. I wonder whether it's possible to have sourcemaps for large modules also cached in some way, or is that not doable because the sourcemap has to cover the entire bundle?

EDIT:
Here's a test case showing the inline sourcemap performance with ThreeJS.

https://gist.github.com/mattdesl/f6a3192c89e1d182c26ceed28130e92c

[minheq]

I have just tried the new serve API. it's great! I was just a bit lost trying to figure out at what path did the server serve the bundle. Finally realized it is the same name as the entrypoint with .js extension!

anyway, esbuild is absolutely amazing!

[evanw]

I was just a bit lost trying to figure out at what path did the server serve the bundle.

Thanks for the feedback. I added more docs about this. The served files mirror the structure of what would be the output directory for a normal build.

One helpful trick is that the web server also has a directory listing feature. If you're serving on port 8000, you can visit http://localhost:8000/ to see all files in the root directory. From there you can click on links to browse around and see all of the different files.

[dmitryk-dk]

Hi, guys! I'm created some wrapper around esbuild and implemented watch mode for it on golang.
The realisation is here.
https://github.com/BrightLocal/FrontBuilder
Just don't judge too harshly))
Maybe it would be helpful.

[unki2aut]

The serve API looks very nice but I didn't see a hot reload option.

I guess this is just for convenience but a small script to do that could look like
https://gist.github.com/unki2aut/4ac81c33be2e8f121e80a26eba1735d7

[zaydek]

@unki2aut I got very curious about using hot reload like you’ve demonstrated so I built out a minimal reproducible repo for anyone here to play with: https://github.com/zaydek/esbuild-hot-reload.

Essentially, you just run yarn start; this fires up a local server based on the env variable PORT=... or falls back to 8080. Then any time you save changes on source TS / TSX files, like src/index.ts, the local server reloads without artificial delays.

Edit: I heavily annotated the esbuild source serve.js with code comments to help anyone out.

[mtgto]

@unki2aut 's script is awesome.

I find chokidar watch are called multiple times.
I update your script and add example to proxy API requests from React App.
https://gist.github.com/mtgto/47d9cbfe7d127dad2946ddfa241a2a1b

[mroderick]

In a project I used entr in combination with esbuild to automatically rebuild the project whenever files change.

It was fast enough to be unnoticeable, so for my use case, that was good enough.

Looking forward to seeing official support for incremental builds and seeing that manifest as either file watching or a server or both 🚀

[zaydek]

So I spent a lot of time researching the ‘watch’ problem, that is, why are file watchers a bad idea.

The way Evan implemented watch mode in esbuild took me a little while to understand. Essentially, he kicks off a long-lived HTTP server that incrementally recompiles your source code per request (e.g. browser refresh). This allows him to strategically delay the HTTP response so that there are no race conditions. This is intelligently designed but doesn’t solve for the use-case where you need to tell esbuild to compile / recompile based on file changes.

So I studied this for my own use-case and thought I’d share my notes here:

• File watching is not as trivial as it seems. For one OS, not the hardest problem in the world. But between many OS’s, the problem gets very gross, very quickly.
• Go almost implemented its own os/fsnotify package but it was eventually abandoned, likely due to unforeseen technical complexity / overhead.
• There are two classes of file watchers: you can poll files / directories every x milliseconds for changes (for example, you simply compare modification times) or you can do something much more low-level which is actually interfacing with the OS.
• If you want to talk to the OS (which should be more performant) and want your implementation to be platform-independent, the closest thing the Go community currently has is: https://github.com/fsnotify/fsnotify.
• If you don’t care about talking to the OS (your implementation will necessarily be less precise / slower) but still want your implementation to be platform-independent, https://github.com/radovskyb/watcher is a popular package that implements a polling-based solution.
• If you’re like me and want the simplest form-factor that you can actually understand / debug yourself, you can roll your own naive watcher in Go in <50 LOC:

package main

import (
	"fmt"
	"os"
	"path/filepath"
	"time"
)

func main() {
	var (
		// Maps modification timestamps to path names.
		modMap = map[string]time.Time{}
		
		// Channel that simply notifies for any recursive change.
		ch     = make(chan struct{})
	)

	// Starts a goroutine that polls every 100ms.
	go func() {
		for range time.Tick(100 * time.Millisecond) {
			// Walk directory `src`. This means we are polling recursively.
			if err := filepath.Walk("src", func(path string, info os.FileInfo, err error) error {
				if err != nil {
					return err
				}

				// Get the current path’s modification time; if no such modification time
				// exists, simply create a first write.
				if prev, ok := modMap[path]; !ok {
					modMap[path] = info.ModTime()
				} else {
					// Path has been modified; therefore get the new modification time and
					// update the map. Finally, emit an event on our channel.
					if next := info.ModTime(); prev != next {
						modMap[path] = next
						ch <- struct{}{}
					}
				}
				return nil
			}); err != nil {
				panic(err)
			}
		}
	}()

	for range ch {
		fmt.Println("something changed")
	}
}

This doesn’t tell you what or why something changed -- simply that something did. For my use case, this is probably more than enough. And you can still parametrize the polling interval.

Anyway, I hope this helps a soul. 🤠

[chowey]

Another option, which is built on the HTTP server approach, is to keep track of the modtime of all files that were used in the last build. Then, when the new request comes, to first check if any of those files have a different modtime. If none have a different modtime, then you can re-use the result from last time.

Due to how the operating system caches os.Stat(), this seems to be performant. Here's an untested simple implementation:

const metafile = "_internal_metadata.json"

type ESBuildHandler struct {
	options  api.BuildOptions
	result   api.BuildResult
	outdir   string
	modified map[string]time.Time
	l        sync.RWMutex
}

func NewESBuildHandler(options api.BuildOptions) *ESBuildHandler {
	h := &ESBuildHandler{options: options}
	// Use incremental building.
	h.options.Incremental = true
	// Export metadata so we know which files were accessed.
	h.options.Metafile = metafile
	// Keep track of the outdir so we can resolve incoming paths.
	if outdir, err := filepath.Abs(h.options.Outdir); err == nil {
		h.outdir = outdir
	}
	return h
}

func (h *ESBuildHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	name := filepath.Join(h.outdir, r.URL.Path)

	if h.needsRegenerate() {
		h.regenerate()
	}

	h.l.RLock()
	defer h.l.RUnlock()
	for _, file := range h.result.OutputFiles {
		if file.Path == name {
			http.ServeContent(w, r, r.URL.Path, time.Time{}, bytes.NewReader(file.Contents))
			return
		}
	}

	http.NotFound(w, r)
}

func (h *ESBuildHandler) needsRegenerate() bool {
	h.l.RLock()
	defer h.l.RUnlock()

	if h.modified == nil || len(h.modified) == 0 {
		return true
	}

	for path, modtime := range h.modified {
		fi, err := os.Stat(path)
		if err != nil || !fi.ModTime().Equal(modtime) {
			return true
		}
	}

	return false
}

func (h *ESBuildHandler) regenerate() {
	h.l.Lock()
	defer h.l.Unlock()

	if h.result.Rebuild != nil {
		h.result = h.result.Rebuild()
	} else {
		h.result = api.Build(h.options)
	}

	// Keep track of modtimes.
	h.modified = make(map[string]time.Time)
	for _, file := range h.result.OutputFiles {
		if strings.HasSuffix(file.Path, metafile) {
			var metadata struct {
				Inputs map[string]struct{} `json:"inputs"`
			}
			json.Unmarshal(file.Contents, &metadata)
			for input := range metadata.Inputs {
				if fi, err := os.Stat(input); err == nil {
					h.modified[input] = fi.ModTime()
				}
			}
			return
		}
	}
}

[evanw]

Watch mode has just been released in version 0.8.38. From the release notes:

With this release, you can use the --watch flag to run esbuild in watch mode which watches the file system for changes and does an incremental build when something has changed. The watch mode implementation uses polling instead of OS-specific file system events for portability.

Note that it is still possible to implement watch mode yourself using esbuild's incremental build API and a file watcher library of your choice if you don't want to use a polling-based approach. Also note that this watch mode feature is about improving developer convenience and does not have any effect on incremental build time (i.e. watch mode is not faster than other forms of incremental builds).

The new polling system is intended to use relatively little CPU vs. a traditional polling system that scans the whole directory tree at once. The file system is still scanned regularly but each scan only checks a random subset of your files to reduce CPU usage. This means a change to a file will be picked up soon after the change is made but not necessarily instantly. With the current heuristics, large projects should be completely scanned around every 2 seconds so in the worst case it could take up to 2 seconds for a change to be noticed. However, after a change has been noticed the change's path goes on a short list of recently changed paths which are checked on every scan, so further changes to recently changed files should be noticed almost instantly.

More documentation including information about API options is available here: https://esbuild.github.io/api/#watch.

[DrSensor]

In case someone using watchexec, this is my workaround for integrating it

const $rebuild = flag('--rebuild-on', '-r')

const { build } = require('esbuild')
    , pkg = require('../package.json')
    , { compilerOptions: tsc, ...tsconfig } = require('../tsconfig.json')

/** @type {import('esbuild').BuildOptions} */
const options = {
    format: 'cjs',
    platform: 'node',
    entryPoints: ['src/main.ts'],
    bundle: true,
    outdir: tsc.outDir,
    sourcemap: true,

    incremental: $rebuild.exists,
};

(async () => {
    const { join, dirname: dir, basename: base, extname: ext } = require('path')
        , { entryPoints, outdir, incremental } = options

    const es = await build(options)

    if (incremental) process.on($rebuild.values[0], () =>
        es.rebuild())
})()

This makes esbuild do rebuild on specific signal

watchexec -nc --signal SIGCHLD -w src/ -- build.js --rebuild-on SIGCHLD

[deanc]

@evanw I'm having some issues trying this out. I have a fairly straightforward build config:

const esbuild = require("esbuild");
const sassPlugin = require("esbuild-plugin-sass");

const watchMode = process.env.ESBUILD_WATCH === "true" || false;
if(watchMode) {
  console.log("Running in watch mode");
}

esbuild
  .build({
    entryPoints: ["./entry.jsx"],
    bundle: true,
    minify: true,
    sourcemap: true,
    watch: watchMode,
    outfile: "./bundle.esbuild.js",
    define: { "process.env.NODE_ENV": '"production"' },
    target: ["es2020"],
    plugins: [sassPlugin()],
    loader: { '.ttf': 'file' },
  })
  .catch(() => process.exit(1));

Watch mode is being passed in as boolean true but the process exits, with no error - as if it's not "watching". Am I misunderstanding how this is supposed to function?

Same result if I add this part also:

)
  .then((result) => {
    // Call "stop" on the result when you're done
    result.stop();
  })

[zaydek]

If anyone is interested in implementing server-sent events (in Go) with the new esbuild watch mode (this enables auto-reloading the browser tab on source changes), check this out: https://github.com/zaydek/esbuild-watch-demo.

@evanw The watch mode works great! I’m pleased with your implementation.

This is an awesome API. I don’t need to orchestrate rebuilds anymore and watching ‘just works’ because esbuild is already aware what the source files are.

[fabiospampinato]

I just found out that it takes like ~100ms for "chokidar" to notify me of updates, ~50ms for my "watcher" to do the same (no idea why it's half the time), and that's using the native filesystem watcher that Node gives us access to under macOS!

These are kind of ridiculous numbers really. By the time Node is able to tell me that something changed esbuild has already finished rebuilding the entire thing 🤣

So thanks for adding a watch mode I guess, not just because it's actually usable for many use cases, but also because it makes other alternatives seem incredibly slow.