Middlware to generate and set ETag header

[davejohnston]

Hi,
I'm trying to create a middleware that will generate an ETag from the request body and set the header.
I was looking at the Before function to let me write the header, but the problem I'm facing is how to access the request body bytes?

c.Response().Before(func() {
   body := <get body somehow?>
    etag := generateEtag(body)
    c.Response().Header().Set("ETag", etag))
})

My http handlers return return c.JSON(http.StatusOK, data)
Is there anyway to access data before it gets marshalled and written ?

[aldas]

BodyDump might give you some ideas

echo/middleware/body_dump.go

Lines 76 to 78 in 82a964c

	mw := io.MultiWriter(c.Response().Writer, resBody)
	writer := &bodyDumpResponseWriter{Writer: mw, ResponseWriter: c.Response().Writer}
	c.Response().Writer = writer

although this is not that memory efficient if you buffer everything in memory and then calculate etag.

[pablor21]

I have the same question.
This is what I have so far

func(c echo.Context) (err error) {
	if config.Skipper(c) {
		return next(c)
	}
	// Response
	resBody := new(bytes.Buffer)
	mw := io.MultiWriter(c.Response().Writer, resBody)
	writer := &etagResponseWriter{Writer: mw, ResponseWriter: c.Response().Writer}
	c.Response().Writer = writer

	if err = next(c); err != nil {
		return err
	}

	c.Response().Header().Set("ETag", "some etag")

	return
}

The etagResponseWriter is just a copy of bodyDumpResponseWriter from body_dump middleware.
The problem I'm facing is that the ETag header is never written in to the response.

Somebody have any idea of how to fix this?

[aldas]

First thing I would recommend is to check when execution reaches c.Response().Header().Set("ETag", "some etag") if request (body) has already been sent (c.Response().Committed) to the client. As HTTP headers are written send to the client before body, adding/removing headers does not work - because you can not modify something that has already been sent.

Chunked responses have concept of Trailer headers https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer but I have not played enough with standard library to know if these are implemented or how exactly to use them.

[aldas]

Also - it is always worth searching how others have implemented it

First repo I found is https://github.com/go-http-utils/etag

[pablor21]

Yes, that's the problem, I'm trying to buffer the response instead of write to the connection, but seems that it doesn't work because Echo writes the response directly to the underlying connection, that's good for memory I guess, but makes things like ETags a very hard thing to achieve...
I'll continue investigating, maybe I can find something...
As a workaround right now, I'm writing the etags from the route handler...

[aldas]

yep, this is little bit chicken and the egg problem. Headers are written before (should be known before) the content is composed/written to the client. So you need to a) buffer for etag calculation b) read data twice - first for etag and second time when passed to the response writer.

p.s. if you response is not completely dynamic - you could cache etags so only first request to "resource" is buffered for etag calculation.

[pablor21]

Thanks to @aldas suggestions and hints I came up with a solution that may be useful for somebody looking to write etags, the solution is working good so far but you must know that the response will be buffered before is sent to the client (This is obvious since there is no way to calculate the etag in dynamic responses without knowing the content of the response)

Here is the middleware code

package middlewares

import (
	"bytes"
	"crypto/sha1"
	"encoding/hex"
	"fmt"
	"hash"
	"hash/crc32"
	"net/http"
	"strconv"

	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

type (
	// EtagConfig defines the config for Etag middleware.
	EtagConfig struct {
		// Skipper defines a function to skip middleware.
		Skipper middleware.Skipper
		// Weak defines if the Etag is weak or strong.
		Weak bool
		// HashFn defines the hash function to use. Default is crc32q.
		HashFn func(config EtagConfig) hash.Hash
	}
)

var (
	// DefaultEtagConfig is the default Etag middleware config.
	DefaultEtagConfig = EtagConfig{
		Skipper: middleware.DefaultSkipper,
		Weak:    true,
		HashFn: func(config EtagConfig) hash.Hash {
			if config.Weak {
				const crcPol = 0xD5828281
				crc32qTable := crc32.MakeTable(crcPol)
				return crc32.New(crc32qTable)
			} else {
				return sha1.New()
			}
		},
	}
	normalizedETagName        = http.CanonicalHeaderKey("Etag")
	normalizedIfNoneMatchName = http.CanonicalHeaderKey("If-None-Match")
	weakPrefix                = "W/"
)

// Etag returns a Etag middleware.
func Etag() echo.MiddlewareFunc {
	c := DefaultEtagConfig
	return EtagWithConfig(c)
}

// EtagWithConfig returns a Etag middleware with config.
func EtagWithConfig(config EtagConfig) echo.MiddlewareFunc {

	if config.Skipper == nil {
		config.Skipper = DefaultEtagConfig.Skipper
	}

	return func(next echo.HandlerFunc) echo.HandlerFunc {
		return func(c echo.Context) (err error) {
			skipper := config.Skipper
			if skipper == nil {
				skipper = DefaultEtagConfig.Skipper
			}

			if skipper(c) {
				return next(c)
			}

			// get the hash function
			hashFn := config.HashFn
			if hashFn == nil {
				hashFn = DefaultEtagConfig.HashFn
			}

			originalWriter := c.Response().Writer
			res := c.Response()
			req := c.Request()
			// ResponseWriter
			hw := bufferedWriter{rw: res.Writer, hash: hashFn(config), buf: bytes.NewBuffer(nil)}
			res.Writer = &hw
			err = next(c)
			// restore the original writer
			res.Writer = originalWriter
			if err != nil {
				return err
			}

			resHeader := res.Header()

			if hw.hash == nil ||
				resHeader.Get(normalizedETagName) != "" ||
				strconv.Itoa(hw.status)[0] != '2' ||
				hw.status == http.StatusNoContent ||
				hw.buf.Len() == 0 {
				writeRaw(originalWriter, hw)
				return
			}

			etag := fmt.Sprintf("%v-%v", strconv.Itoa(hw.len),
				hex.EncodeToString(hw.hash.Sum(nil)))

			if config.Weak {
				etag = weakPrefix + etag
			}

			resHeader.Set(normalizedETagName, etag)

			ifNoneMatch := req.Header.Get(normalizedIfNoneMatchName) // get the If-None-Match header
			headerFresh := ifNoneMatch == etag || ifNoneMatch == weakPrefix+etag

			if headerFresh {
				originalWriter.WriteHeader(http.StatusNotModified)
				originalWriter.Write(nil)
			} else {
				writeRaw(originalWriter, hw)
			}
			return
		}
	}
}

// bufferedWriter is a wrapper around http.ResponseWriter that
// buffers the response and calculates the hash of the response.
type bufferedWriter struct {
	rw     http.ResponseWriter
	hash   hash.Hash
	buf    *bytes.Buffer
	len    int
	status int
}

// Header returns the header map that will be sent by
func (hw bufferedWriter) Header() http.Header {
	return hw.rw.Header()
}

// WriteHeader sends an HTTP response header with the provided status code.
func (hw *bufferedWriter) WriteHeader(status int) {
	hw.status = status
}

// Write writes the data to the buffer to be sent as part of an HTTP reply.
func (hw *bufferedWriter) Write(b []byte) (int, error) {
	if hw.status == 0 {
		hw.status = http.StatusOK
	}
	// write to the buffer
	l, err := hw.buf.Write(b)
	if err != nil {
		return l, err
	}
	// write to the hash
	l, err = hw.hash.Write(b)
	hw.len += l
	return l, err
}

// WriteTo writes the buffered data to the underlying io.Writer.
func writeRaw(res http.ResponseWriter, hw bufferedWriter) {
	res.WriteHeader(hw.status)
	res.Write(hw.buf.Bytes())
}

[aldas]

NB: if you are dealing with files you could read them twice - first with reader that calculates etag on the fly, only reads like 4kb etc chunks to keep needed memory down and second time - when the file is sent to the client. In that way you are not holding everything (file contents) in the memory + add caching for etag so you do not need to calculate it for each request

[pablor21]

To send files I think is way better to send the last-modified header, the ETag requirement came up because I'm serving json responses from a list that I obtain from the database, when I have only one resource to send I can use last-modified because I can keep track of the updates in the database, but when I have a list of entities/models (or however you want to call it) I have no way to know the modified date since there could be deletions or updates

[pablor21]

If somebody hits this problem, I've just published this solution as a package so it's easy to use in other projects:
https://pkg.go.dev/github.com/pablor21/echo-etag/v4
https://github.com/pablor21/echo-etag

to install:

go get github.com/pablor21/echo-etag/v4

ps: I'm not sure if this is a good place to publish this, @aldas feel free to modify or delete this post if something is wrong or I should not publish something I've done (and is not maintained by echo maintainers/colaborators) in your repo

[aldas]

This is OK.

NB: do not forget add tests and CI flows. This is how Echo does CI: https://github.com/labstack/echo/tree/master/.github/workflows

[pablor21]

Thank you for your help.
Just added tests and workflows