Quieter HTTP 206 logging:

[jacobweinstock]

Description

Logging all 206 partial content requests from the middleware package creates a lot of messages per ISO request. In testing this was around 3000 requests. This is too many logs and will create too much noise.

There is an HTTP middleware in Smee that logs a single line for all requests. This PR stops the middleware logger from logging anything when the X-Global-Logging header is set in the HTTP response. This allows HTTP handler functions to decide if middleware logging should occur or not.

For the ISO HTTP handler function we disable the middleware logging and only sample the HTTP 206 request messages. The following is an example output from an ISO mount using the 0.002% sampling rate. My estimate is that this sampling rate us about 5 - 10 log messages per ISO mount.

The first 8 lines are from when the system is booting up the ISO mounted as a virtual CDROM/DVD. The last 3 lines are from when HookOS boots up and mounts the ISO as a CDROM device.

{"level":"info","ts":1732046233.118308,"caller":"iso/iso.go:195","msg":"magic string found, patching the content","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:35630","totalCount":48,"contentRange":"bytes 195100672-195133439/195420160"}
{"level":"info","ts":1732046241.343277,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:58796","totalCount":512}
{"level":"info","ts":1732046246.5436597,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:35488","totalCount":828}
{"level":"info","ts":1732046257.9578667,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:60176","totalCount":1424}
{"level":"info","ts":1732046269.1992714,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:46840","totalCount":2089}
{"level":"info","ts":1732046273.146468,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:49070","totalCount":2334}
{"level":"info","ts":1732046282.2612996,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:39434","totalCount":2897}
{"level":"info","ts":1732046282.5963988,"caller":"iso/iso.go:152","msg":"HTTP GET method received with a 206 status code","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:39606","totalCount":2917}
{"level":"info","ts":1732046385.6623178,"caller":"iso/iso.go:142","msg":"HTTP HEAD method received","method":"HEAD","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:36474","totalCount":3010}
{"level":"info","ts":1732046393.1938548,"caller":"iso/iso.go:142","msg":"HTTP HEAD method received","method":"HEAD","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:36502","totalCount":3012}
{"level":"info","ts":1732046393.6070883,"caller":"iso/iso.go:195","msg":"magic string found, patching the content","method":"GET","urlPath":"/hook-latest-lts-x86_64-efi-initrd.iso/iso/de-ad-be-ef-fe-ed/hook.iso","xff":"192.168.2.205, 192.168.2.205","remoteAddr":"192.168.2.205:36814","totalCount":3045,"contentRange":"bytes 195035136-195158015/195420160"}

Also added is some checking/validation around content-size. A max content size of 512Kb is implemented.

And finally, the Tiltfile was updated and implements live updates. Any associated kustomize directories and files were removed and deploying Smee is done via the stack helm chart.

Why is this needed

Fixes: #

How Has This Been Tested?

How are existing users impacted? What migration steps/scripts do we need?

Checklist:

I have:

• updated the documentation and/or roadmap (if required)
• added unit or e2e tests
• provided instructions on how to upgrade

[codecov]

Codecov Report

Attention: Patch coverage is 37.50000% with 55 lines in your changes missing coverage. Please review.

Project coverage is 50%. Comparing base (e74e9fa) to head (edd662d).
Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/iso/iso.go	40%	41 Missing and 7 partials ⚠️
internal/ipxe/http/middleware.go	0%	6 Missing ⚠️
cmd/smee/main.go	0%	1 Missing ⚠️

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #550   +/-   ##
===================================
  Coverage    49%    50%           
===================================
  Files        27     27           
  Lines      2951   2975   +24     
===================================
+ Hits       1461   1502   +41     
+ Misses     1447   1421   -26     
- Partials     43     52    +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests