No longer receiving logs from agent

[tcurdt]

We've been ingesting logs (into Grafana Cloud) for months with the attached configuration.
It suddenly stopped and logs are not showing up at all anymore. Metrics are still showing up.

$ grafana-agent -version
agent, version v0.40.3 (branch: HEAD, revision: b0f0bcdad)
  build user:       root@ba9b2b4409f7
  build date:       2024-03-14T15:57:31Z
  go version:       go1.22.1
  platform:         linux/amd64
  tags:             netgo,builtinassets,promtail_journal_enabled

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

We are seeing this in the agent logs

Mar 03 06:49:05 node01 grafana-agent[932]: ts=2024-03-03T05:49:05.326953038Z caller=client.go:430 level=error component=logs logs_config=agent component=client host=logs-prod-us-central1.grafana.net msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): entry for stream '{container=\"caddy-f4947cb4-25dc-2ea3-d62d-9fea683e9653\", job=\"integrations/docker\", service=\"loadbalancer\", stream=\"stderr\"}' has timestamp too old: 2024-02-23T21:47:02Z, oldest acceptable timestamp is: 2024-02-25T05:49:05Z"

Mar 21 13:11:42 node01 grafana-agent[932]: ts=2024-03-21T12:11:42.792930071Z caller=target.go:100 level=error component=logs logs_config=agent target=docker/50b1bbd8ad579b637ed73d8b6a21f21112832ff67569594d3104095c1aef3aaf msg="could not inspect container info" container=50b1bbd8ad579b637ed73d8b6a21f21112832ff67569594d3104095c1aef3aaf err="Error response from daemon: No such container: 50b1bbd8ad579b637ed73d8b6a21f21112832ff67569594d3104095c1aef3aaf"

We haven't changed anything. Here is the config we been using for months.

grafana.txt