Backport of agent: prevent very old servers re-joining a cluster with stale data into release/1.15.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #17171 to be assessed for backporting due to the inclusion of the label backport/1.15.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@loshz
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Description

This PR introduces the concept of Server Metadata, which is server specific information written to a file stored in the configured data directory. During this initial phase, we only store the last seen timestamp in Unix format:

consul/agent/consul/server_metadata.go

Lines 17 to 19 in 5103a1a

	type ServerMetadata struct {
	LastSeenUnix int64 `json:"last_seen_unix"`
	}

2 new agent methods for attempting to prevent old servers from rejoining an existing cluster.

• persistServerMetadata(): Periodically write a server's metadata to a file in the configured data directory every hour.
• checkServerLastSeen(): Attempt to read a server's last seen file and check the Unix timestamp against a configurable max age. If the last seen file does not exist, we treat this as an initial startup and return no error.

Example

We attempt to start a previously running server with the following last seen timestamp 1672531200 (2023-Jan-01 00:00:00).

Setting the new config to server_rejoin_age_max = "3d" will prevent this server from starting and require an operator to manually remove this file from the data directory if they need to force the server to rejoin.

Testing & Reproduction steps

• Added unit tests for new Server Metatdata helper funcs.
• Added unit tests for agent Server Metatdata implementation.

Links

• https://hashicorp.atlassian.net/browse/NET-3772
• https://github.com/hashicorp/consul/blob/main/docs/config/checklist-adding-config-fields.md

PR Checklist

• Write validation function for new config field.
• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

---

Overview of commits

• c8efed4 - 38cb8ca - 15e1e38 - 9baff64 - f672df4 - 45371a2 - 5ef0eb6 - ec03223 - 7ede1e5 - 4d52023 - 03adbe2 - 9ebc6a1 - 5103a1a - 0102b59 - ce52131 - 3474e20 - 21e26a0 - d0beddd - 1928587 - 4332107 - 2480556 - 7178431 - 5c74255 - bd54fa0

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR

[loshz]

Replaced by #17357