[BUG] - Traefik rate limited by Let's Encrypt after several pod restarts

[sblair-metrostar]

Describe the bug

Traefik documentation warns to persist the acme.json certificate storage to avoid being rate limited by Let's Encrypt, but Nebari's deployment isn't configuring that. If the traefik pod is restarted enough times within a given week without this setup, Let's Encrypt will rate limit the certificate issue request and cause the site to default back to self-signed.

This has happened to me many, many times when I'm developing or testing changes to the platform or dealing with conda storage related node crashes, usually requiring a workaround involving installation of cert-manager and switching to zerossl or another cert issuing service until the Let's Encrypt backoff period elapses.

Expected behavior

traefik pod restarts should not result in new certificates being issued when configured for lets-encrypt.

Options:

1. Configure a persistent volume mount for the acme.json storage path.
2. Replace Traefik's native Let's Encrypt integration with cert-manager for more issuer options and improved scalability.

OS and architecture in which you are running Nebari

Linux, x64

How to Reproduce the problem?

1. Configure the Nebari certificate with lets-encrypt

certificate:
  type: lets-encrypt
  acme_email: notreal@example.com
  acme_server: https://acme-v02.api.letsencrypt.org/directory

2. Delete the traefik pod 5 or more times in 1 week.

Command output

No response

Versions and dependencies used.

Nebari 2023.11.1

Compute environment

AWS

Integrations

No response

Anything else?

No response

[Princess4ogb]

I am working on this ticket, Please find the gitlab issue as below.
https://gitlab.jatic.net/jatic/team-metrostar/t-e-platform/-/issues/347

[kcpevey]

Resolved by #2352