secret revisions clean up

[ycliuhw]

Hi,
I noticed that this charm creates secrets, but it doesn’t implement handlers for the secret_removed event.
As a result, all obsolete revisions remain indefinitely, even if they’re no longer used or consumed by anyone.

We should consider implementing the cleanup hook by watching the secret_removed event.

Thanks!

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-8395.

This message was autogenerated

[ben-ballot]

My 2 cents on this:
I see in the juju secret lifecycle that the _on_secret_remove hook is sent when a new revision becomes obsolete: https://documentation.ubuntu.com/juju/3.6/reference/secret/#secret-lifecycle

That means adding a _on_secret_remove hook in the code like, in https://github.com/canonical/postgresql-operator/blob/main/lib/charms/data_platform_libs/v0/data_interfaces.py

Cheers

[ycliuhw]

Yes, then the charm will be able to clean up any obsolete/unused revisions by itself.

[kot0dama]

Could it be related to the k8s flavor similar issue canonical/postgresql-k8s-operator#529 ?

[jameinel]

@kot0dama Juju 2.9 doesn't have support for Juju secrets. However, for versions of Juju which do have secret support, we would, indeed, be creating Kubernetes secrets as a result of this, and not implementing secret-remove would mean that we would create lots of K8s secret artifacts that did not get cleaned up.

[manadart]

This issue has for some time and continues to, contribute to PS6 being on fire.

IS and others trying to diagnose and rectify the issues are being hampered by the behaviour of this charm. A swift resolution is desired by all.

[taurus-forever]

@manadart thank you for highlighting this, @marceloneppel has prepared PR (testing now), we will do our best to include it the next 14/stable we are working on with priority.

Can we manually clean old Juju Secret revisions on PS6 to decrease temperature there? Tnx!

[manadart]

Can we manually clean old Juju Secret revisions on PS6 to decrease temperature there? Tnx!

@jameinel worked with the SREs to do this. It made a significant improvement to performance, but we could see that they start to accumulate again quite quickly.

[jameinel]

After 1 week, we are essentially back to where we were. This was just before and just after we cleaned up the secret revisions (on Tuesday): [image: image.png] But this is the behavior now at the end of the week: [image: image.png] We can certainly run it again, but it does seem like it returns to baseline performance rather quickly. John =:->

…

On Fri, Oct 3, 2025 at 6:01 AM Joseph Phillips ***@***.***> wrote: *manadart* left a comment (canonical/postgresql-operator#1180) <#1180 (comment)> Can we manually clean old Juju Secret revisions on PS6 to decrease temperature there? Tnx! @jameinel <https://github.com/jameinel> worked with the SREs to do this. It made a significant improvement to performance, but we could see that they start to accumulate again quite quickly. — Reply to this email directly, view it on GitHub <#1180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABRQ7KBW4ZAERVQPAKYIL33VZCOXAVCNFSM6AAAAACHH3VR76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRVGEYDOOBZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jameinel]

Note that we also should look at why we are still creating new revisions. We were able to cleanup the old ones, and go from 60k down to around 5k. However, we are seeing growth of around 1-2k / day, which shouldn't be happening. In testing, I can confirm that calling secret-set with identical contents is not creating a new revision.
I tried triggering it as a map ordering issue (secret-set $secret foo=1 bar=2 in a loop, or alternating with bar=2 foo=1) and I was not able to trigger new revisions as long as all the secret values were supplied. Order did not seem to matter.

So there should be something about the content of the secret that is actually changing, causing us to create new revisions.

[taurus-forever]

Note that we also should look at why we are still creating new revisions. ...

Quick look on PostgreSQL 14/edge: adding/removing node member from cluster:

> diff -u <(juju show-secret d3f7fb07ln9deqvhit1g --reveal --revision 182) <(juju show-secret d3f7fb07ln9deqvhit1g --reveal --revision 98)
...
   content:
     password: secretpassword
-    read-only-uris: postgresql://relation-4:secretpassword@10.198.5.134:5432/postgresql_test_app_database
+    read-only-uris: postgresql://relation-4:secretpassword@10.198.5.121,10.198.5.134:5432/postgresql_test_app_database
     uris: postgresql://relation-4:secretpassword@10.198.5.237:5432/postgresql_test_app_database
     username: relation-4
 ...

Another secret change is a primary switchover:

> diff -u <(juju show-secret d3f7fb07ln9deqvhit1g --reveal --revision 182) <(juju show-secret d3f7fb07ln9deqvhit1g --reveal --revision 188)
...
   content:
     password: secretpassword
     read-only-uris: postgresql://relation-4:secretpassword@10.198.5.237:5432/postgresql_test_app_database
-    uris: postgresql://relation-4:secretpassword@10.198.5.237:5432/postgresql_test_app_database
+    uris: postgresql://relation-4:secretpassword@10.198.5.134:5432/postgresql_test_app_database
     username: relation-4
...

We are storing them as secrets due to credentials inside URI.
Do we see the classic Positive Feedback loop? PS6 instabilities => IP manipulations/switchovers => Juju Secrets => Juju Controller instabilities? CC: @jameinel