[DPE-3885] Enhance the error message if charm was deployed without --trust flag

[lucasgameiroborges]

Issue

#431

Solution

Considering this is an error that can be solved by human action directly on the live charm, I chose the following approach: If the 403 Unauthorized error is catched, the charm enters Blocked status with specific message and a helpful explanation about the problem is written in the logs:

juju status:

Unit                         Workload  Agent      Address       Ports  Message
untrusted-postgresql-k8s/0*  blocked   executing  10.1.110.75          Insufficient permissions, try: `juju trust untrusted-postgresql-k8s --scope=cluster`

juju logs:

unit-postgresql-k8s-0: 22:26:23 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-untrusted-postgresql-k8s-1: 13:24:43 ERROR unit.untrusted-postgresql-k8s/1.juju-log
            Access to k8s cluster resources is not authorized. This happens when RBAC is enabled and the deployed application was not trusted by the juju admin.
            To fix this issue, run `juju trust untrusted-postgresql-k8s --scope=cluster` (or remove & re-deploy untrusted-postgresql-k8s with `--trust`)

[dragomirp]

We should add integration test that the charm will block and unblock correctly.

[lucasgameiroborges]

Added a small set of integration tests for the feature. Due to RBAC not being enabled by default in our reusable workflows, I had to use sub-processes to interact with microk8s directly, as well as re-establish model connection after microk8s restart. Running the test multiple times locally and in CI without fails, it looks stable.

About the OSError: [Errno 9] Bad file descriptor error messages, I'm still not quite sure what triggers them (probably a consequence of microk8s interference), but they don't affect the test behavior in any way, besides spamming the CI logs. The evolution of the model state works as expected.

[marceloneppel]

It looks great! Thanks @lucasgameiroborges!

I left only one comment.

[taurus-forever]

Nice! Thank you!