Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetch-lib. Fix for #315 #327

Merged
merged 4 commits into from
Apr 26, 2024
Merged

Fetch-lib. Fix for #315 #327

merged 4 commits into from
Apr 26, 2024

Conversation

sed-i
Copy link
Contributor

@sed-i sed-i commented Apr 25, 2024 •
edited
Loading

#315 had an issue with tls_certificates v3. Checking here so noctua doesn't force push on my commit.

In this PR:

  • Make sure scheme and certs are set on the same condition.
  • mkdir -p "plugins", "notifiers", "alerting" to reduce "error" level logs on startup.
  • Fetch lib.

@sed-i sed-i marked this pull request as draft April 25, 2024 17:34
@sed-i sed-i changed the title Fix for #315 Fetch-lib. Fix for #315 Apr 25, 2024
@sed-i
Copy link
Contributor Author

sed-i commented Apr 25, 2024

Tests passed locally, but grafana had a brief error:

unit-grafana-1: 13:58:18 ERROR unit.grafana/1.juju-log certificates:2: Could not restart grafana at this time: cannot perform the following tasks:
- Start service "grafana" (cannot start service: exited quickly with code 1)
----- Logs from task 0 -----
2024-04-25T17:58:18Z INFO Most recent service output:
    (...)
    logger=provisioning.plugins t=2024-04-25T17:58:18.516812528Z level=error msg="Failed to read plugin provisioning files from directory" path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory"
    logger=provisioning.notifiers t=2024-04-25T17:58:18.516834278Z level=error msg="Can't read alert notification provisioning files from directory" path=/etc/grafana/provisioning/notifiers error="open /etc/grafana/provisioning/notifiers: no such file or directory"
    logger=provisioning.alerting t=2024-04-25T17:58:18.516849248Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
    logger=provisioning.alerting t=2024-04-25T17:58:18.516855357Z level=info msg="starting to provision alerting"
    logger=provisioning.alerting t=2024-04-25T17:58:18.516862268Z level=info msg="finished to provision alerting"
    logger=modules t=2024-04-25T17:58:18.516971128Z level=warn msg="No modules registered..."
    logger=ngalert.state.manager t=2024-04-25T17:58:18.516974078Z level=info msg="Warming state cache for startup"
    logger=grafanaStorageLogger t=2024-04-25T17:58:18.517046737Z level=info msg="storage starting"
    logger=ngalert.state.manager t=2024-04-25T17:58:18.517146777Z level=info msg="State cache has been initialized" states=0 duration=172.249µs
    logger=ngalert.multiorg.alertmanager t=2024-04-25T17:58:18.517188197Z level=info msg="starting MultiOrg Alertmanager"
    logger=ticker t=2024-04-25T17:58:18.517197237Z level=info msg=starting first_tick=2024-04-25T17:58:20Z
    logger=server t=2024-04-25T17:58:18.518468975Z level=error msg="Stopped background service" service=*api.HTTPServer reason="cert_file cannot be empty when using HTTPS"
    logger=tracing t=2024-04-25T17:58:18.518571835Z level=info msg="Closing tracing"
    logger=ticker t=2024-04-25T17:58:18.518727614Z level=info msg=stopped last_tick=2024-04-25T17:58:10Z
    logger=grafana.update.checker t=2024-04-25T17:58:18.518759164Z level=error msg="Update check failed" error="failed to get latest.json repo from github.com: Get \"https://raw.githubusercontent.com/grafana/grafana/main/latest.json\": context canceled" duration=1.539637ms
    logger=auth t=2024-04-25T17:58:18.521339439Z level=error msg="failed to lock and execute cleanup of expired auth token" error="context canceled"
    logger=serviceaccounts t=2024-04-25T17:58:18.522056807Z level=warn msg="Failed to get usage metrics" error="context canceled"
    logger=infra.usagestats t=2024-04-25T17:58:18.522125307Z level=error msg="Failed to get last sent time" error="context canceled"
    logger=infra.usagestats.collector t=2024-04-25T17:58:18.522235617Z level=error msg="Failed to get system stats" error="context canceled"
    Error: ✗ *api.HTTPServer run error: cert_file cannot be empty when using HTTPS
2024-04-25T17:58:18Z ERROR cannot start service: exited quickly with code 1

Specifically:

  • open /etc/grafana/provisioning/plugins: no such file or directory
  • open /etc/grafana/provisioning/notifiers: no such file or directory
  • cert_file cannot be empty when using HTTPS

Seems like the first two are non-critical errors, and the empty cert_file is what brings it down.

Context:

25 Apr 2024 13:56:42-04:00  juju-unit  allocating   
25 Apr 2024 13:56:42-04:00  workload   waiting      installing agent
25 Apr 2024 13:56:54-04:00  workload   waiting      agent initialising
25 Apr 2024 13:57:28-04:00  workload   maintenance  installing charm software
25 Apr 2024 13:57:28-04:00  juju-unit  executing    running install hook
25 Apr 2024 13:57:30-04:00  juju-unit  executing    running grafana-relation-created hook
25 Apr 2024 13:57:30-04:00  juju-unit  executing    running certificates-relation-created hook
25 Apr 2024 13:57:31-04:00  juju-unit  executing    running replicas-relation-created hook
25 Apr 2024 13:57:32-04:00  juju-unit  executing    running leader-elected hook
25 Apr 2024 13:57:33-04:00  juju-unit  executing    running grafana-pebble-ready hook
25 Apr 2024 13:57:37-04:00  juju-unit  executing    running litestream-pebble-ready hook
25 Apr 2024 13:57:38-04:00  juju-unit  executing    running database-storage-attached hook
25 Apr 2024 13:58:04-04:00  juju-unit  executing    running config-changed hook
25 Apr 2024 13:58:08-04:00  juju-unit  executing    running start hook
25 Apr 2024 13:58:10-04:00  workload   unknown      
25 Apr 2024 13:58:10-04:00  juju-unit  executing    running litestream-pebble-ready hook
25 Apr 2024 13:58:10-04:00  juju-unit  executing    running grafana-pebble-ready hook

unit-grafana-1: 13:58:12 INFO unit.grafana/1.juju-log Restarted grafana-k8s
unit-grafana-1: 13:58:13 INFO unit.grafana/1.juju-log Restarted grafana-k8s
unit-grafana-1: 13:58:14 INFO unit.grafana/1.juju-log Initializing dashboard provisioning path
unit-grafana-1: 13:58:15 INFO unit.grafana/1.juju-log Restarted grafana-k8s

25 Apr 2024 13:58:15-04:00  juju-unit  executing    running replicas-relation-joined hook for grafana/0


(BOOM)

unit-grafana-1: 13:58:18 ERROR unit.grafana/1.juju-log certificates:2: Could not restart grafana at this time: cannot perform the following tasks:
- Start service "grafana" (cannot start service: exited quickly with code 1)
...


25 Apr 2024 13:58:16-04:00  juju-unit  executing    running certificates-relation-changed hook
25 Apr 2024 13:58:19-04:00  juju-unit  executing    running certificates-relation-joined hook for ca/0
25 Apr 2024 13:58:19-04:00  juju-unit  idle         
25 Apr 2024 13:58:19-04:00  juju-unit  executing    running replicas-relation-changed hook for grafana/0
25 Apr 2024 13:58:20-04:00  juju-unit  executing    running certificates-relation-changed hook for ca/0
25 Apr 2024 13:58:23-04:00  juju-unit  idle         
25 Apr 2024 13:58:24-04:00  juju-unit  executing    running grafana-relation-changed hook
25 Apr 2024 13:58:25-04:00  juju-unit  idle         
25 Apr 2024 13:58:25-04:00  juju-unit  executing    running grafana-relation-joined hook for grafana/0
25 Apr 2024 13:58:26-04:00  juju-unit  executing    running grafana-relation-changed hook for grafana/0
25 Apr 2024 13:58:58-04:00  juju-unit  idle         
25 Apr 2024 13:58:59-04:00  juju-unit  executing    running replicas-relation-changed hook for grafana/0
25 Apr 2024 13:59:00-04:00  juju-unit  executing    running certificates-relation-changed hook
25 Apr 2024 13:59:10-04:00  juju-unit  idle         
25 Apr 2024 13:59:10-04:00  workload   active       
25 Apr 2024 13:59:16-04:00  juju-unit  executing    running replicas-relation-changed hook for grafana/0
25 Apr 2024 13:59:17-04:00  juju-unit  idle         
25 Apr 2024 14:00:25-04:00  workload   maintenance  stopping charm software
25 Apr 2024 14:00:25-04:00  juju-unit  executing    running stop hook
25 Apr 2024 14:00:26-04:00  workload   maintenance  Application is terminating.
25 Apr 2024 14:00:26-04:00  workload   maintenance  
25 Apr 2024 14:01:03-04:00  juju-unit  executing    running upgrade-charm hook
25 Apr 2024 14:01:15-04:00  juju-unit  executing    running config-changed hook
25 Apr 2024 14:01:20-04:00  juju-unit  executing    running start hook
25 Apr 2024 14:01:22-04:00  juju-unit  executing    running grafana-pebble-ready hook
25 Apr 2024 14:01:31-04:00  juju-unit  idle         
25 Apr 2024 14:01:31-04:00  juju-unit  executing    running litestream-pebble-ready hook
25 Apr 2024 14:01:32-04:00  juju-unit  executing    running certificates-relation-changed hook
25 Apr 2024 14:01:35-04:00  juju-unit  idle         
25 Apr 2024 14:01:36-04:00  juju-unit  executing    running grafana-relation-changed hook
25 Apr 2024 14:01:37-04:00  juju-unit  idle         
25 Apr 2024 14:02:07-04:00  juju-unit  executing    running replicas-relation-changed hook for grafana/0
25 Apr 2024 14:02:07-04:00  juju-unit  executing    running certificates-relation-changed hook
25 Apr 2024 14:02:20-04:00  juju-unit  idle         
25 Apr 2024 14:02:20-04:00  workload   active       
25 Apr 2024 14:02:23-04:00  juju-unit  executing    running replicas-relation-changed hook for grafana/0
25 Apr 2024 14:02:24-04:00  juju-unit  idle

@sed-i sed-i marked this pull request as ready for review April 25, 2024 19:53
PietroPasotti
PietroPasotti approved these changes Apr 26, 2024
View reviewed changes
Copy link
Contributor

@PietroPasotti PietroPasotti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! Thanks a lot

@PietroPasotti PietroPasotti merged commit 23c37dc into main Apr 26, 2024
13 checks passed
@PietroPasotti PietroPasotti deleted the chore/auto-libs-mod branch April 26, 2024 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PietroPasotti PietroPasotti PietroPasotti approved these changes

@Abuelodelanada Abuelodelanada Awaiting requested review from Abuelodelanada

@dstathis dstathis Awaiting requested review from dstathis

@IbraAoad IbraAoad Awaiting requested review from IbraAoad

@lucabello lucabello Awaiting requested review from lucabello

@mmkay mmkay Awaiting requested review from mmkay

@simskij simskij Awaiting requested review from simskij

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sed-i @PietroPasotti @observability-noctua-bot