[prometheus] Securing Prometheus API and UI endpoints using basic auth in Container #1255

prab2014 · 2021-08-17T08:40:07Z

Team,

I am running Prometheus on containers using helm chart prometheus-community/prometheus .

Chart - https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

Need to add basic authentication in prometheus to secure the UI and API endpoints . I am following the docs "https://prometheus.io/docs/guides/basic-auth/" and "https://prometheus.io/docs/prometheus/latest/configuration/https/" for the same .

Have created the web.yml file with basic auth settings .

I am not able to do helm upgrade as not sure how to set the flag "web.config.file" in prometheus . On local environment i can easily do this by restarting the prometheus and running the below command

Tried below upgrade commands but it doe not work as expected

helm upgrade --set-file web.config.file="C:\prometheus\prometheus\web.yaml" --reuse-values [release] prometheus-community/prometheus -n "namespace"
helm upgrade --set-file extraFlags.web.config.file="C:\prometheus\prometheus\web.yaml" --reuse-values [release]prometheus-community/prometheus -n "namespace"

.\prometheus. exe --web.config-file == "my web.yml file path"

Prometheus version - 2.26
valu.yml = https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml

Any help/suggestion would be great here.

prab2014 · 2021-08-24T08:58:31Z

Any updates here?

cykl · 2021-08-27T14:17:22Z

Basic auth is fairly easy to enable

server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'

The issue is that is also enable basic authentication on endpoints used for liveness and readiness probes, resulting in the pod being killed. Would be great to describe an idiomatic way to secure prometheus.

prab2014 · 2021-08-29T16:02:42Z

Thanks.
I have tried this earlier but as mentioned ,the pods were not working post helm upgrade with these changes.(401 error)
Also, added the basic auth settings at ingress level but got the ingress error( i am still working on this).
Do let me know the best way to secure prometheus api endpoints and web ui. Also, mechanism to integrate the promethues auth settings in Grafana too.

prab2014 · 2021-08-30T10:13:01Z

Basic auth is fairly easy to enable
server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'
The issue is that is also enable basic authentication on endpoints used for liveness and readiness probes, resulting in the pod being killed. Would be great to describe an idiomatic way to secure prometheus.

Thanks.

I have tried this earlier but as mentioned ,the pods were not working post helm upgrade with these changes.(401 error)
Also, added the basic auth settings at ingress level but got the ingress error( i am still working on this).
Do let me know the best way to secure prometheus api endpoints and web ui. Also, mechanism to integrate the promethues auth settings in Grafana too.

prab2014 · 2021-08-31T14:34:06Z

Basic auth is fairly easy to enable
server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'
The issue is that is also enable basic authentication on endpoints used for liveness and readiness probes, resulting in the pod being killed. Would be great to describe an idiomatic way to secure prometheus.
Thanks.

I have tried this earlier but as mentioned ,the pods were not working post helm upgrade with these changes.(401 error)
Also, added the basic auth settings at ingress level but got the ingress error( i am still working on this).
Do let me know the best way to secure prometheus api endpoints and web ui. Also, mechanism to integrate the promethues auth settings in Grafana too.

prab2014 · 2021-08-31T14:34:52Z

Basic auth is fairly easy to enable
server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'
The issue is that is also enable basic authentication on endpoints used for liveness and readiness probes, resulting in the pod being killed. Would be great to describe an idiomatic way to secure prometheus.

Hi,any updates here?

prab2014 · 2021-09-06T11:05:42Z

Basic auth is fairly easy to enable
server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'
The issue is that is also enable basic authentication on endpoints used for liveness and readiness probes, resulting in the pod being killed. Would be great to describe an idiomatic way to secure prometheus.

Hi, What would be the best way to add basic auths?

isac-seven · 2021-09-22T14:54:50Z

I have the same problem, while it doesn't get fixed I'm using basic auth at the ingress level.

prometheus:
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth
      nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'

As described here.

prab2014 · 2021-09-23T10:08:09Z

I have the same problem, while it doesn't get fixed I'm using basic auth at the ingress level.

prometheus:
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth
      nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'

As described here.

i have done the same stuff. Strange, why Prometheus inbuilt feature is not working here.

stale · 2021-10-23T11:17:16Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale · 2021-11-06T12:23:54Z

This issue is being automatically closed due to inactivity.

boris-yakimov · 2021-11-16T13:45:19Z

FYI - for whomever might still be looking for a solution for this, it seems a workaround is now supported in Prometheus Helm chart >= 14.10. Via support for custom headers (which can be used to autheticate the healthchecks), it was imeplemented here - 0d69e90 and merged here - #1422

probeHeaders:
  - name: Authorization
    value: Basic YWRtaW46c29tZXBhc3M=

Where the user and pass are base64 encoded in format user:pass and sent as an Authorization header.

prab2014 · 2021-11-16T13:55:36Z

FYI - for whomever might still be looking for a solution for this it seems a workaround is now supported in Prometheusm Helm chart >= 14.10. Via support for custom headers (which can be used to autheticate the healthchecks), it was imeplemented here - 0d69e90 and merged here - #1422
probeHeaders:
  - name: Authorization
    value: Basic YWRtaW46c29tZXBhc3M=
Where the user and pass are base64 encoded in format user:pass and sent with an Authorization header.

ok, this is great news.I will check the implementaions

looorent · 2021-12-01T13:21:24Z

Thanks @burizz , I confirm combining extraArgs with server.probeHeaders works fine.
Here is a template:

server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
  probeHeaders:
    - name: Authorization
      value: Basic <'username:password' in base64> 

serverFiles:
  web.config.yml:
    basic_auth_users:
      <username>: '<bcrypt password>'

AntonioVelichkov · 2021-12-01T15:11:54Z

probeHeaders works for Prometheus Server, but not for Alertmanger. So if I use basic-auth for Alertmanager the pod will still fail with Readiness probe failed: HTTP probe failed with statuscode: 401.

Has anyone found a workaround for this or we need to wait for probeHeaders for Alertmanger?

mindrunner · 2021-12-21T23:13:00Z

How does this work with kube-prometheus-stack?

brsolomon-deloitte · 2021-12-22T12:52:20Z

Seconding @mindrunner 's question since kube-prometheus-stack values.yaml seems to offer neither probeHeaders nor serverFiles.

Kushagratandon12 · 2021-12-28T14:41:09Z

Hey @cykl @prab2014 @burizz I was facing this same issue and found out one typo mistake that we are making

server:
  extraArgs:
    web.config.file: /etc/config/web_config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'

If you noticed the web.config.file we are creating or putting the data is web_config.yml + while in serverFiles we are specifying web.config.yml. This is what the small mistake we were making pasting correct code_snip

server:
  extraArgs:
    web.config.file: /etc/config/web.config.yml
serverFiles:
  web.config.yml:
    basic_auth_users:
      user: 'bcrypt password'

taskovskig · 2022-03-11T13:10:56Z

This is the full working solution (version 15.5.0):

server:
  tcpSocketProbeEnabled: true
  extraArgs:
    web.config.file: /etc/config/web.config.yml
  probeHeaders:
    - name: Authorization
      value: Basic 'base64 ${prometheus_admin_user_password}'
serverFiles:
  web.config.yml:
    basic_auth_users:
      admin: 'bcrypt ${prometheus_admin_password}'

If you deep dive into deployment template, you'll see that tcpSocketProbeEnabled: true disables the probes.

MDanialSaleem · 2022-06-06T03:14:02Z

It might be useful to get the base64 user:pass value from a secret instead.
Usecase:
-> store all secret values in external secret store (AWS secrets manager, Hashicorp vault etc)
-> use value from that secret, thereby removing the need to store a sensitive value in values file.

ravindraprasad85 · 2022-10-17T18:47:40Z

@taskovskig - Above does not worked for me, Can you please share your values.yaml files where you got this worked.
Do we need nginx as well to add password based authentication in kube-prometheus-stack?

Below is my content from values.yaml

  server:
    tcpSocketProbeEnabled: true
    extraArgs:
      web.config.file: /etc/config/web-config.yaml
    probeHeaders:
      - name: Authorization
        value: Basic V3IwbmdTY3IxcHRAMzIx

  serverFiles:
    web-config.yaml:
      basic_auth_users:
        admin: $2a$10$bAJQyQR0.Q1zohqXs7tZvuSncCr0p2Qrzsorn5d0ysoBVzT6BXum2``

rshiva777 · 2023-01-03T02:26:16Z

I am unable to find any parameters with serverFiles in the kube-prometheus-stack values.yaml

Where can we enable authentication for prometheus using the kube-prometheus-stack helm chart and what parameters need to be updated on values.yaml. Could anyone please help on this

yeatun · 2023-01-24T04:24:08Z

hey is it possible to implement this basic auth in prometheus.yaml that are provide istio,if it is please tell me the process

YuriiSmolii · 2023-02-14T13:40:43Z

Hi folks, recently I have tried to implement basic auth using configurations provided above:

Chart version: 19.3.3

values.yml

server:
  tcpSocketProbeEnabled: true
  extraArgs:
    web.config.file: /etc/config/web.config.yml
  probeHeaders:
    - name: Authorization
      value: Basic 'bW9udXNlcjpzdHJvbmdwYXNzMjAy'

serverFiles:
  web.config.yml:
    basic_auth_users:
      monuser: '$2a$12$A4DJ9RThnu3EZjMtyLbfReOeQW8uCmJ2RX5lXbDqSdDT8u4jIxM.K'

P.S I've tried with and without '' around credentials. Config don't work for me (
Does anyone have updates ? @taskovskig @looorent

swetharao12 · 2023-05-10T15:05:46Z

Hi All, Anyone has any update on above? I am trying to put the authentication on the prometheus server and would like to make it work for the kubernetes cluster.
works on the prometheus server using below document: https://prometheus.io/docs/guides/basic-auth/

rajibmitra · 2023-07-10T14:27:15Z

Hey everyone,
I came across an issue with the solution mentioned in this thread. It seems that the solution is not working as expected when using basic authentication in Helm charts.
If anyone has successfully enabled basic authentication and has a working values.yml file, could you please share.
Thank you!

huiweiguozi · 2024-04-05T00:03:55Z

It took me so much time to investigate this issue. I finally figured it out with the guidance above from taskovskig

A simple example could explain everything.

server:
  tcpSocketProbeEnabled: true
  extraArgs:
    web.config.file: /etc/config/web.config.yml
  probeHeaders:
    - name: Authorization
      value: Basic 'base64 ${prometheus_admin_user_password}'
serverFiles:
  web.config.yml:
    basic_auth_users:
      admin: 'bcrypt ${prometheus_admin_password}'

If the username=test, and password=test,
then use the python code to generate the prometheus_admin_password, result is $2b$12$/u2Vwu7QRcycE9JwJ6SzWOkdZMxIGiajWL3rakyGaXRMUX9hPncJS

import bcrypt

password = 'test'
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
print(hashed_password.decode())

and 'base64 ${prometheus_admin_user_password}' should be generated like this:

echo -n 'test:test' | base64

result: dGVzdDp0ZXN0

Thus, the configuration block that fully works for me is:

server:
  extraArgs:
    web.config.file: /etc/config/web.config.yml
  probeHeaders:
    - name: Authorization
      value: Basic 'dGVzdDp0ZXN0'
serverFiles:
  web.config.yml:
    basic_auth_users:
      test: '$2b$12$/u2Vwu7QRcycE9JwJ6SzWOkdZMxIGiajWL3rakyGaXRMUX9hPncJS'

prab2014 changed the title ~~Securing Prometheus API and UI endpoints using basic auth in Container~~ [prometheus-community/prometheus] Securing Prometheus API and UI endpoints using basic auth in Container Aug 17, 2021

prab2014 mentioned this issue Aug 27, 2021

Configuration for Basic Authentication ( prometheus/alertmanager) #1186

Closed

zanhsieh changed the title ~~[prometheus-community/prometheus] Securing Prometheus API and UI endpoints using basic auth in Container~~ [prometheus] Securing Prometheus API and UI endpoints using basic auth in Container Aug 30, 2021

prab2014 closed this as completed Aug 31, 2021

prab2014 reopened this Aug 31, 2021

morsik mentioned this issue Oct 12, 2021

[prometheus] Allow to specify custom headers for probes, closes #1255 #1422

Merged

3 tasks

stale bot added the lifecycle/stale label Oct 23, 2021

stale bot closed this as completed Nov 6, 2021

brsolomon-deloitte mentioned this issue Dec 21, 2021

[prometheus-kube-stack] Lack of Alertmanger.probeHeaders makes checks fails 4xx #1621

Closed

heojay mentioned this issue Feb 11, 2022

[prometheus] Allow to specify custom probeHeaders for Alertmanager #1798

Merged

3 tasks

nikolic-milan mentioned this issue Jan 31, 2023

[kube-prometheus-stack] Securing exposed endpoint with basic auth #2972

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[prometheus] Securing Prometheus API and UI endpoints using basic auth in Container #1255

[prometheus] Securing Prometheus API and UI endpoints using basic auth in Container #1255

prab2014 commented Aug 17, 2021 •

edited

Loading

prab2014 commented Aug 24, 2021

cykl commented Aug 27, 2021

prab2014 commented Aug 29, 2021

prab2014 commented Aug 30, 2021

prab2014 commented Aug 31, 2021

prab2014 commented Aug 31, 2021

prab2014 commented Sep 6, 2021

isac-seven commented Sep 22, 2021

prab2014 commented Sep 23, 2021

stale bot commented Oct 23, 2021

stale bot commented Nov 6, 2021

boris-yakimov commented Nov 16, 2021 •

edited

Loading

prab2014 commented Nov 16, 2021

looorent commented Dec 1, 2021

AntonioVelichkov commented Dec 1, 2021

mindrunner commented Dec 21, 2021

brsolomon-deloitte commented Dec 22, 2021

Kushagratandon12 commented Dec 28, 2021

taskovskig commented Mar 11, 2022 •

edited

Loading

MDanialSaleem commented Jun 6, 2022

ravindraprasad85 commented Oct 17, 2022 •

edited

Loading

rshiva777 commented Jan 3, 2023

yeatun commented Jan 24, 2023

YuriiSmolii commented Feb 14, 2023

swetharao12 commented May 10, 2023 •

edited

Loading

rajibmitra commented Jul 10, 2023 •

edited

Loading

huiweiguozi commented Apr 5, 2024 •

edited

Loading

[prometheus] Securing Prometheus API and UI endpoints using basic auth in Container #1255

[prometheus] Securing Prometheus API and UI endpoints using basic auth in Container #1255

Comments

prab2014 commented Aug 17, 2021 • edited Loading

prab2014 commented Aug 24, 2021

cykl commented Aug 27, 2021

prab2014 commented Aug 29, 2021

prab2014 commented Aug 30, 2021

prab2014 commented Aug 31, 2021

prab2014 commented Aug 31, 2021

prab2014 commented Sep 6, 2021

isac-seven commented Sep 22, 2021

prab2014 commented Sep 23, 2021

stale bot commented Oct 23, 2021

stale bot commented Nov 6, 2021

boris-yakimov commented Nov 16, 2021 • edited Loading

prab2014 commented Nov 16, 2021

looorent commented Dec 1, 2021

AntonioVelichkov commented Dec 1, 2021

mindrunner commented Dec 21, 2021

brsolomon-deloitte commented Dec 22, 2021

Kushagratandon12 commented Dec 28, 2021

taskovskig commented Mar 11, 2022 • edited Loading

MDanialSaleem commented Jun 6, 2022

ravindraprasad85 commented Oct 17, 2022 • edited Loading

Below is my content from values.yaml

rshiva777 commented Jan 3, 2023

yeatun commented Jan 24, 2023

YuriiSmolii commented Feb 14, 2023

swetharao12 commented May 10, 2023 • edited Loading

rajibmitra commented Jul 10, 2023 • edited Loading

huiweiguozi commented Apr 5, 2024 • edited Loading

prab2014 commented Aug 17, 2021 •

edited

Loading

boris-yakimov commented Nov 16, 2021 •

edited

Loading

taskovskig commented Mar 11, 2022 •

edited

Loading

ravindraprasad85 commented Oct 17, 2022 •

edited

Loading

swetharao12 commented May 10, 2023 •

edited

Loading

rajibmitra commented Jul 10, 2023 •

edited

Loading

huiweiguozi commented Apr 5, 2024 •

edited

Loading