Get: auto redirect is disabled - azure-devops-exporter k8s

[Crulex]

Hello,

could you please help, I am trying to deploy this exporter in k8s cluster and got CrashLoopBackOff on the pod.
Seems like exporter is not able connect to Azure REST API, I generated PAT with full access permissions.
From the kctl logs I got:

_

RESTY 2022/07/12 12:05:26 ERROR Get "https://spsprodweu1.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2F(Organization_Name)%2F_apis%2Fprojects%3F%2524top%3D100%26api-version%3D5.1&redirect=1&hid=cc01a906-d406-467d-bb70-56ce6c4df2a5&context=eyJodCI6MiwiaGlkIjoiZjYyMjRhNzAtMzFhYi00NDZlLTgxZWQtYzcwZDVlNDY1MmYwIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2": auto redirect is disabled, Attempt 1

_

Regards,
M

PS. This link is working correctly in web browser - it is redirecting to Azure REST API (https://dev.azure.com/(Organization_name)/_apis/projects?%24top=100&api-version=5.1

[mblaschke]

can you post your settings (dont post PAT token and org)?
it seems that the api calls are redirected to oauth login for whatever reason

[Crulex]

Sure.

apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-devops-exporter
labels:
app: azure-devops-exporter
namespace: xxxx
spec:
replicas: 1
selector:
matchLabels:
app: azure-devops-exporter
template:
metadata:
labels:
app: azure-devops-exporter
spec:
containers:
- name: azure-devops-exporter
image: webdevops/azure-devops-exporter:21.8.1
resources:
limits:
cpu: "2000m"
memory: "2048Mi"
requests:
cpu: "500m"
memory: "512Mi"
env:
- name: VERBOSE
value: 'true'
- name: ALLOW_REDIRECTS
value: 'true'
# - name: AZURE_DEVOPS_URL
# value: https://dev.azure.com
- name: AZURE_DEVOPS_ORGANISATION
value: xxxx
- name: AZURE_DEVOPS_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: xxxx
key: xxx

[mblaschke]

if you use dev.azure.com please don't set AZURE_DEVOPS_URL

[Crulex]

Even without AZURE_DEVOPS_URL i've got same issue.
May it be related to AKS cluster settings, network policy?

apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-devops-exporter
labels:
app: azure-devops-exporter
namespace: xxxxx
spec:
replicas: 1
selector:
matchLabels:
app: azure-devops-exporter
template:
metadata:
labels:
app: azure-devops-exporter
spec:
containers:
- name: azure-devops-exporter
image: webdevops/azure-devops-exporter:21.8.1
resources:
limits:
cpu: "2000m"
memory: "2048Mi"
requests:
cpu: "500m"
memory: "512Mi"
env:
- name: AZURE_DEVOPS_ORGANISATION
value: xxxxx
- name: AZURE_DEVOPS_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: xxxxx
key: xxx

[mblaschke]

Can you access the Azure DevOps API with the accesstoken using Postman? There should be not a redirect (which means you're redirected to the login page).

BTW: you should not overcommit memory, the Azure DevOps exporter should not use more then 512 Mi. Overcommiting memory could trigger a Kernel OOMKiller.

[Crulex]

I tried with PAT and Azure AD access token and in Postman I get Azure Devops Sing In page.
In our organization we have two-factor authentication it may matters?

[mblaschke]

The exporter only works with PAT (because it doesn't offer AzureAD auth). The 2fa should not be active when using PAT token because it's a technical authentication.

Please double check that your PAT token is correct configured (needs to be encoded base64 inside a Kubernetes secret).

[Crulex]

Thank you, I generated new PAT and everything is working... :)

[Bootcody]

Hey guys, I've got exactly the same issue today.
I configured ADOE to get PAT from Azure KV and now having exactly the same issue.
I confirm I can do API calls using this PAT with no problem. Also adding this PAT as container arguments with plain text PAT works absolutely fine:

env:

• name: MY_PAT
  value: "MY_PAT_HERE"
  ... and then...
  --azuredevops.access-token=$(MY_PAT)

I tried to re-generate my PAT (what helped @Crulex above), also to generate a completely new one, I still get this:

{"file":"main.go:41","func":"main","level":"info","msg":"starting azure-devops-exporter v21.8.1 (f84fc36; go1.17; by webdevops.io)"}

...
3 attempts to login and finally:

"panic: Get "https://spsprodweu4.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2FORG%2F_apis%2Fprojects%3F%2524top%3D300%26api-version%3D5.1&redirect=1&hid=ID&context=bla-bla": auto redirect is disabled"

...
goroutine 1 [running]:
main.getAzureDevOpsProjects()
/go/src/github.com/webdevops/azure-devops-exporter/main.go:208 +0x425
main.initMetricCollector()
/go/src/github.com/webdevops/azure-devops-exporter/main.go:246 +0xd2
main.main()
/go/src/github.com/webdevops/azure-devops-exporter/main.go:48 +0x1f8

BTW, adding

• name: VERBOSE
  value: 'true'
  - name: DEBUG
  value: 'true'

doesn't help to see any more details.

Any help would be much appreciated.
Thx

[mblaschke]

Have you created a PAT for the correct organisation? Please dont use a "all accessible org" PAT.

[Crulex]

@Bootcody + You need to encode it in base64 before you paste it into secret.

[Bootcody]

Thanks for your prompt response @mblaschke!
Yes, the PAT is all fine.

When I feed this PAT in the deployment file as plain text, it all works fine.
Old and new PATs, both no problem.

I was testing this whole morning and see that my PAT is not being picked up properly.
First, I found that I put it in the KV not BASE64 encoded (thanks for also pointing this out @Crulex).
Then I encoded it and confirmed AccessTokenFile picks it up from the proper location.
I also confirmed the file contains the proper string BASE64 encoded. But same issue still.

This type of error basically means ADOE doesn't have proper PAT passed over.
As I tested it with a wrong PAT (intentionally) as well having the same exact result.

Having no sh/ash/bash makes troubleshooting a bit slower as well...

But any advice from this point? What exactly does ADOE expect from the token file?

UPD: You know how it works, right? You do a lot of testing. Write huge text with details and then find out that the issue is actually you... Apparently, using a PS script to encode the string was not the best idea. Right after publishing this response I tried with another encoding option and suddenly all started working -.-

[mblaschke]

have you used this?

$patb64 = [Convert]::ToBase64String([System.Text.Encoding]::Unicode.GetBytes($Pat))

then you get a utf16 encoded pat token so you should use:

$patb64 = [Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($Pat))

[mblaschke]

use helm to deploy your apps to AKS/kubernetes. there you can easily encode values using base64:
https://helm.sh/docs/chart_template_guide/function_list/

[Bootcody]

Indeed, Unicode was used at first. With ASCII it works fine.

Thanks for the advice, I will take a look. Still fairly new to the subject.

[mblaschke]

solution: please make sure your PAT is up2date and encoded correctly if you use them in a Kubernetes secret: as ASCII base64 (use helm b64enc)