Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PULUMI_ACCESS_TOKEN must be set error when using azure blob as backend #1010

Closed
dkezri opened this issue Aug 9, 2023 · 17 comments
Closed

PULUMI_ACCESS_TOKEN must be set error when using azure blob as backend #1010

dkezri opened this issue Aug 9, 2023 · 17 comments
Labels
kind/bug Some behavior is incorrect or out of spec

Comments

@dkezri
Copy link

dkezri commented Aug 9, 2023

What happened?

Hi,

I m trying to use azure blob as backend that is required by my organization, but i get error:

PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

i have this environment variables

env:
AZCLI_VERSION: latest
AZURE_WRITE: false
DOTNET_VERSION: 7.0.x
AZURE_REGION: norwayeast
PULUMI_WORK_DIR: PulumiAzure
PULUMI_STACK_NAME: dev
DOTNET_ROOT: /usr/share/dotnet
AZURE_HTTP_USER_AGENT:
AZUREPS_HOST_ENVIRONMENT:
AZURE_STORAGE_ACCOUNT: pulumistateaccountdev
AZURE_CONTAINER_NAME: pulumistatecontainerdev
AZURE_STORAGE_KEY: rI/otrVwLuo0WrV+GUC7V5azr23RVc56AewKt5wv10SLDnXqpH4I**********
AZURE_STORAGE_SAS_TOKEN: se=.******************
ARM_USE_OIDC: true
ARM_CLIENT_ID: ***
ARM_TENANT_ID: ***
ARM_SUBSCRIPTION_ID: ***

Configured range: ^3
/opt/hostedtoolcache/pulumi/3.77.1/x64/pulumi version
v3.77.1
Pulumi version 3.77.1 is already installed on this machine. Skipping download
Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions
err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

at Object.createCommandError (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
at Object.<anonymous> (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
at Generator.throw (<anonymous>)
at rejected (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

Expected Behavior

the pulumi state is not created in azure blob

Steps to reproduce

i use github action with azure cli

Output of pulumi about

Configured range: ^3
/opt/hostedtoolcache/pulumi/3.77.1/x64/pulumi version
v3.77.1
Pulumi version 3.77.1 is already installed on this machine. Skipping download
Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions
err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

at Object.createCommandError (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
at Object.<anonymous> (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
at Generator.throw (<anonymous>)
at rejected (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@dkezri dkezri added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Aug 9, 2023
@pierskarsenbarg
Copy link
Member

pierskarsenbarg commented Aug 9, 2023

@dkezri I was talking to someone else about this yesterday. If you use PULUMI_BACKEND_URL and set that to be whatever value you're setting cloud-url to be, this should work.

I think this is still an issue so it should be left open for now, but that's at least a workaround for now

@Frassle Frassle removed the needs-triage Needs attention from the triage team label Aug 9, 2023
@Frassle
Copy link
Member

Frassle commented Aug 9, 2023

I wonder if this is just an actions issue? It seems to work fine locally on the command line.
Will have a check of the actions code.

@pierskarsenbarg pierskarsenbarg transferred this issue from pulumi/pulumi Aug 9, 2023
@pierskarsenbarg
Copy link
Member

Moved to the GH action repo

@dkezri
Copy link
Author

dkezri commented Aug 9, 2023

@dkezri I was talking to someone else about this yesterday. If you use PULUMI_BACKEND_URL and set that to be whatever value you're setting cloud-url to be, this should work.

I think this is still an issue so it should be left open for now, but that's at least a workaround for now

I am using this configuration in my git hub action:

  - name: setup azure pulumi state resources
    uses: azure/powershell@v1
    with:
      inlineScript: |
       ./pulumi_azure_blob_setup.ps1
      azPSVersion: "latest"  
    env:
      Environment: dev
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
     
  - uses: pulumi/actions@v4
    with:
      command: up
      github-token: ${{ secrets.GIT_ACCESS_TOKEN }}
      stack-name: ${{ env.PULUMI_STACK_NAME }}
      work-dir: ${{ env.PULUMI_WORK_DIR }}
      cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
      upsert: true
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}

@pierskarsenbarg
Copy link
Member

If you updated that and added PULUMI_BACKEND_URL to the env section in the Pulumi action:

- uses: pulumi/actions@v4
    with:
      command: up
      github-token: ${{ secrets.GIT_ACCESS_TOKEN }}
      stack-name: ${{ env.PULUMI_STACK_NAME }}
      work-dir: ${{ env.PULUMI_WORK_DIR }}
      cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
      upsert: true
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
      PULUMI_BACKEND_URL: azblob://${{ env.AZURE_CONTAINER_NAME }}

That should work

@dkezri
Copy link
Author

dkezri commented Aug 9, 2023

@pierskarsenbarg
i did exactly what you suggested above, but now it gives another error:

Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=InvalidAuthenticationInfo) =====
Description=Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:08d50ab4-601e-0047-04ce-ca147f000000
Time:2023-08-09T14:36:57.0675933Z, Details: (none)

err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=InvalidAuthenticationInfo) =====
Description=Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:08d50ab4-601e-0047-04ce-ca147f000000

@Frassle
Copy link
Member

Frassle commented Aug 9, 2023

Have you managed to get this working locally? This looks like incorrect auth tokens, but that should be simpler to verify locally than running through GHA.

@dkezri
Copy link
Author

dkezri commented Aug 9, 2023

@Frassle this is only an github action issue, it works when i am using #PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} then it create state in pulumi backend and also creates new resources in Azure, my point is to used Azure blob as backend

@Frassle
Copy link
Member

Frassle commented Aug 9, 2023

Yes, but what I'm asking is if you've managed to get it creating stacks in Azure on your local machine?

That is if you try and pulumi login azblob://pulumistatecontainerdev with the full envvars above:

AZURE_STORAGE_ACCOUNT: pulumistateaccountdev
AZURE_CONTAINER_NAME: pulumistatecontainerdev
AZURE_STORAGE_KEY: rI/otrVwLuo0WrV+GUC7V5azr23RVc56AewKt5wv10SLDnXqpH4I**********
AZURE_STORAGE_SAS_TOKEN: se=.******************

Does that login successfully?

@Frassle
Copy link
Member

Frassle commented Aug 9, 2023

I think your access keys are invalid. Double check with the azure CLI as well, as something like az storage blob list --container-name pulumistatecontainerdev might give better errors than the raw http of the gocloud library.

@dkezri
Copy link
Author

dkezri commented Aug 10, 2023

@Frassle i get information about container:
PS C:\Users\dkezri> az storage blob list --container-name pulumistatecontainerdev --account-name pulumistateaccountdev

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add --auth-mode login in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
[
{
"container": "pulumistatecontainerdev",
"content": "",
"deleted": null,
"encryptedMetadata": null,
"encryptionKeySha256": null,
"encryptionScope": null,
"hasLegalHold": null,
"hasVersionsOnly": null,
"immutabilityPolicy": {
"expiryTime": null,
"policyMode": null
},
"isAppendBlobSealed": null,
"isCurrentVersion": null,
"lastAccessedOn": null,
"metadata": {},

@Frassle
Copy link
Member

Frassle commented Aug 10, 2023

Hitting the limits of what I can help with here, azure auth isn't something I've worked with much.
I'll see if I can get someone with more Azure knowledge to comment here, but overall I don't believe this to be a bug in pulumi, just a problem setting up the correct environment.

@MatteoCalabro-TomTom
Copy link

Joining the discussion because I am hitting the same problem. I also suspect this is GA issue, because I have the same setup working locally with Pulumi.

Locally, I do not use storage key or SAS token. I only use CLI login with RBAC and it works flawlessly.

This is my workflow:

jobs:
  deploy:
    name: Deploy Infrastructure
    environment: dev
    steps:
      - uses: azure/login@v1
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Test Federated Credentials Login
        shell: bash
        run: |
          az account show -o table
          az storage account list -o table
          az keyvault list -o table
      - uses: actions/checkout@v3
      - uses: pulumi/actions@v4
        with:
          command: preview
          stack-name: dev
          work-dir: deployments/platform
          cloud-url: ${{ env.PULUMI_BACKEND_URL }}
          secrets-provider: ${{ env.PULUMI_SECRETS_PROVIDER }}

OIDC login with Azure CLI gives the correct permissions, identity is able to read all the necessary resources, but pulumi actions fails with error:

/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/.pulumi -f /runner/_work/_temp/cd9f01fb-0f4c-41d7-9edf-ad5dff61c39e
Logging into azblob://stacks

/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
                    : new CommandError(result);
^
CommandError: code: -2
 stdout: 
 stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:c[36](***)ba214-a01e-0064-01d5-dcd9d1000000
Time:2023-09-01T13:08:22.22[38](***)768Z, Details: (none)

 err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:c36ba214-a01e-0064-01d5-dcd9d1000000
Time:2023-09-01T13:08:22.2238768Z, Details: (none)


    at Object.createCommandError (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
    at Object.<anonymous> (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
    at Generator.throw (<anonymous>)
    at rejected (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Identity has the same roles as my account

@MatteoCalabro-TomTom
Copy link

I tried using the pulumi CLI in run steps and it seems it has the same authorization error. It must be related to OIDC authentication (federated credentials), because pulumi won't be able to read the blob storage, regardless of the assigned roles.

For reference, the roles assigned to the identity are:

  • Contributor (subscription)
  • Storage Blob Data Contributor (storage account)
  • Storage Queue Data Contributor (storage account)
  • Storage File Data Privileged Contributor (storage account)

@Frassle
Copy link
Member

Frassle commented Sep 1, 2023

Looking into this a bit today, the error message "PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions" can only be hit if the CLI isn't in "filestate" mode.
It determines that by reading the current backend url from "~/.pulumi/credentials.json", which the action should be setting by calling pulumi login.

So somehow the backend information is getting lost.

@mshakhmaykin
Copy link

I had the same issue when used AWS S3 bucket as a backend, and I can confirm that setting $PULUMI_BACKEND_URL env variable resolved the problem for me.

peterussell added a commit to peterussell/checklisttrainer.com that referenced this issue Nov 8, 2023
@dkezri
Copy link
Author

dkezri commented Nov 16, 2023

I fixed it by removing the step of automation using "./pulumi_azure_blob_setup.ps1" script from the github action.
So i ended up to create the azure storage account and container manually using Azure portal, also added my oidc application to the keyvault policy in order to read and decrypt the key. now everything is working fine. here is my github action for deploying to Azure dev environment:

name: Deploy to dev environment on Azure
on:
  workflow_dispatch:
  push:
    branches: [ "development" ]

permissions:
  id-token: write
  contents: read

env:
  AZCLI_VERSION: "latest"
  DOTNET_VERSION: 7.0.x
  PULUMI_WORK_DIR: "./PulumiAzure"
  PULUMI_STACK_NAME: dev
  AZURE_RESOURCE_GROUP: pulumi-state-dev-rg
  AZURE_STORAGE_ACCOUNT: pulumidevst
  AZURE_CONTAINER_NAME: dashboard-state-dev-blobc

jobs:
  CI:
     name: CI
     uses: ./.github/workflows/ci-base.yaml
     with:
       runs-on: '["ubuntu-latest"]'
       dotnet-version: 7.0.x

  Dev:
    name: Deploy to dev
    needs: CI
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GIT_ACCESS_TOKEN }}
      
      - name: Setup DotNet
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: ${{ env.DOTNET_VERSION }}

      - name: Azure CLI Login
        uses: azure/login@v1
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID_DEV }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID_DEV }} 

      - name: Azure CLI script
        uses: azure/CLI@v1
        with:
          azcliversion: ${{ env.AZCLI_VERSION }}
          inlineScript: |
            AZURE_STORAGE_KEY=$(az storage account keys list -n ${{ env.AZURE_STORAGE_ACCOUNT }} -g ${{ env.AZURE_RESOURCE_GROUP }} --query '[0].value' -o tsv)
            echo "AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY" >> $GITHUB_ENV

      - name: pulumi up step
        uses: pulumi/actions@v4
        with:
          command: up
          stack-name: ${{ env.PULUMI_STACK_NAME }}
          cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
          work-dir: ${{ env.PULUMI_WORK_DIR }}
        env:
          ARM_USE_OIDC: true
          ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID_DEV }}
          ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
          ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID_DEV }}
          AZURE_STORAGE_ACCOUNT: ${{ env.AZURE_STORAGE_ACCOUNT }}
          AZURE_STORAGE_KEY: ${{ env.AZURE_STORAGE_KEY }}
          AZURE_KEYVAULT_AUTH_VIA_CLI: true
      - run: echo "Azure instructure is now deployed to dev environment"
      - run: az logout

@dkezri dkezri closed this as completed Nov 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Development

No branches or pull requests

5 participants