[Bug] unable to unmarshal conftest output or Anyone able to approve Atlantis policy failures

[kumaresh0]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

1. Anyone ( non policy owners ) able to approve Atlantis policy failures ( seems major bug )
   I am using below server config with custom_policy_check: true & policy_check: true

2. if I use custom_policy_check: flase & policy_check: true i get the unable to unmarshal conftest output error

But based on this documentation https://www.runatlantis.io/docs/policy-checking.html#step-2-define-the-policy-configuration we tried the config as i mentioned below

Reproduction Steps

Nothing special just used Alpine image and added the below server-side workflow and repo side workflow then triggered policy failure, if anyone comments the atlantis approve_policies policy failures are fixed with approval

Logs

Issue screen shot

Environment details

If not already included, please provide the following:

• Atlantis version: v0.27.1
• Deployment method: ecs
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: Yes ( it' already latest release )
• Atlantis flags: No custom flags ( using default flags with repo details )
  ECS environment variables

  container_definition_environment = [
    {
      name  = "ATLANTIS_CHECKOUT_STRATEGY"
      value = var.checkout_strategy
    },
    {
      name  = "ATLANTIS_ALLOW_COMMANDS"
      value = var.atlantis_allow_commands
    },
    {
      name  = "ATLANTIS_ALLOW_REPO_CONFIG"
      value = var.allow_repo_config
    },
    {
      name  = "ATLANTIS_RESOURCE_BUCKET_NAME"
      value = data.aws_s3_bucket.atlantis_bucket.id

    },
    {
      name  = "ATLANTIS_LOG_LEVEL"
      value = var.atlantis_log_level
    },
    {
      name  = "ATLANTIS_ASSUME_ROLE_NAME"
      value = var.atlantis_assume_role_name
    },
    {
      name  = "ATLANTIS_PORT"
      value = var.atlantis_port
    },
    {
      name  = "ATLANTIS_CONFIG"
      value = var.atlantis_config
    },
    {
      name  = "ATLANTIS_ATLANTIS_URL"
      value = local.atlantis_url
    },
    {
      name  = "ATLANTIS_WRITE_GIT_CREDS"
      value = true
    },
    {
      name  = "ATLANTIS_GH_USER"
      value = var.atlantis_github_user
    },
    {
      name  = "ATLANTIS_REPO_ALLOWLIST"
      value = join(",", var.atlantis_repo_allowlist)
    },
    {
      name  = "ATLANTIS_REPO_CONFIG"
      value = var.atlantis_repo_config
    },
    {
      name  = "ATLANTIS_ENABLE_POLICY_CHECKS"
      value = var.atlantis_enable_policy_checks
    },
    {
      name  = "ATLANTIS_HIDE_PREV_PLAN_COMMENTS"
      value = var.atlantis_hide_prev_plan_comments
    },
    {
      name  = "ATLANTIS_DISABLE_APPLY"
      value = var.atlantis_disable_apply
    },
    {
      name  = "ATLANTIS_DEFAULT_TF_VERSION"
      value = var.atlantis_default_terraform_version
    },
    {
      name  = "ATLANTIS_POLICY_DIRECTORY"
      value = var.atlantis_policy_directory
    },
    {
      name  = "ATLANTIS_POLICY_REPO_NAME"
      value = var.atlantis_policy_repo_name
    },
    {
      name  = "ATLANTIS_POLICY_REPO_REF"
      value = var.atlantis_policy_repo_ref
    }
  ]

Atlantis server-side config file:

repos:
- id: "/.*/"
  branch: "/master/"
  pre_workflow_hooks:
    - description: Pull the Atlantis policies.
  post_workflow_hooks:
    - run: python3 /home/atlantis/review.py 
  allow_custom_workflows: false
  custom_policy_check: true
  policy_check: true
  allowed_overrides: [workflow, apply_requirements, delete_source_branch_on_merge]
  apply_requirements: [approved, mergeable]
  import_requirements: [approved, mergeable]

metrics:
  prometheus:
    endpoint: /metrics

policies:
    conftest_version: 0.45.0
    owners:
      users:
        - user1
        - user2
        - user3
    policy_sets:
        - name: audit_policy
          path: /home/atlantis/policy/security
          source: local
          approve_count: 2
          owners:
            users:
              - securityuser1
        - name: comman_policy
          path: /home/atlantis/policy/regula
          source: local

workflows:
  default:
    plan:
      steps:
      - init:
          extra_args: [ "-reconfigure"]
      - run: TF_WORKSPACE=$WORKSPACE tflint --config /home/atlantis/.tflint.hcl
      - plan
      - run: terraform$ATLANTIS_TERRAFORM_VERSION show -no-color -json $PLANFILE > tfplan.json
    apply:
      steps:
      - apply
    policy_check: &policy_check
      steps:
      - run: |
          aws sts get-caller-identity --output json | jq '{"aws": .}' | jq '{"external": .}' > external-data.json
      - run: conftest test tfplan.json --namespace main --namespace security -o table --policy /home/atlantis/policy -d external-data.json
    import:
      steps:
      - import

Repo atlantis.yaml file:

version: 3
automerge: true
delete_source_branch_on_merge: true

projects:
- name: test-case/default-workflow
  dir: test-case/default-workflow
  workspace: default
  autoplan:
    when_modified: ["*.tf", "../modules/**.tf", "Terrafile"]
    enabled: true
  workflow: default

Our Atlantis is deployed in ECS fargate with ghcr.io/runatlantis/atlantis:v0.27-alpine

Additional Context

[kumaresh0]

similar issuer reported here : #4308

[kumaresh0]

Adding some update on this:

From console, i can see policy test results

example

Resources: {"aws_security_group.inline_invalid_security_group"}

4 tests, 2 passed, 0 warnings, 2 failures, 0 exceptions

But in PR it show unable to unmarshal conftest output and anyone able to approve the policy failures

[Mezage]

I hope this issue is getting traction, our team really needs to upgrade Atlantis but we cant since custom policies were introduced, this policy approval bug has been an issue. Anyone can approve policies, so policy set approvers doesnt even matter. :/

[nitrocode]

We're running into this issue too. We might need a reproducible example here.

{"level":"error","ts":"2024-10-30T18:36:33.314Z","caller":"events/instrumented_project_command_runner.go:78","msg":"Error running policy_check operation: unable to unmarshal conftest output","json":{"repo":"snip/snip","pull":"2985"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:78\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}