VBS Copays endpoint has been erroring since 11PM 12/17

[digitaldrk]

Issue Description

The VBS 'GetStatementsByEDIPIAndVistaAccountNumber' has been erroring.

I have sent a VA email at around 9:15AM 12/19 to the VBS team about this issue and messaged Denise in OCTO Slack.

Russel from VBS responded. He says he sees 401 (unauthorized). He provided a swagger link. It seems our api key is still valid.

I do not see any changes to code on our side. He says they have not changed their code.

I am going to try and fire a curl or Faraday request from a prod server. This will maybe give us an idea of what the error is exactly. We use a proxy, I wonder if that's our problem.

Link to possible clue. FWD proxy was updated at the time we began seeing errors

Tasks

• Not sure what task is required at the moment

Design and Front End sync on implementation of design

• Required
• Complete
• Not Required

Acceptance Criteria

• VBS endpoint is responding properly again

Testing

• N/A if non development work
• Testing passed and documented in this ticket based off the "Then" statement in the description

Test procedures:
If additional testing steps or credentials to perform testing are needed list them here

Ticket Creation - Definition of Ready

Ticket requirements

• Description written
• Tasks defined
• Acceptance criteria written
• Testing defined
• Add Labels
• Assign Type
• Assign Priority (Normal = Medium)
• Assign Group(s)
• Assign to a Project
• Attach to an Parent Epic
• Estimate Points

[digitaldrk]

I pinged support. I think this has to do with a change in fwd proxy:

https://dsva.slack.com/archives/CBU0KDSB1/p1734641816686429

[digitaldrk]

Current status:

We have among others, the following values we use to authenticate with VBS:

• host
• url
• apiKey

Devops made a change to load balancing and forward proxy in AWS, this broke our functionality with VBS.

After deep troubleshooting, I noticed that staging still works. I found that the host values we store in AWS "parameter store" are fundamentally different between staging and prod. In staging, it is just a domain like this: something.va.gov. In prod, it has https like this: https.something.va.gov.

I updated the value in parameter store, platform happened to be doing a deployment anyways, and we should have been good to go but we're not.

Evidently, when we set the host value in staging, we are using the AWS parameter store host value. In Prod, we are using the AWS parameter store url value... SUPER frustrating.

So now, I have to make sure all code using these credentials are expecting the right value. Hope to have this done today. This is/should be a very straight forward change but I don't want to rev up the platform team on christmas eve without making SURE it's going to work this time.

Not sure when another deployment is leaving the train station during the holidays.

[digitaldrk]

We are seeing successful requests rolling in.

This necessitated the following:

• Debugging between staging and prod env's where I wrote a 'raw' Faraday request to locate the problem. I ran these requests in Argo and found the issue was the host value set, we pass this into the header.
• Update to AWS parameter store for the hostvalue.
• PR to vsp infra manifest
• OOB deployment to pick up new values

[digitaldrk]

Post Mortem for incident