Wazuh `agentless_cluster` `system tests` are failing due to changes in `token_response['json']['data']['token']` in API `/security/user/authenticate` endpoint

[pro-akim]

Running system test in v4.8.0-alpha1
In agentless_cluster environment was possible to detect some issues:

In some of the following tests:

test_jwt_invalidation/test_revoke_endpoint.py
test_jwt_invalidation/test_change_security_resources.py
test_jwt_invalidation/test_change_rbac_mode.py
test_jwt_invalidation/test_update_password.py

The following error message was detected:

        Returns:
            API token (str): Usable API token.
        """
        login_endpoint = '/security/user/authenticate'
        login_method = 'POST'
        login_body = ''
        if auth_context is not None:
            login_endpoint = '/security/user/authenticate/run_as'
            login_body = 'body="{}"'.format(json.dumps(auth_context).replace('"', '\\"').replace(' ', ''))
    
        try:
            token_response = self.get_host(host).ansible('uri', f"url=https://localhost:{port}{login_endpoint} "
                                                                f"user={user} password={password} "
                                                                f"method={login_method} {login_body} validate_certs=no "
                                                                f"force_basic_auth=yes",
                                                         check=check)
            return token_response['json']['data']['token']
        except KeyError:
>           raise KeyError(f'Failed to get token: {token_response}')
E           KeyError: "Failed to get token: {'changed': False, 'connection': 'close', 'content_length': '65', 'content_type': 'application/problem+json; charset=utf-8', 'date': 'Mon, 11 Dec 2023 10:43:15 GMT', 'elapsed': 0, 'msg': 'Status code was 400 and not [200]: HTTP Error 400: Bad Request', 'redirected': False, 'status': 400, 'url': 'https://localhost:55000/security/user/authenticate/run_as'}"

On the other hand, in test_jwt_invalidation/test_update_password.py, the following variation of the message:

    def get_api_token(self, host, user='wazuh', password='wazuh', auth_context=None, port=55000, check=False):
        """Return an API token for the specified user.
    
        Args:
            host (str): Hostname.
            user (str, optional): API username. Default `wazuh`
            password (str, optional): API password. Default `wazuh`
            auth_context (dict, optional): Authorization context body. Default `None`
            port (int, optional): API port. Default `55000`
            check (bool, optional): Ansible check mode("Dry Run"),
                by default it is enabled so no changes will be applied. Default `False`
    
        Returns:
            API token (str): Usable API token.
        """
        login_endpoint = '/security/user/authenticate'
        login_method = 'POST'
        login_body = ''
        if auth_context is not None:
            login_endpoint = '/security/user/authenticate/run_as'
            login_body = 'body="{}"'.format(json.dumps(auth_context).replace('"', '\\"').replace(' ', ''))
    
        try:
            token_response = self.get_host(host).ansible('uri', f"url=https://localhost:{port}{login_endpoint} "
                                                                f"user={user} password={password} "
                                                                f"method={login_method} {login_body} validate_certs=no "
                                                                f"force_basic_auth=yes",
                                                         check=check)
            return token_response['json']['data']['token']
        except KeyError:
>           raise KeyError(f'Failed to get token: {token_response}')
E           KeyError: "Failed to get token: {'changed': False, 'connection': 'close', 'content_length': '59', 'content_type': 'application/problem+json; charset=utf-8', 'date': 'Mon, 11 Dec 2023 10:44:58 GMT', 'elapsed': 0, 'msg': 'Status code was 401 and not [200]: HTTP Error 401: Unauthorized', 'redirected': False, 'status': 401, 'url': 'https://localhost:55000/security/user/authenticate/run_as'}"

/usr/local/lib/python3.10/dist-packages/wazuh_testing/tools/system.py:229: KeyError

Reports are here:
report_agentless_cluster.zip

Research should be done to see if the API is failing or there were some changes in the endpoint that should be changed in the test

[pro-akim]

Update

/security/user/authenticate/run_as seems to be the issue.
Those tests that does not include the mentioned method are working ok.

When the mentioned method is being used with the following input:

user=wazuh-wui password=wazuh-wui method=POST body="{\"username\":\"elastic\"}" validate_certs=no force_basic_auth=yes check=False

The test is failing showing the following message:

{'changed': False, 'connection': 'close', 'content_length': '65', 'content_type': 'application/problem+json; charset=utf-8', 'date': 'Mon, 18 Dec 2023 16:11:47 GMT', 'elapsed': 0, 'msg': 'Status code was 400 and not [200]: HTTP Error 400: Bad Request', 'redirected': False, 'status': 400, 'url': 'https://localhost:55000/security/user/authenticate/run_as'}

[pro-akim]

Update

In wazuh v4.7.0

Request parameters:

url=https://localhost:55000/security/user/authenticate/run_as user=testing_user password=Testing1* method=POST body="{\"username\":\"testing\"}" validate_certs=no force_basic_auth=yes check=False

Response

{'cache_control': 'no-cache, no-store, must-revalidate, max-age=0', 'changed': False, 'connection': 'close', 'content_length': '515', 'content_security_policy': 'none', 'content_type': 'application/json; charset=utf-8', 'cookies': {}, 'cookies_string': '', 'date': 'Mon, 18 Dec 2023 16:44:26 GMT', 'elapsed': 0, 'expires': '0', 'json': {'data': {'token': 'eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzAyOTE3ODY2LCJleHAiOjE3MDI5MTg3NjYsInN1YiI6InRlc3RpbmdfdXNlciIsInJ1bl9hcyI6dHJ1ZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSIsImhhc2hfYXV0aF9jb250ZXh0IjoiZjQ0ODVkMGYzM2Y4ZjFlMmZiNGIxYjA4NTI2OTFiYTMifQ.AFRTyfMUI643QLpyeWjvEe7lbdMBRHbyZvvadJv0iBGx3v6h-YWjeV9ceGI9IYL-JO0N9tcsrYRh06sIRlOiYacwAEzSUlQq_Qbh8Szj-rbae3779eSvQSZMrHrCGwrQAVfO-RfzQgxFJmTdRfd9-lss1VRdHgX94RUH7ojFuifCndT6'}, 'error': 0}, 'msg': 'OK (515 bytes)', 'pragma': 'no-cache', 'redirected': False, 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin', 'status': 200, 'strict_transport_security': 'max-age=63072000; includeSubdomains', 'url': 'https://localhost:55000/security/user/authenticate/run_as', 'x_content_type_options': 'nosniff', 'x_frame_options': 'DENY', 'x_xss_protection': '1; mode=block'} 

[pro-akim]

Update

Manually testing, the endpoint /security/user/authenticate/run_as worked in v4.8.0-alpha1

curl -k -u sales-member-1:Sales-Member-1 -X POST https://localhost:55000/security/user/authenticate/run_as -H 'content-type: application/json' -d '{
        "name": "Initial_auth",
        "auth": {
                "name": "Bill",
                "office": ["20", "21", "30"]
        }
}'

root@wazuh-master: ~# curl -k -X POST "https://localhost:55000/security/users?pretty=true" -H  "accept: application/json" -H  "Authorization: Bearer $TOKEN" -H  "Content-Type: application/json" -d "{\"username\":\"sales-member-1\",\"password\":\"Sales-Member-1\"}"
{
   "data": {
      "affected_items": [
         {
            "id": 100,
            "username": "sales-member-1",
            "allow_run_as": false,
            "roles": []
         }
      ],
      "total_affected_items": 1,
      "total_failed_items": 0,
      "failed_items": []
   },
   "message": "User was successfully created",
   "error": 0
}

root@wazuh-master: ~# curl -k -X PUT "https://localhost:55000/security/users/100/run_as?allow_run_as=true" -H  "Authorization: Bearer $TOKEN"

{"data": {"affected_items": [{"id": 100, "username": "sales-member-1", "allow_run_as": true, "roles": []}], "total_affected_items": 1, "total_failed_items": 0, "failed_items": []}, "message": "Parameter allow_run_as has been enabled for the user", "error": 0}root@wazuh-master: ~# curl -k -X PUT "https://localhost:55000/security/users/100/run_as?allow_run_as=root@wazuh-master: ~# curl -k -X GET "https://localhost:55000/security/users?pretty=true" -H  "accept: application/json" -H  "Authorization: Bearer $TOKEN" -H  "Content-Type: application/json"    
{
   "data": {
      "affected_items": [
         {
            "id": 1,
            "username": "wazuh",
            "allow_run_as": true,
            "roles": [
               1
            ]
         },
         {
            "id": 2,
            "username": "wazuh-wui",
            "allow_run_as": true,
            "roles": [
               1
            ]
         },
         {
            "id": 100,
            "username": "sales-member-1",
            "allow_run_as": true,
            "roles": []
         }
      ],
      "total_affected_items": 3,
      "total_failed_items": 0,
      "failed_items": []
   },
   "message": "All specified users were returned",
   "error": 0
}

root@wazuh-master: ~# curl -k -u sales-member-1:Sales-Member-1 -X POST https://localhost:55000/security/user/authenticate/run_as -H 'content-type: application/json' -d '{
>         "name": "Initial_auth",
>         "auth": {
>                 "name": "Bill",
>                 "office": ["20", "21", "30"]
>         }
> }'
{"data": {"token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzAyOTc3MTUxLCJleHAiOjE3MDI5NzgwNTEsInN1YiI6InNhbGVzLW1lbWJlci0xIiwicnVuX2FzIjp0cnVlLCJyYmFjX3JvbGVzIjpbXSwicmJhY19tb2RlIjoid2hpdGUiLCJoYXNoX2F1dGhfY29udGV4dCI6IjY2YTI4ZWZiNWY4ZGIyZTkzZGEyYjVmYjEwN2JlYzM1In0.AZSV_ExIt6o9kwNqfvShEDLzPBufjAlZGhnUq1iNUTqBa8ld3MGeNMOwVVSrppf0ShluB5Tl0sMphkl16LYb6MpqAUrPL3Xlo0PA5RI_8UrczOymtBhZGKq3VOluLcrBVm0jCuNLFZwVskoZQTuMroFPor_GYiZT92IoDjLIyGKo3gze"}, "error": 0}

However, some changes in the function of the endpoint could happen.

[pro-akim]

Update

When the infrastructure is freshly deployed (Wazuh and Wazuh-qa v4.8.0-alpha1):

root@wazuh-master: ~# curl -k -u wazuh-wui:wazuh-wui -X POST https://localhost:55000/security/user/authenticate/run_as -H 'content-type: application/json' -d '{"username":"elastic"}'
{"data": {"token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzAyOTg2NzY4LCJleHAiOjE3MDI5ODc2NjgsInN1YiI6IndhenVoLXd1aSIsInJ1bl9hcyI6dHJ1ZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSIsImhhc2hfYXV0aF9jb250ZXh0IjoiN2UzYmQ3YzVlYTBlMDdiZWIzNGU5ZGJlZjcwZWFlNmQifQ.AHpyT6yy-a30VcUT7OOeWdXUIhB46L76R07oWbHVQHHglQm8O4sLf0ls1GjmtKxl8dbUAs-kr1pqjX8KeERM0ejeAP_c0wqT_B6imSY-fS_PKRccrSj-rCvfG7cCIeyq-Ql5oxkmb88-r-HMnTsis9VDzigPOjFY1zA2otqQ_YZyqLlb"}, "error": 0}

After running tests (all related to run_as method have failed)

root@wazuh-master: ~# curl -k -u wazuh-wui:wazuh-wui -X POST https://localhost:55000/security/user/authenticate/run_as -H 'content-type: application/json' -d '{"username":"elastic"}'
curl: (7) Failed to connect to localhost port 55000: Connection refused

API logs did not show any relevant log:

2023/12/19 14:07:33 INFO: Checking RBAC database integrity...
2023/12/19 14:07:33 INFO: /var/ossec/api/configuration/security/rbac.db file was detected
2023/12/19 14:07:33 INFO: RBAC database integrity check finished successfully
2023/12/19 14:07:36 INFO: Listening on 0.0.0.0:55000..

---

After several repetitions, the following observations were made:

At times:

cat /var/ossec/api/configuration/api.yaml
''

Other times:

root@wazuh-master: ~# cat /var/ossec/api/configuration/api.yaml
access:
  max_login_attempts: 9998

When api.yaml is empty (''), the API responds as described before. However, when it contains content, the API still responds to manual API calls:

root@wazuh-master: ~# curl -k -u wazuh-wui:wazuh-wui -X POST https://localhost:55000/security/user/authenticate/run_as -H 'content-type: application/json' -d '{"username":"elastic"}'
{"data": {"token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzAyOTk1NTE3LCJleHAiOjE3MDI5OTY0MTcsInN1YiI6IndhenVoLXd1aSIsInJ1bl9hcyI6dHJ1ZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSIsImhhc2hfYXV0aF9jb250ZXh0IjoiN2UzYmQ3YzVlYTBlMDdiZWIzNGU5ZGJlZjcwZWFlNmQifQ.ASAIiUAjnqkMnTRBKOFn8cEmKBRAJ_xELUa3KhR32o1-mZI6gCSwM2-2oTkuRXUJIpZDu_Z_dhL4vCEDfCRAJn0vAbX_T-sIsSymgOwOf20Bawd6tfD2LJQB0QEd5MMLHhhEXmPjfvKwoUGt9z1nMZt-mEgbtK34jtdYILvXW22XFGIb"}, "error": 0}

If I replace '' with content, there's a delay in response.

Despite these changes, running the tests leads to API failures (which don't occur in version v4.7.0). Testing Wazuh v4.7.0 with Wazuh-QA v4.8.0-alpha1 doesn't show failures, indicating that the testing framework is still functioning correctly. Report is attached here:
report_4.7.0.zip

[pro-akim]

Summary

1. Wazuh 4.7.0 is functioning properly, and tests with both Wazuh-qa v4.7.0 and v4.8.0-alpha1 have passed successfully.
2. Wazuh v4.8.0-alpha1 exhibits certain erratic behaviors in the API at the endpoint /security/user/authenticate/run_as. All agentless_cluster tests using that method have failed because attempting to obtain the token resulted in a status_code=400 and 401.
3. During manual tests, the method has been observed to function correctly. However, after running the tests (where only the token retrieval method with the conflicting endpoint is executed), it starts failing, showing connection issues.
4. No relevant records have been observed in the api.log logs.
5. Checking cat /var/ossec/api/configuration/api.yaml, it has been noticed that at times, it's empty, and at other times, it contains content. When the correct content is present, the token can be obtained manually. However, during test execution, the tests fail to obtain the token.
6. When replacing the empty content of api.yaml with YAML-like content, token retrieval is restored after a waiting period. However, the automated tests continue to fail consistently.
7. To ensure that the tests are functioning properly, Wazuh v4.7.0 is re-run with Wazuh-qa v4.8.0-alpha1, and all tests pass satisfactorily.

Some research over Wazuh v4.8.0 security/user/authenticate/run_as should be done

• A new issue is opened API /security/user/authenticate/run_as endpoint is not always working as expected wazuh#20947