[Core] Token Auth minor UX Improvements

[sampan-s-nayak]

Description

• convert debug logs in authentication_token_loader to info so that users are aware of where the token being used is being loaded from
• When we raise the AuthenticationError, if RAY_AUTH_MODE is not set to token we should explicitly print that in the error message
• in error messages suggest storing tokens in filesystem instead of env
• add state api tests in test_token_auth_integration.py

[sampan-s-nayak]

@edoakes I removed this as this was throwing a nested stack trace:

ray start --head
Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.

Local node IP: 172.31.5.49
Traceback (most recent call last):
  File "/home/ubuntu/clone/ray/python/ray/scripts/scripts.py", line 946, in start
    ensure_token_if_auth_enabled(system_config, create_token_if_missing=False)
  File "/home/ubuntu/clone/ray/python/ray/_private/authentication/authentication_token_setup.py", line 101, in ensure_token_if_auth_enabled
    raise AuthenticationError(
ray.exceptions.AuthenticationError: Token authentication is enabled but no authentication token was found.. Ensure that you have `RAY_AUTH_MODE=token` set and the token for the cluster is available as the `RAY_AUTH_TOKEN` environment variable or a local file. For more information, see: https://docs.ray.io/en/latest/ray-security/auth.html

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.conda/envs/ray-dev/bin/ray", line 7, in <module>
    sys.exit(main())
  File "/home/ubuntu/clone/ray/python/ray/scripts/scripts.py", line 2822, in main
    return cli()
  File "/home/ubuntu/.conda/envs/ray-dev/lib/python3.10/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.conda/envs/ray-dev/lib/python3.10/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.conda/envs/ray-dev/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.conda/envs/ray-dev/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.conda/envs/ray-dev/lib/python3.10/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/clone/ray/python/ray/autoscaler/_private/cli_logger.py", line 823, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/clone/ray/python/ray/scripts/scripts.py", line 948, in start
    raise RuntimeError(
RuntimeError: Failed to load authentication token. To generate a token for local development, use `ray get-auth-token --generate`. For remote clusters, ensure that the token is propagated to all nodes of the cluster when token authentication is enabled.

I feel exposing the AuthenticationError to users may be better than catching and throwing a new runtime exception. please let me know your thoughts

[sampan-s-nayak]

I have included the message from the runtime error in the AuthenticationError so the message will still be quite helpful

[gemini-code-assist]

Code Review

This pull request introduces several valuable UX improvements for token authentication. The changes enhance error messages to be more informative, improve logging for token loading, and add crucial integration tests for the state API. The implementation is solid, and the new tests provide good coverage. I have a couple of minor suggestions to improve code style and robustness.