Using Google Secrets Manager

[nikaro]

Hello,

As privy, the recommended solution, is likely unmaintained i am trying to use google-cloud-secret-manager:

from google.cloud import secretmanager_v1


def _get_secret(name: str) -> str:
    secret_manager = secretmanager_v1.SecretManagerServiceClient()
    secret = secret_manager.access_secret_version(name=name).payload.data.decode("utf-8").strip()
    return secret


ts_auth_key = _get_secret("projects/mygcpproject/secrets/MY_SECRET/versions/latest")

server.shell(
    name="Start Tailscale service",
    commands=[f"echo {ts_auth_key}"],
    _sudo=True,
)

This is failing with this error:

➤ pyinfra @local test.py

--> Loading config...

--> Loading inventory...

--> Connecting to hosts...
    [@local] Connected

--> Preparing Operations...
    Loading: test.py
--> An exception occurred in: test.py:

  File "project/.direnv/python-3.11/lib/python3.11/site-packages/pyinfra_cli/util.py", line 60, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "test.py", line 11, in <module>
    ts_auth_key = _get_secret("projects/mygcpproject/secrets/MY_SECRET/versions/latest")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test.py", line 7, in _get_secret
    secret = secret_manager.access_secret_version(name=name).payload.data.decode("utf-8").strip()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1518, in access_secret_version
    response = rpc(
               ^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 372, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 207, in retry_target
    result = target()
             ^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
Traceback (most recent call last):
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "project/.direnv/python-3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-11-21T11:19:07.887731+01:00", grpc_status:4}"
>

The above exception was the direct cause of the following exception:

google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

But if i remove the pyinfra parts and just run my _get_secret() function and print the result, it works fine.

What could it be?

[nikaro]

I've tried using python.call() calling _get_secret() which was modified to set a global variable. Same error.

[Fizzadar]

I think I finally found out the underlying issue: https://stackoverflow.com/questions/60957202/using-grpc-with-gevent

[nikaro]

I ended up writing my own wrapper around SOPS to work around this is the meantime: https://github.com/nikaro/sopsy

[maisim]

Great ! Thank you for sharing !

[romain-dartigues]

Dropping in the conversation ✈️.

Has the GNOME libsecret Python connector been considered (implementing the org.freedesktop.Secret.Service specification)?
Some people already use it to unlock their SSH keys, with git, KeePassXC already implement it, etc. It could be a quick-win for Linux users.

[xvello]

Heya @romain-dartigues, a PR to add a libsecret example to https://github.com/pyinfra-dev/pyinfra/blob/3.x/docs/examples/secret_storage.md would be great! There's no need to add the dependency on pyinfra itself, as the inventory data is generated from custom python code.

keyring or SecretStorage could also be used to access the same dbus API.

[romain-dartigues]

Hi @xvello,

Thank you for your feedback, I'll work on it.