Add support for IAM Auth for Google CloudSQL DBs #22445

kpcraig · 2023-08-18T19:39:53Z

This pull requests adds Google Cloud IAM-based auto-authorization capability to the MySQL and PostgreSQL database types.

Specifically, it adds an auth_type (with supported value gcp_iam) and credentials field to the database/config request. When set, the database backend will use GCP IAM authorization (supplied either via the environment or via the credentials field), and can understand the GCP location triple in the connection_url field:

Example:

  bin/vault write database/config/mysqldb \
    plugin_name="mysql-database-plugin" \
    allowed_roles="my-role" \
    auth_type="gcp_iam" \
    connection_url='db-service-acct@cloudsql-mysql(hc-0ef364feb79745e0941e057c8c4:us-central1:mysql-instance)/'

Note that MSSQL on GCP doesn't support auto IAM connections, and so isn't supported here.

In technical implementation, the new auth_type field signals the switch into the IAM auth path, which in GCPs case involves using their custom dialer/driver implementation.

…-sql-poc

sdk/database/helper/connutil/cloudsql.go

github-actions · 2023-08-22T16:28:21Z

Build Results:
All builds succeeded! ✅

sdk/database/dbplugin/databasemiddleware.go

sdk/database/helper/connutil/cloudsql.go

sdk/database/helper/connutil/sql.go

physical/mssql/mssql.go

plugins/database/mysql/connection_producer.go

sdk/database/helper/connutil/cloudsql.go

plugins/database/mysql/connection_producer.go

fairclothjm

LGTM! Great work ya'll!

sdk/database/helper/connutil/sql.go

sdk/database/dbplugin/v5/testing/test_helpers.go

calvn

Left a couple of small comments around formatting

fairclothjm · 2023-09-05T20:50:23Z

plugins/database/mysql/connection_producer.go

@@ -29,6 +35,8 @@ type mySQLConnectionProducer struct {
 	MaxConnectionLifetimeRaw interface{} `json:"max_connection_lifetime" mapstructure:"max_connection_lifetime" structs:"max_connection_lifetime"`
 	Username                 string      `json:"username" mapstructure:"username" structs:"username"`
 	Password                 string      `json:"password" mapstructure:"password" structs:"password"`
+	AuthType                 string      `json:"auth_type" mapstructure:"auth_type" structs:"auth_type"`
+	ServiceAccountJSON       string      `json:"service_account_json" mapstructure:"service_account_json" structs:"service_account_json"`


nit: I was going to suggest adding a comment to this field to document that it is only for GCP. But maybe we should consider renaming it instead? I think in Vault-land we usually use GOOGLE_APPLICATION_CREDENTIALS for service account JSON.

Interesting point! This field was originally named credentials, and was renamed to service_account_json with the same intent of making it GCP specific. I would say service_account_json suffices enough to make it explicit that this is a GCP related field because in the IAM context, I think only GCP uses the term 'service accounts' as far as I know 🤔 (I believe AWS just calls them IAM users). If it is not clear enough I can add in a comment like you suggested!

Here's the original renaming discussion for context: #22445 (comment)

We're still making use of the GOOGLE_APPLICATION_CREDENTIALS under the hood as the environment variable though, since that is what the gcloud sdk also expects (similar to Vault)

austingebauer

LGTM! Great job on this 🎉

austingebauer · 2023-09-06T02:04:26Z

sdk/database/helper/connutil/sql.go

@@ -29,8 +37,15 @@ type SQLConnectionProducer struct {
 	MaxConnectionLifetimeRaw interface{} `json:"max_connection_lifetime" mapstructure:"max_connection_lifetime" structs:"max_connection_lifetime"`
 	Username                 string      `json:"username" mapstructure:"username" structs:"username"`
 	Password                 string      `json:"password" mapstructure:"password" structs:"password"`
+	AuthType                 string      `json:"auth_type" mapstructure:"auth_type" structs:"auth_type"`


Not blocking but we could validate that the given auth_type is one we support and provide an error message indicating misconfiguration. Right now I think it would accept anything as input.

Added auth type validation with some basic tests in f9c5f7f

sdk/database/helper/connutil/cloudsql.go

vinay-gopalan and others added 15 commits July 24, 2023 13:20

add initial poc

725f146

format code

ca75032

add options for credential authentication

daaa0a2

add thread safe cache and mysql implementation

e473221

fix mssql package conflicts

90e314d

individual dialers per conn

9642f2f

move some stuff around

8051c2b

update parameters to be credentials and credentials_json

3439cf6

Merge branch 'VAULT-17992/kay-gcloud-sql-poc' into VAULT-17992/gcloud…

0d75032

…-sql-poc

finish merge

8884d61

add unique driver login flow for gcp

c3439ee

resolve conflicts and update credential options

4e95fa1

test outline

bb523bb

Merge branch 'main' into VAULT-17992/gcloud-sql-poc

f19d08b

fix gomod after merge

2d18dc7

github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Aug 18, 2023

kpcraig added this to the 1.15 milestone Aug 18, 2023

kpcraig added 2 commits August 18, 2023 15:59

fix go sum

ab33183

fix go.sum?

407fb94

github-advanced-security bot found potential problems Aug 18, 2023

View reviewed changes

sdk/database/helper/connutil/cloudsql.go Fixed Show fixed Hide fixed

vinay-gopalan changed the title ~~Draft: A PR~~ Add support for IAM Auth for Google CloudSQL DBs Aug 18, 2023

vinay-gopalan added 2 commits August 22, 2023 09:22

update tests with credentials and add a new user test for postgres

8465080

remove unnecessary log

4065b4f

vinay-gopalan marked this pull request as ready for review August 22, 2023 16:25

vinay-gopalan requested a review from a team as a code owner August 22, 2023 16:25

vinay-gopalan requested review from a team and removed request for a team August 22, 2023 16:27

vinay-gopalan reviewed Aug 22, 2023

View reviewed changes

sdk/database/dbplugin/databasemiddleware.go Outdated Show resolved Hide resolved

vinay-gopalan reviewed Aug 22, 2023

View reviewed changes

sdk/database/helper/connutil/cloudsql.go Outdated Show resolved Hide resolved

fixing in dialer closing logic

1ee56f4

austingebauer reviewed Aug 31, 2023

View reviewed changes

sdk/database/helper/connutil/sql.go Outdated Show resolved Hide resolved

vinay-gopalan reviewed Aug 31, 2023

View reviewed changes

physical/mssql/mssql.go Outdated Show resolved Hide resolved

fairclothjm reviewed Aug 31, 2023

View reviewed changes

plugins/database/mysql/connection_producer.go Outdated Show resolved Hide resolved

fairclothjm reviewed Aug 31, 2023

View reviewed changes

plugins/database/mysql/connection_producer.go Show resolved Hide resolved

fairclothjm reviewed Aug 31, 2023

View reviewed changes

sdk/database/helper/connutil/cloudsql.go Show resolved Hide resolved

fairclothjm reviewed Aug 31, 2023

View reviewed changes

plugins/database/mysql/connection_producer.go Outdated Show resolved Hide resolved

vinay-gopalan added 2 commits September 1, 2023 13:59

update API field to be service_account_json

e1499f6

fix unnecessary package formatting changes

7974576

fairclothjm approved these changes Sep 1, 2023

View reviewed changes

calvn reviewed Sep 1, 2023

View reviewed changes

sdk/database/helper/connutil/sql.go Outdated Show resolved Hide resolved

calvn reviewed Sep 1, 2023

View reviewed changes

sdk/database/helper/connutil/sql.go Outdated Show resolved Hide resolved

calvn reviewed Sep 1, 2023

View reviewed changes

sdk/database/dbplugin/v5/testing/test_helpers.go Outdated Show resolved Hide resolved

calvn reviewed Sep 1, 2023

View reviewed changes

vinay-gopalan added 7 commits September 5, 2023 12:23

formatting feedback

b0a9ad0

resolve go.mod conflicts with main

5114daa

remove extra newline

49c2486

fix sdk go sum entries

1b0fa91

revert back to github.com/denisenkom/go-mssqldb package for msssql

bda4fb2

more formatting improvements

7910b20

more import formatting

6d4727e

fairclothjm reviewed Sep 5, 2023

View reviewed changes

austingebauer approved these changes Sep 6, 2023

View reviewed changes

vinay-gopalan added 2 commits September 6, 2023 14:10

add auth type validation with tests

f9c5f7f

resolve go.mod conflicts with main

16eeea8

vercel bot deployed to Preview September 6, 2023 21:19 View deployment

vinay-gopalan added the backport/1.15.x label Sep 6, 2023

vinay-gopalan merged commit 2172786 into main Sep 6, 2023
103 checks passed

vinay-gopalan deleted the VAULT-17992/gcloud-sql-poc branch September 6, 2023 21:40

hc-github-team-secure-vault-core mentioned this pull request Sep 6, 2023

Backport of Add support for IAM Auth for Google CloudSQL DBs into release/1.15.x #22834

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for IAM Auth for Google CloudSQL DBs #22445

Add support for IAM Auth for Google CloudSQL DBs #22445

kpcraig commented Aug 18, 2023 •

edited

Loading

github-actions bot commented Aug 22, 2023

fairclothjm left a comment

calvn left a comment

fairclothjm Sep 5, 2023

vinay-gopalan Sep 5, 2023

vinay-gopalan Sep 5, 2023

austingebauer left a comment

austingebauer Sep 6, 2023

vinay-gopalan Sep 6, 2023

Add support for IAM Auth for Google CloudSQL DBs #22445

Add support for IAM Auth for Google CloudSQL DBs #22445

Conversation

kpcraig commented Aug 18, 2023 • edited Loading

github-actions bot commented Aug 22, 2023

fairclothjm left a comment

Choose a reason for hiding this comment

calvn left a comment

Choose a reason for hiding this comment

fairclothjm Sep 5, 2023

Choose a reason for hiding this comment

vinay-gopalan Sep 5, 2023

Choose a reason for hiding this comment

vinay-gopalan Sep 5, 2023

Choose a reason for hiding this comment

austingebauer left a comment

Choose a reason for hiding this comment

austingebauer Sep 6, 2023

Choose a reason for hiding this comment

vinay-gopalan Sep 6, 2023

Choose a reason for hiding this comment

kpcraig commented Aug 18, 2023 •

edited

Loading