Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow SOPS to use custom AWS KMS and STS Endpoint #1679

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
7f908e6
Allow SOPS to use custom AWS KMS and STS Endpoint
anandavj Nov 19, 2024
725cb39
build(deps): Bump github/codeql-action in the ci group
dependabot[bot] Nov 18, 2024
00f1ee7
build(deps): Bump the rust group in /functional-tests with 3 updates
dependabot[bot] Nov 18, 2024
5ee78da
build(deps): Bump the ci group with 2 updates
dependabot[bot] Nov 25, 2024
5ec3720
build(deps): Bump the go group across 1 directory with 18 updates
dependabot[bot] Nov 25, 2024
214a053
Patch harmless parts of go.mod before looking for changes in CI.
felixfontein Nov 30, 2024
29cffad
Run 'make vendor' in CodeQL build step.
felixfontein Nov 30, 2024
2e12a90
Move Python patch tool to .github/utils/.
felixfontein Nov 30, 2024
d755eae
Correctly handle comments during JSON serialization.
felixfontein Oct 9, 2024
be3b48a
gofmt
anandavj Dec 1, 2024
59d274f
remove manual update on proto
anandavj Dec 1, 2024
afe398e
define protobuf definition for aws endpoint and re-generate
anandavj Dec 1, 2024
f895c62
Patch harmless parts of go.mod before looking for changes in CI.
felixfontein Nov 30, 2024
944ad9f
Move Python patch tool to .github/utils/.
felixfontein Nov 30, 2024
f6a6a10
chore(deps): Update Rust dependencies
reneleonhardt Jun 11, 2024
a79294e
fix(config): remove unused variables
nicklasfrahm Nov 25, 2024
71a1b3c
keyservice: update protobuf to an recent release
Foxboron Aug 5, 2024
3f9c620
Adjust protoc command line for latest versions of plugins; fix protoc…
felixfontein Dec 1, 2024
35338f7
Update go.mod.
felixfontein Dec 1, 2024
d927a74
build(deps): Bump the go group with 8 updates
dependabot[bot] Dec 2, 2024
43f17a8
Bump version to 3.9.2.
felixfontein Dec 1, 2024
02cb2c0
Add changelog for 3.9.2.
felixfontein Dec 1, 2024
58607af
build(deps): Bump the go group with 10 updates
dependabot[bot] Dec 9, 2024
0295b78
build(deps): Bump the ci group with 2 updates
dependabot[bot] Dec 9, 2024
2af48ab
build(deps): Bump golang.org/x/crypto from 0.30.0 to 0.31.0
dependabot[bot] Dec 13, 2024
3326e78
Sign Commit
anandavj Dec 14, 2024
501eb42
Fix Protobuf error on generated
anandavj Dec 14, 2024
c57a49d
Patch harmless parts of go.mod before looking for changes in CI.
felixfontein Nov 30, 2024
70eba1e
Move Python patch tool to .github/utils/.
felixfontein Nov 30, 2024
6ff4856
Merge branch 'main' into main
anandavj Dec 14, 2024
9a82a2b
Merge branch 'main' into main
anandavj Dec 22, 2024
f79b2b7
Merge branch 'main' into main
anandavj Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 35 additions & 3 deletions cmd/sops/main.go
Original file line number Diff line number Diff line change
@@ -501,6 +501,14 @@ func main() {
Name: "aws-profile",
Usage: "The AWS profile to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-kms-endpoint",
Usage: "The AWS KMS Endpoint to use for requests to AWS. Ex: https://kms.ap-southeast-2.amazonaws.com",
},
cli.StringFlag{
Name: "aws-sts-endpoint",
Usage: "The AWS STS Endpoint to use for requests to AWS. Ex: https://sts.ap-southeast-2.amazonaws.com",
},
cli.StringSliceFlag{
Name: "gcp-kms",
Usage: "the GCP KMS Resource ID the new group should contain. Can be specified more than once",
@@ -545,7 +553,7 @@ func main() {
group = append(group, pgp.NewMasterKeyFromFingerprint(fp))
}
for _, arn := range kmsArns {
group = append(group, kms.NewMasterKeyFromArn(arn, kms.ParseKMSContext(c.String("encryption-context")), c.String("aws-profile")))
group = append(group, kms.NewMasterKeyFromArn(arn, kms.ParseKMSContext(c.String("encryption-context")), c.String("aws-profile"), c.String("aws-kms-endpoint"), c.String("aws-sts-endpoint")))
}
for _, kms := range gcpKmses {
group = append(group, gcpkms.NewMasterKeyFromResourceID(kms))
@@ -852,6 +860,14 @@ func main() {
Name: "aws-profile",
Usage: "The AWS profile to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-kms-endpoint",
Usage: "The AWS KMS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-sts-endpoint",
Usage: "The AWS STS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "gcp-kms",
Usage: "comma separated list of GCP KMS resource IDs",
@@ -1169,6 +1185,14 @@ func main() {
Name: "aws-profile",
Usage: "The AWS profile to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-kms-endpoint",
Usage: "The AWS KMS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-sts-endpoint",
Usage: "The AWS STS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "gcp-kms",
Usage: "comma separated list of GCP KMS resource IDs",
@@ -1529,6 +1553,14 @@ func main() {
Name: "aws-profile",
Usage: "The AWS profile to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-kms-endpoint",
Usage: "The AWS KMS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "aws-sts-endpoint",
Usage: "The AWS STS Endpoint to use for requests to AWS",
},
cli.StringFlag{
Name: "gcp-kms",
Usage: "comma separated list of GCP KMS resource IDs",
@@ -2006,7 +2038,7 @@ func getEncryptConfig(c *cli.Context, fileName string) (encryptConfig, error) {

func getMasterKeys(c *cli.Context, kmsEncryptionContext map[string]*string, kmsOptionName string, pgpOptionName string, gcpKmsOptionName string, azureKvOptionName string, hcVaultTransitOptionName string, ageOptionName string) ([]keys.MasterKey, error) {
var masterKeys []keys.MasterKey
for _, k := range kms.MasterKeysFromArnString(c.String(kmsOptionName), kmsEncryptionContext, c.String("aws-profile")) {
for _, k := range kms.MasterKeysFromArnString(c.String(kmsOptionName), kmsEncryptionContext, c.String("aws-profile"), c.String("aws-kms-endpoint"), c.String("aws-sts-endpoint")) {
masterKeys = append(masterKeys, k)
}
for _, k := range pgp.MasterKeysFromFingerprintString(c.String(pgpOptionName)) {
@@ -2185,7 +2217,7 @@ func keyGroups(c *cli.Context, file string) ([]sops.KeyGroup, error) {
return nil, common.NewExitError("Invalid KMS encryption context format", codes.ErrorInvalidKMSEncryptionContextFormat)
}
if c.String("kms") != "" {
for _, k := range kms.MasterKeysFromArnString(c.String("kms"), kmsEncryptionContext, c.String("aws-profile")) {
for _, k := range kms.MasterKeysFromArnString(c.String("kms"), kmsEncryptionContext, c.String("aws-profile"), c.String("aws-kms-endpoint"), c.String("aws-sts-endpoint")) {
kmsKeys = append(kmsKeys, k)
}
}
16 changes: 10 additions & 6 deletions config/config.go
Original file line number Diff line number Diff line change
@@ -100,10 +100,12 @@ type gcpKmsKey struct {
}

type kmsKey struct {
Arn string `yaml:"arn"`
Role string `yaml:"role,omitempty"`
Context map[string]*string `yaml:"context"`
AwsProfile string `yaml:"aws_profile"`
Arn string `yaml:"arn"`
Role string `yaml:"role,omitempty"`
Context map[string]*string `yaml:"context"`
AwsProfile string `yaml:"aws_profile"`
AwsKmsEndpoint string `yaml:"aws_kms_endpoint"`
AwsStsEndpoint string `yaml:"aws_sts_endpoint"`
}

type azureKVKey struct {
@@ -130,6 +132,8 @@ type creationRule struct {
PathRegex string `yaml:"path_regex"`
KMS string
AwsProfile string `yaml:"aws_profile"`
AwsKmsEndpoint string `yaml:"aws_kms_endpoint"`
AwsStsEndpoint string `yaml:"aws_sts_endpoint"`
Age string `yaml:"age"`
PGP string
GCPKMS string `yaml:"gcp_kms"`
@@ -218,7 +222,7 @@ func extractMasterKeys(group keyGroup) (sops.KeyGroup, error) {
keyGroup = append(keyGroup, pgp.NewMasterKeyFromFingerprint(k))
}
for _, k := range group.KMS {
keyGroup = append(keyGroup, kms.NewMasterKeyWithProfile(k.Arn, k.Role, k.Context, k.AwsProfile))
keyGroup = append(keyGroup, kms.NewMasterKeyWithProfile(k.Arn, k.Role, k.Context, k.AwsProfile, k.AwsKmsEndpoint, k.AwsStsEndpoint))
}
for _, k := range group.GCPKMS {
keyGroup = append(keyGroup, gcpkms.NewMasterKeyFromResourceID(k.ResourceID))
@@ -261,7 +265,7 @@ func getKeyGroupsFromCreationRule(cRule *creationRule, kmsEncryptionContext map[
for _, k := range pgp.MasterKeysFromFingerprintString(cRule.PGP) {
keyGroup = append(keyGroup, k)
}
for _, k := range kms.MasterKeysFromArnString(cRule.KMS, kmsEncryptionContext, cRule.AwsProfile) {
for _, k := range kms.MasterKeysFromArnString(cRule.KMS, kmsEncryptionContext, cRule.AwsProfile, cRule.AwsKmsEndpoint, cRule.AwsStsEndpoint) {
keyGroup = append(keyGroup, k)
}
for _, k := range gcpkms.MasterKeysFromResourceIDString(cRule.GCPKMS) {
10 changes: 6 additions & 4 deletions keyservice/keyservice.go
Original file line number Diff line number Diff line change
@@ -53,10 +53,12 @@ func KeyFromMasterKey(mk keys.MasterKey) Key {
return Key{
KeyType: &Key_KmsKey{
KmsKey: &KmsKey{
Arn: mk.Arn,
Role: mk.Role,
Context: ctx,
AwsProfile: mk.AwsProfile,
Arn: mk.Arn,
Role: mk.Role,
Context: ctx,
AwsProfile: mk.AwsProfile,
AwsKmsEndpoint: mk.AwsKmsEndpoint,
AwsStsEndpoint: mk.AwsStsEndpoint,
},
},
}
124 changes: 73 additions & 51 deletions keyservice/keyservice.pb.go
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is automatically generated, you should not manually modify it, but instead modify the protobuf definition and re-generate it. Right now protobuf regeneration doesn't work (see #1576) though, but I hope that will get fixed soon.

Copy link
Author

@anandavj anandavj Dec 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the review, I've modify the protobuf definition and re-generate it
I use #1688 makefile but I won't include it on this PR as it's already covered in another PR

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions keyservice/keyservice.proto
Original file line number Diff line number Diff line change
@@ -22,6 +22,8 @@ message KmsKey {
string role = 2;
map<string, string> context = 3;
string aws_profile = 4;
string aws_kms_endpoint = 5;
string aws_sts_endpoint = 6;
}

message GcpKmsKey {
2 changes: 2 additions & 0 deletions keyservice/server.go
Original file line number Diff line number Diff line change
@@ -323,5 +323,7 @@ func kmsKeyToMasterKey(key *KmsKey) kms.MasterKey {
Role: key.Role,
EncryptionContext: ctx,
AwsProfile: key.AwsProfile,
AwsKmsEndpoint: key.AwsKmsEndpoint,
AwsStsEndpoint: key.AwsStsEndpoint,
}
}
Loading