Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port OS Environment varialbes to viper config #127

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+107 −1,885
Draft

Port OS Environment varialbes to viper config #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
27468e5
port os env variables to viper
jakeschuurmans Sep 25, 2023
befda1e
linting
jakeschuurmans Sep 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
asrr:
s3:
region:
endpoint:
bucket:
access_key:
secret_key:
s3:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between the s3 and asrr.s3 stanzas?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DoctorVin s3 is the destination bucket. asrr s3 is the source bucket for AsRockrack firmware images. It's a hack because our firmware images for ASRR come from a ftp site that's not available publicly so at the time this was added we agreed to keep the fw images in that s3 bucket and sync from there instead of using the private ftp they provide.

access_key:
secret_key:
serverservice:
client_secret:
client_id:
oidc_provider_endpoint:
audience_endpoint:
github:
openbmc_token:
4 changes: 1 addition & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ require (
github.com/google/uuid v1.3.0
github.com/prometheus/client_golang v1.16.0
github.com/rclone/rclone v1.63.1
github.com/spf13/viper v1.16.0
github.com/stretchr/testify v1.8.4
go.hollow.sh/serverservice v0.16.0
golang.org/x/exp v0.0.0-20230811145659-89c5cff77bcb
Expand Down Expand Up @@ -107,7 +108,6 @@ require (
github.com/spf13/cast v1.5.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.16.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tklauser/go-sysconf v0.3.11 // indirect
github.com/tklauser/numcpus v0.6.0 // indirect
Expand All @@ -126,7 +126,6 @@ require (
go.opentelemetry.io/otel v1.16.0 // indirect
go.opentelemetry.io/otel/metric v1.16.0 // indirect
go.opentelemetry.io/otel/trace v1.16.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/goleak v1.2.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.25.0 // indirect
Expand All @@ -141,7 +140,6 @@ require (
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/api v0.137.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230815205213-6bfd019c3878 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230815205213-6bfd019c3878 // indirect
google.golang.org/grpc v1.57.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
Expand Down
1,840 changes: 15 additions & 1,825 deletions go.sum
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 18 additions & 8 deletions internal/app/app.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ import (
"os"

"github.com/bmc-toolbox/common"
"github.com/sirupsen/logrus"

"github.com/metal-toolbox/firmware-syncer/internal/config"
"github.com/metal-toolbox/firmware-syncer/internal/vendors"
"github.com/metal-toolbox/firmware-syncer/internal/vendors/asrockrack"
Expand All @@ -15,6 +13,8 @@ import (
"github.com/metal-toolbox/firmware-syncer/internal/vendors/intel"
"github.com/metal-toolbox/firmware-syncer/internal/vendors/mellanox"
"github.com/metal-toolbox/firmware-syncer/internal/vendors/supermicro"
"github.com/sirupsen/logrus"
"github.com/spf13/viper"
)

var (
Expand Down Expand Up @@ -46,6 +46,16 @@ func New(configFile string, logLevel int) (*Syncer, error) {
logger.SetLevel(logrus.InfoLevel)
}

// Load up configs
v := viper.New()
v.SetConfigFile("config.yaml")
v.AddConfigPath(".")
err := v.ReadInConfig()

if err != nil {
logger.Error("Failed to find viper config file")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably be a Fatal here.

}

// Load configuration
cfgSyncer, err := config.LoadSyncerConfig(configFile)
if err != nil {
Expand All @@ -67,7 +77,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case common.VendorDell:
var dup vendors.Vendor

dup, err = dell.NewDUP(context.TODO(), firmwares, cfgSyncer, logger)
dup, err = dell.NewDUP(context.TODO(), firmwares, cfgSyncer, logger, v)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of passing the viper pointer, it would be nicer (IMO) to parse this into one (or several?) structs that were specific to the vendors?

if err != nil {
logger.Error("Failed to initialize Dell vendor: " + err.Error())
return nil, err
Expand All @@ -77,7 +87,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case common.VendorAsrockrack:
var asrr vendors.Vendor

asrr, err = asrockrack.New(context.TODO(), firmwares, cfgSyncer, logger)
asrr, err = asrockrack.New(context.TODO(), firmwares, cfgSyncer, logger, v)
if err != nil {
logger.Error("Failed to initialize ASRockRack vendor:" + err.Error())
return nil, err
Expand All @@ -87,7 +97,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case common.VendorSupermicro:
var sm vendors.Vendor

sm, err = supermicro.New(context.TODO(), firmwares, cfgSyncer, logger)
sm, err = supermicro.New(context.TODO(), firmwares, cfgSyncer, logger, v)
if err != nil {
logger.Error("Failed to initialize Supermicro vendor: " + err.Error())
return nil, err
Expand All @@ -97,7 +107,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case common.VendorMellanox:
var mlx vendors.Vendor

mlx, err = mellanox.New(context.TODO(), firmwares, cfgSyncer, logger)
mlx, err = mellanox.New(context.TODO(), firmwares, cfgSyncer, logger, v)
if err != nil {
logger.Error("Failed to initialize Mellanox vendor: " + err.Error())
return nil, err
Expand All @@ -107,7 +117,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case common.VendorIntel:
var i vendors.Vendor

i, err = intel.New(context.TODO(), firmwares, cfgSyncer, logger)
i, err = intel.New(context.TODO(), firmwares, cfgSyncer, logger, v)
if err != nil {
logger.Error("Failed to initialize Intel vendor: " + err.Error())
return nil, err
Expand All @@ -117,7 +127,7 @@ func New(configFile string, logLevel int) (*Syncer, error) {
case "equinix":
var e vendors.Vendor

e, err = equinix.New(context.TODO(), firmwares, cfgSyncer, logger)
e, err = equinix.New(context.TODO(), firmwares, cfgSyncer, logger, v)
if err != nil {
logger.Error("Failed to initialize Equinix vendor: " + err.Error())
return nil, err
Expand Down
11 changes: 11 additions & 0 deletions internal/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strings"
"time"

"github.com/sirupsen/logrus"
"github.com/spf13/viper"
"gopkg.in/yaml.v2"

serverservice "go.hollow.sh/serverservice/pkg/api/v1"
Expand Down Expand Up @@ -146,3 +148,12 @@ func ParseRepositoryURL(repositoryURL string) (endpoint, bucket string, err erro

return u.Host, bucket, nil
}

func LoadEnvironmentVariable(v *viper.Viper, logger *logrus.Logger, key string) string {
val := v.Get(key)
if val == nil {
logger.Warnf("Failed to find %s in config.yaml", key)
}

return val.(string)
}
20 changes: 8 additions & 12 deletions internal/inventory/serverservice.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@ package inventory
import (
"context"
"net/url"
"os"
"strings"

"github.com/coreos/go-oidc"
"github.com/google/uuid"
"github.com/metal-toolbox/firmware-syncer/internal/config"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/spf13/viper"
serverservice "go.hollow.sh/serverservice/pkg/api/v1"
"golang.org/x/exp/slices"
"golang.org/x/oauth2/clientcredentials"

serverservice "go.hollow.sh/serverservice/pkg/api/v1"
)

var (
Expand All @@ -27,25 +27,22 @@ type ServerService struct {
logger *logrus.Logger
}

func New(ctx context.Context, serverServiceURL, artifactsURL string, logger *logrus.Logger) (*ServerService, error) {
func New(ctx context.Context, serverServiceURL, artifactsURL string, logger *logrus.Logger, v *viper.Viper) (*ServerService, error) {
if artifactsURL == "" {
return nil, errors.New("missing artifacts URL")
}

clientSecret := os.Getenv("SERVERSERVICE_CLIENT_SECRET")

clientSecret := config.LoadEnvironmentVariable(v, logger, "serverservice.client_secret")
if clientSecret == "" {
return nil, errors.New("missing server service client secret")
}

clientID := os.Getenv("SERVERSERVICE_CLIENT_ID")

clientID := config.LoadEnvironmentVariable(v, logger, "serverservice.client_id")
if clientID == "" {
return nil, errors.New("missing server service client id")
}

oidcProviderEndpoint := os.Getenv("SERVERSERVICE_OIDC_PROVIDER_ENDPOINT")

oidcProviderEndpoint := config.LoadEnvironmentVariable(v, logger, "serverservice.oidc_provider_endpoint")
if oidcProviderEndpoint == "" {
return nil, errors.New("missing server service oidc provider endpoint")
}
Expand All @@ -55,8 +52,7 @@ func New(ctx context.Context, serverServiceURL, artifactsURL string, logger *log
return nil, err
}

audience := os.Getenv("SERVERSERVICE_AUDIENCE_ENDPOINT")

audience := config.LoadEnvironmentVariable(v, logger, "serverservice.audience_endpoint")
if audience == "" {
return nil, errors.New("missing server service audience URL")
}
Expand Down
23 changes: 10 additions & 13 deletions internal/vendors/asrockrack/asrockrack.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,15 @@ package asrockrack

import (
"context"
"os"

"github.com/metal-toolbox/firmware-syncer/internal/config"
"github.com/metal-toolbox/firmware-syncer/internal/inventory"
"github.com/metal-toolbox/firmware-syncer/internal/vendors"

"github.com/pkg/errors"
rcloneFs "github.com/rclone/rclone/fs"
rcloneOperations "github.com/rclone/rclone/fs/operations"
"github.com/sirupsen/logrus"
"github.com/spf13/viper"
serverservice "go.hollow.sh/serverservice/pkg/api/v1"
)

Expand All @@ -28,20 +27,18 @@ type ASRockRack struct {
tmpFs rcloneFs.Fs
}

func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger) (vendors.Vendor, error) {
func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger, v *viper.Viper) (vendors.Vendor, error) {
// RepositoryURL required
if cfgSyncer.RepositoryURL == "" {
return nil, errors.Wrap(config.ErrProviderAttributes, "RepositoryURL not defined")
}

// TODO: For now set this configuration from env vars but ideally this should come from
// somewhere else. Maybe a per provider config?
srcS3Config := &config.S3Bucket{
Region: os.Getenv("ASRR_S3_REGION"),
Endpoint: os.Getenv("ASRR_S3_ENDPOINT"),
Bucket: os.Getenv("ASRR_S3_BUCKET"),
AccessKey: os.Getenv("ASRR_S3_ACCESS_KEY"),
SecretKey: os.Getenv("ASRR_S3_SECRET_KEY"),
Region: config.LoadEnvironmentVariable(v, logger, "asrr.s3.region"),
Endpoint: config.LoadEnvironmentVariable(v, logger, "asrr.s3.endpoint"),
Bucket: config.LoadEnvironmentVariable(v, logger, "asrr.s3.bucket"),
AccessKey: config.LoadEnvironmentVariable(v, logger, "asrr.s3.access_key"),
SecretKey: config.LoadEnvironmentVariable(v, logger, "asrr.s3.secret_key"),
}

// parse S3 endpoint and bucket from cfgSyncer.RepositoryURL
Expand All @@ -54,12 +51,12 @@ func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersio
Region: cfgSyncer.RepositoryRegion,
Endpoint: s3DstEndpoint,
Bucket: s3DstBucket,
AccessKey: os.Getenv("S3_ACCESS_KEY"),
SecretKey: os.Getenv("S3_SECRET_KEY"),
AccessKey: config.LoadEnvironmentVariable(v, logger, "s3.access_key"),
SecretKey: config.LoadEnvironmentVariable(v, logger, "s3.secret_key"),
}

// init inventory
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger)
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger, v)
if err != nil {
return nil, err
}
Expand Down
10 changes: 5 additions & 5 deletions internal/vendors/dell/dell.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ package dell

import (
"context"
"os"

"github.com/metal-toolbox/firmware-syncer/internal/config"
"github.com/metal-toolbox/firmware-syncer/internal/inventory"
"github.com/metal-toolbox/firmware-syncer/internal/vendors"
"github.com/spf13/viper"

"github.com/pkg/errors"
rcloneFs "github.com/rclone/rclone/fs"
Expand All @@ -29,7 +29,7 @@ type DUP struct {
}

// NewDUP returns a new DUP firmware syncer object
func NewDUP(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger) (vendors.Vendor, error) {
func NewDUP(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger, v *viper.Viper) (vendors.Vendor, error) {
// RepositoryURL required
if cfgSyncer.RepositoryURL == "" {
return nil, errors.Wrap(config.ErrProviderAttributes, "RepositoryURL not defined")
Expand All @@ -45,12 +45,12 @@ func NewDUP(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVer
Region: cfgSyncer.RepositoryRegion,
Endpoint: s3Endpoint,
Bucket: s3Bucket,
AccessKey: os.Getenv("S3_ACCESS_KEY"),
SecretKey: os.Getenv("S3_SECRET_KEY"),
AccessKey: config.LoadEnvironmentVariable(v, logger, "s3.access_key"),
SecretKey: config.LoadEnvironmentVariable(v, logger, "s3.secret_key"),
}

// init inventory
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger)
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger, v)
if err != nil {
return nil, err
}
Expand Down
12 changes: 6 additions & 6 deletions internal/vendors/equinix/equinix.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ import (
"fmt"
"net/http"
"net/url"
"os"
"strings"
"time"

"github.com/google/go-github/v53/github"
"github.com/metal-toolbox/firmware-syncer/internal/config"
"github.com/metal-toolbox/firmware-syncer/internal/inventory"
"github.com/metal-toolbox/firmware-syncer/internal/vendors"
"github.com/spf13/viper"
"golang.org/x/oauth2"

"github.com/pkg/errors"
Expand All @@ -36,14 +36,14 @@ type Equinix struct {
tmpFs fs.Fs
}

func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger) (vendors.Vendor, error) {
func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersion, cfgSyncer *config.Syncer, logger *logrus.Logger, v *viper.Viper) (vendors.Vendor, error) {
// RepositoryURL required
if cfgSyncer.RepositoryURL == "" {
return nil, errors.Wrap(config.ErrProviderAttributes, "RepositoryURL not defined")
}

ts := oauth2.StaticTokenSource(
&oauth2.Token{AccessToken: os.Getenv("GITHUB_OPENBMC_TOKEN")},
&oauth2.Token{AccessToken: config.LoadEnvironmentVariable(v, logger, "github.openbmc_token")},
)
tc := oauth2.NewClient(ctx, ts)

Expand All @@ -59,12 +59,12 @@ func New(ctx context.Context, firmwares []*serverservice.ComponentFirmwareVersio
Region: cfgSyncer.RepositoryRegion,
Endpoint: s3DstEndpoint,
Bucket: s3DstBucket,
AccessKey: os.Getenv("S3_ACCESS_KEY"),
SecretKey: os.Getenv("S3_SECRET_KEY"),
AccessKey: config.LoadEnvironmentVariable(v, logger, "s3.access_key"),
SecretKey: config.LoadEnvironmentVariable(v, logger, "s3.secret_key"),
}

// init inventory
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger)
i, err := inventory.New(ctx, cfgSyncer.ServerServiceURL, cfgSyncer.ArtifactsURL, logger, v)
if err != nil {
return nil, err
}
Expand Down
Loading
Loading