-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Cloudstack multiple endpoint for preflight checks #2559
Support Cloudstack multiple endpoint for preflight checks #2559
Conversation
Skipping CI for Draft Pull Request. |
Codecov Report
@@ Coverage Diff @@
## main #2559 +/- ##
==========================================
+ Coverage 57.49% 57.85% +0.36%
==========================================
Files 310 310
Lines 25562 25555 -7
==========================================
+ Hits 14696 14786 +90
+ Misses 9529 9453 -76
+ Partials 1337 1316 -21
Continue to review full report at Codecov.
|
return &Validator{ | ||
cmk: cmk, | ||
cmks: cmks, | ||
availabilityZones: []localAvailabilityZone{}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
availabilityZones: []localAvailabilityZone{}, | |
localAvailabilityZones: []localAvailabilityZone{}, |
I think we should keep the local prefix whenever we can to avoid confusion and clearly indicate we're not using the actual spec availabilityZones.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good
internal/test/cleanup/cleanup.go
Outdated
if err := cleanupCloudStackVms(ctx, cmk, clusterName, dryRun); err != nil { | ||
retErr = multierror.Append(retErr, err) | ||
} | ||
cmk.Close(ctx) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we defer this close? or does that not work in loops?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think to make defer
to work I should pull a function that processes one cmk
.
pkg/dependencies/factory.go
Outdated
@@ -266,12 +266,17 @@ func (f *Factory) WithProvider(clusterConfigFile string, clusterConfig *v1alpha1 | |||
return fmt.Errorf("unable to get machine config from file %s: %v", clusterConfigFile, err) | |||
} | |||
|
|||
cmkClientMap := cloudstack.CmkClientMap{} | |||
for name, cmk := range f.dependencies.Cmks { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you add a comment explaining this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
pkg/dependencies/factory.go
Outdated
@@ -414,16 +419,21 @@ func (f *Factory) WithCmk() *Factory { | |||
f.WithExecutableBuilder().WithWriter() | |||
|
|||
f.buildSteps = append(f.buildSteps, func(ctx context.Context) error { | |||
if f.dependencies.Cmk != nil { | |||
if f.dependencies.Cmks != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we also check len(f.dependencies.Cmks) > 0?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
pkg/executables/cmk.go
Outdated
if len(zone.Network.Id) > 0 { | ||
applyCmkArgs(&command, withCloudStackId(zone.Network.Id)) | ||
if len(network.Id) > 0 { | ||
applyCmkArgs(&command, withCloudStackId(network.Id)) | ||
} | ||
if multipleZone { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we still need this parameter? I think we could get rid of it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like multipleZone
is still here
} | ||
|
||
for _, azName := range azNamesToCheck { | ||
cmk, ok := v.cmks[azName] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should use the AZ's credentialRef not the AZ's name to look up the cmk mapping
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point!
return fmt.Errorf("zone network is not set or is empty") | ||
cmk, ok := v.cmks[az.CredentialsRef] | ||
if !ok { | ||
return fmt.Errorf("cannot find CloudStack profile for availability zone %s", az.CredentialsRef) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the error message here is confusing.
return fmt.Errorf("cannot find CloudStack profile for availability zone %s", az.CredentialsRef) | |
return fmt.Errorf("cannot find CloudStack profile named %s for availability zone %s", az.CredentialsRef, az.Name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay
Domain: datacenterConfig.Spec.Domain, | ||
Account: datacenterConfig.Spec.Account, | ||
ManagementApiEndpoint: datacenterConfig.Spec.ManagementApiEndpoint, | ||
Zone: zone, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this seems to be missing a name, doesn't it? I think we need the name too for some of the error messages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding Name as SetDefaults does
v.availabilityZones = append(v.availabilityZones, availabilityZone) | ||
} | ||
} | ||
for _, az := range datacenterConfig.Spec.AvailabilityZones { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need a check for nil before iterating over the items in an array?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need, I think. AvailablityZones is []CloudStackAvailabilityZone and it will be initialized as an empty list.
if (datacenterConfig.Spec.Domain != "" && datacenterConfig.Spec.Account == "") || | ||
(datacenterConfig.Spec.Domain == "" && datacenterConfig.Spec.Account != "") { | ||
return fmt.Errorf("both domain and account must be specified or none of them must be specified") | ||
func (v *Validator) generateLocalAvailabilityZones(ctx context.Context, datacenterConfig *anywherev1.CloudStackDatacenterConfig) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it seems like we're doing two things in this function - generating local AZ's, and also validating their accounts/domains. I'd suggest simplifying this to just generate the AZ's, and do the validation of them in ValidateCloudStackDatacenterConfig
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good
@@ -86,22 +86,33 @@ func (v *Validator) ValidateCloudStackDatacenterConfig(ctx context.Context, data | |||
return err | |||
} | |||
|
|||
for _, az := range v.availabilityZones { | |||
for _, az := range v.localAvailabilityZones { | |||
fmt.Printf("az: %+v\n", az.CloudStackAvailabilityZone) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fmt.Printf("az: %+v\n", az.CloudStackAvailabilityZone) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops. Removed.
@@ -0,0 +1,1646 @@ | |||
apiVersion: anywhere.eks.amazonaws.com/v1alpha1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need this file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just need some more test coverage and we're in business, great work :)
internal/test/cleanup/cleanup.go
Outdated
for _, config := range execConfig.Profiles { | ||
cmk := executableBuilder.BuildCmkExecutable(tmpWriter, config) | ||
if err := cleanupCloudStackVms(ctx, cmk, clusterName, dryRun); err != nil { | ||
retErr = multierror.Append(retErr, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this error look? Just wondering if we need to import a new library or solve it a different way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The multierror module is useful when we want to continue to clean up the resources on encountering an error so that we can clean up as many resources as possible. The appended errors are integrated into a single error. If you think introducing a new module just for testing isn't a good idea I can remove multierror module from this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
my vote would be removing this third party module
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing it
pkg/dependencies/factory.go
Outdated
@@ -50,7 +50,7 @@ type Dependencies struct { | |||
DockerClient *executables.Docker | |||
Kubectl *executables.Kubectl | |||
Govc *executables.Govc | |||
Cmk *executables.Cmk | |||
Cmks map[string]*executables.Cmk |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to build a specific binary for each of the configs? Can't this just be solved by setting the appropriate envvar per command execution somehow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cmk is an object combining cmk
binary and cloud-config
to a certain ACS endpoint. The binary is the same across the list of Cmks but the configurations are different. This way we could keep the most of Cmk.go code as it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can cmk exec takes profile for each cmd? So that we do not need to preset the configuration and have a map of exact same exec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to use single cmk instead of multiple cmk objects
} | ||
return zoneIdentifiers, nil | ||
return cmkZones[0].Id, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the purpose of returning the string here now? It seems like the id will be the same as what would be passed in?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The input argument in this function is the name of a zone. This function resolves the name into the ID while validating the zone's presence by name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds like this would be better named ValidateAndGetZoneId
especially because the other Validate methods only return an error, but yea I get it now.
internal/test/cleanup/cleanup.go
Outdated
for _, config := range execConfig.Profiles { | ||
cmk := executableBuilder.BuildCmkExecutable(tmpWriter, config) | ||
if err := cleanupCloudStackVms(ctx, cmk, clusterName, dryRun); err != nil { | ||
retErr = multierror.Append(retErr, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
my vote would be removing this third party module
internal/test/cleanup/cleanup.go
Outdated
|
||
func cleanupCloudStackVms(ctx context.Context, cmk *executables.Cmk, clusterName string, dryRun bool) error { | ||
if err := cmk.ValidateCloudStackConnection(ctx); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to validate cloudstack connection? the cleanupVms will fail w same connection error anyway if there's connection issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing ValidateCloudStackConnection
pkg/dependencies/factory.go
Outdated
@@ -50,7 +50,7 @@ type Dependencies struct { | |||
DockerClient *executables.Docker | |||
Kubectl *executables.Kubectl | |||
Govc *executables.Govc | |||
Cmk *executables.Cmk | |||
Cmks map[string]*executables.Cmk |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can cmk exec takes profile for each cmd? So that we do not need to preset the configuration and have a map of exact same exec.
writer, | ||
now, | ||
skipIpCheck, | ||
) | ||
} | ||
|
||
func NewProviderCustomNet(datacenterConfig *v1alpha1.CloudStackDatacenterConfig, machineConfigs map[string]*v1alpha1.CloudStackMachineConfig, clusterConfig *v1alpha1.Cluster, providerKubectlClient ProviderKubectlClient, providerCmkClient ProviderCmkClient, writer filewriter.FileWriter, now types.NowFunc, skipIpCheck bool) *cloudstackProvider { | ||
func NewProviderCustomNet(datacenterConfig *v1alpha1.CloudStackDatacenterConfig, machineConfigs map[string]*v1alpha1.CloudStackMachineConfig, clusterConfig *v1alpha1.Cluster, providerKubectlClient ProviderKubectlClient, providerCmkClients CmkClientMap, writer filewriter.FileWriter, now types.NowFunc, skipIpCheck bool) *cloudstackProvider { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not needed. Removing NewProviderCustomNet
|
||
apiKey, err := section.GetKey("api-key") | ||
if err != nil { | ||
return nil, fmt.Errorf("failed to extract value of 'api-key' from %s: %v", section.Name(), err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
return nil, fmt.Errorf("failed to extract value of 'api-key' from %s: %v", section.Name(), err) | |
return nil, fmt.Errorf("extracting value of 'api-key' from %s: %v", section.Name(), err) |
same for below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updating
return nil, fmt.Errorf("failed to extract value of 'api-url' from %s: %v", EksacloudStackCloudConfigB64SecretKey, err) | ||
} | ||
verifySslValue := "true" | ||
if verifySsl, err := section.GetKey("verify-ssl"); err == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about err != nil?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If err == nil then the default value of "true" will be used.
if err != nil { | ||
return nil, fmt.Errorf("failed to extract value of 'api-url' from %s: %v", EksacloudStackCloudConfigB64SecretKey, err) | ||
} | ||
verifySslValue := "true" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the default verifySsl is true?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be a constant then so we can change the default easily?
return &Validator{ | ||
cmk: cmk, | ||
cmks: cmks, | ||
localAvailabilityZones: []localAvailabilityZone{}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you should have localAvailabilityZones
in the validator struct if you are generating it from one of the validation method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point
internal/test/cleanup/cleanup.go
Outdated
} | ||
|
||
return cmk.CleanupVms(ctx, clusterName, dryRun) | ||
if len(failedProfiles) > 0 { | ||
return fmt.Errorf("cleaning up VMs: profiles=%+v, errors=%+v", failedProfiles, errors) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we be doing a profile=error mapping instead if that is the intention here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated. The new code will return an error with this kind of message.
cleaning up VMs: map[acs#1:Unknown exception acs#2:Unable to reach]
} | ||
return zoneIdentifiers, nil | ||
return cmkZones[0].Id, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds like this would be better named ValidateAndGetZoneId
especially because the other Validate methods only return an error, but yea I get it now.
|
||
for _, instance := range execConfig.Profiles { | ||
if err := p.validateManagementApiEndpoint(instance.ManagementUrl); err != nil { | ||
return fmt.Errorf("CloudStack instance %s's managementApiEndpoint %s is invalid", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to print the error message here too?
if err != nil { | ||
return nil, fmt.Errorf("failed to extract value of 'api-url' from %s: %v", EksacloudStackCloudConfigB64SecretKey, err) | ||
} | ||
verifySslValue := "true" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be a constant then so we can change the default easily?
|
||
logger.MarkPass("Connected to", "servers", refNamesToCheck) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you paste an example here for reference?
- ValidateZonePresent -> ValidateZoneAndGetId - ValidateDomainPresent -> ValidateDomainAndGetId Change return value of ValidateDomainAndGetId - from CloudStackResourceIdentifier to string because only domainId is used
} | ||
|
||
err = p.validateMachineConfigsNameUniqueness(ctx, cluster, clusterSpec) | ||
if err != nil { | ||
if err := p.validateMachineConfigsNameUniqueness(ctx, cluster, clusterSpec); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't this be part of validateClusterSpec
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a code clean up change.
from
err = func1()
if err != nil {}
to
if err := func1(); err != nil {}
If you think validateMachineConfigsNameUniqueness is a part of validateClusterSpec then I should be handled in a separate PR because this pattern shows up in vsphere provider as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think vsphere provider code is a good reference. But yea separate pr sounds good to me.
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vivek-koppuru The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Issue #, if available: #2406
Description of changes:
Cloudstack provider will support multiple endpoints, therefore eksa preflight check also needs to support them.
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.