Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Multi-Assignments #833

Merged
merged 59 commits into from
May 16, 2019
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
e797215
First draft of new assignment logic
stefbehl Mar 6, 2019
63e95f6
Enhancements of System Configuration view
stefbehl Mar 7, 2019
aa81448
Drag&drop enhancements for multiple distribution set assignments
stefbehl Mar 8, 2019
11d946d
Misc fixes
stefbehl Mar 13, 2019
aaa1558
test that previous assignments are not canceled
StefanKlt Mar 15, 2019
22b0567
add description and expected events to test
StefanKlt Mar 15, 2019
46bbbd8
extend TenantConfigurationManagement by NullPointerException proof
StefanKlt Mar 25, 2019
c6f763f
Hide "Required Migration Step" if Multi Assignments is enabled
StefanKlt Mar 25, 2019
a664187
Make fields transient
StefanKlt Mar 25, 2019
d855a12
Fix merge conflicts
StefanKlt Mar 26, 2019
b4d7a80
Add IDs for Required Migration Step elements
StefanKlt Mar 27, 2019
7573b7b
Save work in progress
stefbehl Mar 29, 2019
b73e452
Added new DMF message DmfMultiActionRequest
stefbehl Apr 3, 2019
19d48c2
DMF enhancements to send out MultiActionRequest messages
stefbehl Apr 3, 2019
d35d6d5
Minor changes
stefbehl Apr 3, 2019
20f733d
Multi Assignment support for cancellations
stefbehl Apr 5, 2019
c7ab510
fix permission problems and immutable lists
StefanKlt Apr 18, 2019
0bcb9a1
merge download only
StefanKlt May 9, 2019
ef38921
add message dispatcher tests for outgoing multiassignment messages, fix
StefanKlt Apr 30, 2019
9d9adbb
Implement Multi-Assignment support for rollout groups
stefbehl May 2, 2019
7f30eed
Minor changes
stefbehl May 2, 2019
efde30e
Refactoring
stefbehl May 2, 2019
cc4e5d0
Register new deployment event with protobuff framework
stefbehl May 3, 2019
018800d
Allow same DS to be assigned multiple times
stefbehl May 3, 2019
c4c5fae
Fix assignment with pending cancellations
stefbehl May 3, 2019
eb09c54
Reduce repository /DB calls
stefbehl May 3, 2019
661e509
Revert latest perf fix (causing a regression)
stefbehl May 3, 2019
bec7599
test if a rollout sends multiaction messages in multiassignment mode
StefanKlt May 3, 2019
d93b137
Minor changes
stefbehl May 3, 2019
e8c8314
add test that starts and finishes multiple rollouts in multiassignt mode
StefanKlt May 3, 2019
509011e
Do not close new action if DS is already assigned (if multi-assign on)
stefbehl May 3, 2019
e3a18c5
add javadoc to test method
StefanKlt May 6, 2019
d02b8cd
Prevent Multi-Assignments from being disabled via Repo Config UI
stefbehl May 7, 2019
3a9d6a9
Merge branch 'feature_multi_assignments' of https://github.com/bsinno…
stefbehl May 7, 2019
15d64b2
Add link to Provisioning State Machine
stefbehl May 7, 2019
95df24b
test that Multiassignment can not be disabled via mgmt-api
StefanKlt May 8, 2019
e88745b
refactor AmqpMessageHandlerService code
StefanKlt May 8, 2019
38f3b53
Prevent Multi-Assignments feature from being disabled via Mgmt REST API
stefbehl May 8, 2019
7524abe
add license header, remove unused instance variables
StefanKlt May 9, 2019
c54783f
fix tenantConfigurationManagement mock in test
StefanKlt May 9, 2019
8fba9e5
return empty list instead of null
StefanKlt May 9, 2019
d2e8384
add ddi test for multiassignment, fix old test
StefanKlt May 10, 2019
0d09537
Prevent autoclose from being modified if Multi-Assignments is enabled
stefbehl May 10, 2019
89d1c80
Add test for autoClose /multiAssign
stefbehl May 10, 2019
c573c90
Javadoc improvements
stefbehl May 10, 2019
3f651c5
change test method that waits for dmf messages to be dispatched
StefanKlt May 10, 2019
6aeca26
clean up code
StefanKlt May 13, 2019
9e7684b
Fix UI-related PR review findings
stefbehl May 14, 2019
d61ded4
Fix PR review findings
stefbehl May 14, 2019
65b622b
Fix PR review findings
stefbehl May 14, 2019
3e415e9
Fix PR review findings
stefbehl May 14, 2019
a308a96
Fix PR review findings, Sonar issues, and test failures
stefbehl May 14, 2019
a0c699a
Fix PR review findings
stefbehl May 14, 2019
7200bed
Fix PR review findings
stefbehl May 14, 2019
f691f5c
Fix PR review findings
stefbehl May 14, 2019
3528b74
Fix Sonar findings
stefbehl May 14, 2019
070bafb
Fix PR review findings
stefbehl May 14, 2019
f61da79
Fix PR review findings
stefbehl May 15, 2019
353e931
Fix PR review findings
stefbehl May 15, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ public enum SpServerError {
*/
SP_CONFIGURATION_VALUE_INVALID("hawkbit.server.error.configValueInvalid",
"The given configuration value is invalid."),

/**
*
*/
Expand Down Expand Up @@ -232,7 +233,14 @@ public enum SpServerError {
* invalid.
*/
SP_AUTO_ASSIGN_DISTRIBUTION_SET_INVALID("hawkbit.server.error.repo.invalidAutoAssignDistributionSet",
"The given distribution set for auto-assignment is invalid: it is either incomplete (i.e. mandatory modules are missing) or soft deleted");
"The given distribution set for auto-assignment is invalid: it is either incomplete (i.e. mandatory modules are missing) or soft deleted"),

/**
* Error message informing the user that the requested tenant configuration
* change is not allowed.
*/
SP_CONFIGURATION_VALUE_CHANGE_NOT_ALLOWED("hawkbit.server.error.repo.tenantConfigurationValueChangeNotAllowed",
"The requested tenant configuration value modification is not allowed.");

private final String key;
private final String message;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import org.eclipse.hawkbit.dmf.amqp.api.AmqpSettings;
import org.eclipse.hawkbit.repository.ArtifactManagement;
import org.eclipse.hawkbit.repository.ControllerManagement;
import org.eclipse.hawkbit.repository.DeploymentManagement;
import org.eclipse.hawkbit.repository.DistributionSetManagement;
import org.eclipse.hawkbit.repository.EntityFactory;
import org.eclipse.hawkbit.repository.SoftwareModuleManagement;
Expand Down Expand Up @@ -240,9 +241,11 @@ public Binding bindDeadLetterQueueToDeadLetterExchange() {
@Bean
public AmqpMessageHandlerService amqpMessageHandlerService(final RabbitTemplate rabbitTemplate,
final AmqpMessageDispatcherService amqpMessageDispatcherService,
final ControllerManagement controllerManagement, final EntityFactory entityFactory) {
final ControllerManagement controllerManagement, final EntityFactory entityFactory,
final SystemSecurityContext systemSecurityContext,
final TenantConfigurationManagement tenantConfigurationManagement) {
return new AmqpMessageHandlerService(rabbitTemplate, amqpMessageDispatcherService, controllerManagement,
entityFactory);
entityFactory, systemSecurityContext, tenantConfigurationManagement);
}

/**
Expand Down Expand Up @@ -319,10 +322,10 @@ AmqpMessageDispatcherService amqpMessageDispatcherService(final RabbitTemplate r
final AmqpMessageSenderService amqpSenderService, final ArtifactUrlHandler artifactUrlHandler,
final SystemSecurityContext systemSecurityContext, final SystemManagement systemManagement,
final TargetManagement targetManagement, final DistributionSetManagement distributionSetManagement,
final SoftwareModuleManagement softwareModuleManagement) {
final SoftwareModuleManagement softwareModuleManagement, final DeploymentManagement deploymentManagement) {
return new AmqpMessageDispatcherService(rabbitTemplate, amqpSenderService, artifactUrlHandler,
systemSecurityContext, systemManagement, targetManagement, serviceMatcher, distributionSetManagement,
softwareModuleManagement);
softwareModuleManagement, deploymentManagement);
}

private static Map<String, Object> getTTLMaxArgsAuthenticationQueue() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@
*/
package org.eclipse.hawkbit.amqp;

import static org.eclipse.hawkbit.repository.RepositoryConstants.MAX_ACTION_COUNT;

import java.net.URI;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
Expand All @@ -28,19 +31,23 @@
import org.eclipse.hawkbit.dmf.json.model.DmfArtifactHash;
import org.eclipse.hawkbit.dmf.json.model.DmfDownloadAndUpdateRequest;
import org.eclipse.hawkbit.dmf.json.model.DmfMetadata;
import org.eclipse.hawkbit.dmf.json.model.DmfMultiActionRequest;
import org.eclipse.hawkbit.dmf.json.model.DmfSoftwareModule;
import org.eclipse.hawkbit.repository.DeploymentManagement;
import org.eclipse.hawkbit.repository.DistributionSetManagement;
import org.eclipse.hawkbit.repository.RepositoryConstants;
import org.eclipse.hawkbit.repository.SoftwareModuleManagement;
import org.eclipse.hawkbit.repository.SystemManagement;
import org.eclipse.hawkbit.repository.TargetManagement;
import org.eclipse.hawkbit.repository.event.remote.DeploymentEvent;
import org.eclipse.hawkbit.repository.event.remote.TargetAssignDistributionSetEvent;
import org.eclipse.hawkbit.repository.event.remote.TargetAttributesRequestedEvent;
import org.eclipse.hawkbit.repository.event.remote.TargetDeletedEvent;
import org.eclipse.hawkbit.repository.event.remote.entity.CancelTargetAssignmentEvent;
import org.eclipse.hawkbit.repository.model.Action;
import org.eclipse.hawkbit.repository.model.ActionProperties;
import org.eclipse.hawkbit.repository.model.Artifact;
import org.eclipse.hawkbit.repository.model.DistributionSet;
import org.eclipse.hawkbit.repository.model.SoftwareModule;
import org.eclipse.hawkbit.repository.model.SoftwareModuleMetadata;
import org.eclipse.hawkbit.repository.model.Target;
Expand Down Expand Up @@ -79,6 +86,7 @@ public class AmqpMessageDispatcherService extends BaseAmqpService {
private final TargetManagement targetManagement;
private final ServiceMatcher serviceMatcher;
private final DistributionSetManagement distributionSetManagement;
private final DeploymentManagement deploymentManagement;
private final SoftwareModuleManagement softwareModuleManagement;

/**
Expand Down Expand Up @@ -107,7 +115,7 @@ protected AmqpMessageDispatcherService(final RabbitTemplate rabbitTemplate,
final SystemSecurityContext systemSecurityContext, final SystemManagement systemManagement,
final TargetManagement targetManagement, final ServiceMatcher serviceMatcher,
final DistributionSetManagement distributionSetManagement,
final SoftwareModuleManagement softwareModuleManagement) {
final SoftwareModuleManagement softwareModuleManagement, final DeploymentManagement deploymentManagement) {
super(rabbitTemplate);
this.artifactUrlHandler = artifactUrlHandler;
this.amqpSenderService = amqpSenderService;
Expand All @@ -117,6 +125,7 @@ protected AmqpMessageDispatcherService(final RabbitTemplate rabbitTemplate,
this.serviceMatcher = serviceMatcher;
this.distributionSetManagement = distributionSetManagement;
this.softwareModuleManagement = softwareModuleManagement;
this.deploymentManagement = deploymentManagement;
}

/**
Expand All @@ -134,47 +143,146 @@ protected void targetAssignDistributionSet(final TargetAssignDistributionSetEven

LOG.debug("targetAssignDistributionSet retrieved. I will forward it to DMF broker.");

distributionSetManagement.get(assignedEvent.getDistributionSetId()).ifPresent(set -> {
distributionSetManagement.get(assignedEvent.getDistributionSetId()).ifPresent(ds -> {

final Map<SoftwareModule, List<SoftwareModuleMetadata>> modules = Maps
.newHashMapWithExpectedSize(set.getModules().size());
set.getModules()
.forEach(
module -> modules.put(module,
softwareModuleManagement.findMetaDataBySoftwareModuleIdAndTargetVisible(
PageRequest.of(0, RepositoryConstants.MAX_META_DATA_COUNT), module.getId())
.getContent()));
final Map<SoftwareModule, List<SoftwareModuleMetadata>> softwareModules = getSoftwareModulesWithMetadata(
ds);

targetManagement.getByControllerID(assignedEvent.getActions().keySet()).forEach(
target -> sendUpdateMessageToTarget(assignedEvent.getActions().get(target.getControllerId()),
target, modules));
target, softwareModules));

});
}

/**
* Method to send a message to a RabbitMQ exchange after the Distribution
* set has been assign to a Target.
*
* @param e
* the object to be send.
*/
@EventListener(classes = DeploymentEvent.class)
protected void onDeployment(final DeploymentEvent e) {
if (isNotFromSelf(e)) {
return;
}

LOG.debug("DeploymentEvent received. I will forward it to DMF broker.");
sendMultiActionRequestMessages(e.getTenant(), e.getControllerIds());
}

protected void sendMultiActionRequestMessages(final String tenant, final List<String> controllerIds) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check if this method can be optimised by better query support of repository. You basically read for every target all active actions, their distribution sets and last but not least all contained software modules. Not sure if this is working out in case of multiple big rollouts addressing same targets

Copy link
Contributor Author

@stefbehl stefbehl May 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, once the same distribution set is encountered again, the software modules incl their metadata are found in the cache (the softwareModuleMetadata map). The targets are currently fetched separately as we want to send a DMF message to each target. We could try to get the actions for all controller IDs using a single DB query but that would not rid us from validating the AMQP addresses and after all we'd need to sort it back again per target before sending the DMF messages.


final Map<Long, Map<SoftwareModule, List<SoftwareModuleMetadata>>> softwareModuleMetadata = new HashMap<>();
targetManagement.getByControllerID(controllerIds).stream().forEach(target -> {

final List<Action> activeActions = deploymentManagement
.findActiveActionsByTarget(PageRequest.of(0, MAX_ACTION_COUNT), target.getControllerId())
.getContent();

activeActions.forEach(action -> {
final DistributionSet distributionSet = action.getDistributionSet();
softwareModuleMetadata.computeIfAbsent(distributionSet.getId(),
id -> getSoftwareModulesWithMetadata(distributionSet));
});

if (!activeActions.isEmpty()) {
sendMultiActionRequestToTarget(tenant, target, activeActions, softwareModuleMetadata);
}

});

}

protected void sendMultiActionRequestToTarget(final String tenant, final Target target, final List<Action> actions,
final Map<Long, Map<SoftwareModule, List<SoftwareModuleMetadata>>> softwareModulesPerDistributionSet) {

final URI targetAdress = target.getAddress();
if (!IpUtil.isAmqpUri(targetAdress) || CollectionUtils.isEmpty(actions)) {
return;
}

final DmfMultiActionRequest multiActionRequest = new DmfMultiActionRequest();
actions.forEach(action -> {
final DmfActionRequest actionRequest = createDmfActionRequest(target, action,
softwareModulesPerDistributionSet.get(action.getDistributionSet().getId()));
multiActionRequest.addElement(getEventTypeForAction(action), actionRequest);
});

final Message message = getMessageConverter().toMessage(multiActionRequest,
createConnectorMessagePropertiesEvent(tenant, target.getControllerId(), EventTopic.MULTI_ACTION));
amqpSenderService.sendMessage(message, targetAdress);

}

private DmfActionRequest createDmfActionRequest(final Target target, final Action action,
final Map<SoftwareModule, List<SoftwareModuleMetadata>> softwareModules) {
if (action.isCancelingOrCanceled()) {
return createPlainActionRequest(action);
}
return createDownloadAndUpdateRequest(target, action, softwareModules);

}

private static DmfActionRequest createPlainActionRequest(final Action action) {
final DmfActionRequest actionRequest = new DmfActionRequest();
actionRequest.setActionId(action.getId());
return actionRequest;
}

private DmfDownloadAndUpdateRequest createDownloadAndUpdateRequest(final Target target, final Action action,
final Map<SoftwareModule, List<SoftwareModuleMetadata>> softwareModules) {
final DmfDownloadAndUpdateRequest request = new DmfDownloadAndUpdateRequest();
request.setActionId(action.getId());
request.setTargetSecurityToken(systemSecurityContext.runAsSystem(target::getSecurityToken));
if (softwareModules != null) {
softwareModules.entrySet()
.forEach(entry -> request.addSoftwareModule(convertToAmqpSoftwareModule(target, entry)));
}
return request;
}

/**
* Method to get the type of event depending on whether the action is a
* DOWNLOAD_ONLY action or if it has a valid maintenance window available
* or not based on defined maintenance schedule. In case of no maintenance
* schedule or if there is a valid window available, the topic {@link EventTopic#DOWNLOAD_AND_INSTALL} is
* returned else {@link EventTopic#DOWNLOAD} is returned.
* DOWNLOAD_ONLY action or if it has a valid maintenance window available or
* not based on defined maintenance schedule. In case of no maintenance
* schedule or if there is a valid window available, the topic
* {@link EventTopic#DOWNLOAD_AND_INSTALL} is returned else
* {@link EventTopic#DOWNLOAD} is returned.
*
* @param action
* current action properties.
*
* @return {@link EventTopic} to use for message.
*/
private static EventTopic getEventTypeForTarget(final ActionProperties action) {
return (Action.ActionType.DOWNLOAD_ONLY.equals(action.getActionType()) ||
!action.isMaintenanceWindowAvailable()) ? EventTopic.DOWNLOAD : EventTopic.DOWNLOAD_AND_INSTALL;
return (Action.ActionType.DOWNLOAD_ONLY.equals(action.getActionType())
|| !action.isMaintenanceWindowAvailable()) ? EventTopic.DOWNLOAD : EventTopic.DOWNLOAD_AND_INSTALL;
}

/**
* Determines the {@link EventTopic} for the given {@link Action}, depending
* on its action type.
*
* @param action
* to obtain the corresponding {@link EventTopic} for
*
* @return the {@link EventTopic} for this action
*/
private static EventTopic getEventTypeForAction(final Action action) {
if (action.isCancelingOrCanceled()) {
return EventTopic.CANCEL_DOWNLOAD;
}
return getEventTypeForTarget(new ActionProperties(action));
}

/**
* Method to send a message to a RabbitMQ Exchange after the assignment of
* the Distribution set to a Target has been canceled.
*
* @param cancelEvent
* the object to be send.
* that is to be converted to a DMF message
*/
@EventListener(classes = CancelTargetAssignmentEvent.class)
protected void targetCancelAssignmentToDistributionSet(final CancelTargetAssignmentEvent cancelEvent) {
Expand Down Expand Up @@ -211,7 +319,7 @@ protected void targetTriggerUpdateAttributes(final TargetAttributesRequestedEven
protected void sendUpdateMessageToTarget(final ActionProperties action, final Target target,
final Map<SoftwareModule, List<SoftwareModuleMetadata>> modules) {

String tenant = action.getTenant();
final String tenant = action.getTenant();

final URI targetAdress = target.getAddress();
if (!IpUtil.isAmqpUri(targetAdress)) {
Expand Down Expand Up @@ -361,4 +469,17 @@ private DmfArtifact convertArtifact(final Target target, final Artifact localArt
return artifact;
}

}
private Map<SoftwareModule, List<SoftwareModuleMetadata>> getSoftwareModulesWithMetadata(
final DistributionSet distributionSet) {
final Map<SoftwareModule, List<SoftwareModuleMetadata>> moduleMetadata = Maps
.newHashMapWithExpectedSize(distributionSet.getModules().size());
distributionSet.getModules()
.forEach(
module -> moduleMetadata.put(module,
softwareModuleManagement.findMetaDataBySoftwareModuleIdAndTargetVisible(
PageRequest.of(0, RepositoryConstants.MAX_META_DATA_COUNT), module.getId())
.getContent()));
return moduleMetadata;
}

}
Loading