Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development: Extract build agent git service #8595

Merged
merged 32 commits into from
May 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
ce62db3
extract build agent git service
May 14, 2024
5494617
Merge branch 'develop' into chore/extract-build-agent-git-service
May 18, 2024
515da65
fix issue when deleting repo on Windows
BBesrour May 22, 2024
2ec1c5a
remove redundant code
BBesrour May 22, 2024
f4f9fe5
Merge branch 'refs/heads/develop' into chore/extract-build-agent-git-…
BBesrour May 22, 2024
964b06c
resolve merge conflicts
BBesrour May 22, 2024
5cbcb31
Adjust config files
BBesrour May 22, 2024
7a36ada
Adjust config files
BBesrour May 22, 2024
ccdc818
fix tests and style
BBesrour May 22, 2024
12a6884
Merge branch 'refs/heads/develop' into chore/extract-build-agent-git-…
BBesrour May 22, 2024
c6337cb
fix tests
BBesrour May 23, 2024
b4c27fa
fix tests
BBesrour May 23, 2024
6b8d711
fix tests
BBesrour May 23, 2024
7d022e2
fix tests
BBesrour May 23, 2024
b4d570c
fix tests
BBesrour May 23, 2024
1fba116
MUST REVERT - testserver
BBesrour May 23, 2024
3698915
revert last commit
BBesrour May 23, 2024
189605c
- fix multi-node setup
BBesrour May 23, 2024
0e92c16
improve code
BBesrour May 23, 2024
c7fcf1b
Merge branch 'develop' into chore/extract-build-agent-git-service
BBesrour May 23, 2024
efb6b97
change requests
BBesrour May 24, 2024
6adedba
Merge branch 'develop' into chore/extract-build-agent-git-service
BBesrour May 24, 2024
d2d68b0
change requests
BBesrour May 24, 2024
1cd67a8
Merge branch 'develop' into chore/extract-build-agent-git-service
May 25, 2024
8568ba4
implement serveral structural and code improvements
May 25, 2024
2689359
improve annotation
May 25, 2024
7f199b4
move classes and fix architecture test
May 25, 2024
9394900
fix wrong profile
May 25, 2024
398d597
update jacoco settings
May 25, 2024
4815aa3
re-add eureka client rest template config to build agent
May 25, 2024
94471e9
Merge branch 'develop' into chore/extract-build-agent-git-service
May 25, 2024
45d2ddc
improve error message when cloning does not work
May 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -186,13 +186,13 @@ jacocoTestCoverageVerification {
counter = "INSTRUCTION"
value = "COVEREDRATIO"
// TODO: in the future the following value should become higher than 0.92
minimum = 0.898
minimum = 0.897
}
limit {
counter = "CLASS"
value = "MISSEDCOUNT"
// TODO: in the future the following value should become less than 10
maximum = 29
maximum = 32
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions docker/artemis/config/cypress-local.env
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ SPRING_PROFILES_ACTIVE="artemis,scheduling,localvc,localci,buildagent,core,prod,

ARTEMIS_USERMANAGEMENT_USEEXTERNAL="false"
ARTEMIS_VERSIONCONTROL_URL='http://localhost:8080'
ARTEMIS_VERSIONCONTROL_USER='demo'
ARTEMIS_VERSIONCONTROL_PASSWORD='demo'
ARTEMIS_VERSIONCONTROL_USER="${bamboo_artemis_admin_username}"
ARTEMIS_VERSIONCONTROL_PASSWORD="${bamboo_artemis_admin_password}"
ARTEMIS_CONTINUOUSINTEGRATION_ARTEMISAUTHENTICATIONTOKENVALUE='demo'
ARTEMIS_CONTINUOUSINTEGRATION_DOCKERCONNECTIONURI='unix:///var/run/docker.sock'
ARTEMIS_GIT_NAME='artemis'
Expand Down
3 changes: 0 additions & 3 deletions docker/artemis/config/node2.env
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,3 @@ SPRING_PROFILES_ACTIVE='prod,localvc,localci,buildagent,core,docker'
EUREKA_INSTANCE_INSTANCEID='Artemis:2'
EUREKA_INSTANCE_HOSTNAME='artemis-app-node-2'
SPRING_HAZELCAST_INTERFACE='artemis-app-node-2'

ARTEMIS_VERSIONCONTROL_USER='artemis_admin'
ARTEMIS_VERSIONCONTROL_PASSWORD='artemis_admin'
3 changes: 0 additions & 3 deletions docker/artemis/config/node3.env
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,3 @@ SPRING_PROFILES_ACTIVE='prod,buildagent'
EUREKA_INSTANCE_INSTANCEID='Artemis:3'
EUREKA_INSTANCE_HOSTNAME='artemis-app-node-3'
SPRING_HAZELCAST_INTERFACE='artemis-app-node-3'

ARTEMIS_VERSIONCONTROL_USER='artemis_admin'
ARTEMIS_VERSIONCONTROL_PASSWORD='artemis_admin'
4 changes: 2 additions & 2 deletions docker/artemis/config/prod-multinode.env
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ JHIPSTER_REGISTRY_PASSWORD="admin"
JHIPSTER_SECURITY_AUTHENTICATION_JWT_BASE64SECRET="bXktc2VjcmV0LWtleS13aGljaC1zaG91bGQtYmUtY2hhbmdlZC1pbi1wcm9kdWN0aW9uLWFuZC1iZS1iYXNlNjQtZW5jb2RlZAo="

ARTEMIS_VERSIONCONTROL_URL='http://artemis-app-node-2:8080'
ARTEMIS_VERSIONCONTROL_USER='demo'
ARTEMIS_VERSIONCONTROL_PASSWORD='demo'
ARTEMIS_VERSIONCONTROL_USER='artemis_admin'
ARTEMIS_VERSIONCONTROL_PASSWORD='artemis_admin'
BBesrour marked this conversation as resolved.
Show resolved Hide resolved
ARTEMIS_CONTINUOUSINTEGRATION_ARTEMISAUTHENTICATIONTOKENVALUE='demo'
ARTEMIS_CONTINUOUSINTEGRATION_DOCKERCONNECTIONURI='unix:///var/run/docker.sock'
ARTEMIS_GIT_NAME='artemis'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
package de.tum.in.www1.artemis.config;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Conditional;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.authentication.AuthenticationServiceException;

@Configuration
public class ApplicationConfiguration {

// this prevents default authentication when the build agent is enabled without the core profile
@Profile({ "!" + PROFILE_CORE, PROFILE_BUILDAGENT })
/**
* Provides a no-op AuthenticationManager that throws an exception, effectively disabling authentication, when the 'buildagent' profile is active
* and the 'core' profile is not active.
*
* @return the no-op AuthenticationManager
*/
@Bean
@Conditional(BuildAgentWithoutCoreCondition.class)
public AuthenticationManager noopAuthenticationManager() {
return authentication -> {
throw new AuthenticationServiceException("Authentication is disabled");
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
package de.tum.in.www1.artemis.config;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import java.util.Arrays;
import java.util.Collection;

import org.springframework.boot.autoconfigure.condition.AllNestedConditions;
import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;
import org.springframework.context.annotation.Conditional;
import org.springframework.core.type.AnnotatedTypeMetadata;

/**
* Custom condition that checks for the presence of the 'buildagent' profile and the absence of the 'core' profile.
* This condition will be used to conditionally exclude certain configurations based on the active Spring profiles.
*/
public class BuildAgentWithoutCoreCondition extends AllNestedConditions {

/**
* Constructor for BuildAgentWithoutCoreCondition.
* Specifies that the conditions should be evaluated during the configuration parsing phase.
*/
public BuildAgentWithoutCoreCondition() {
super(ConfigurationPhase.PARSE_CONFIGURATION);
}

/**
* Inner class representing the condition that the 'buildagent' profile is active.
*/
@Conditional(BuildAgentProfileCondition.class)
@SuppressWarnings("unused")
static class OnBuildAgentProfile {
}

/**
* Inner class representing the condition that the 'core' profile is not active.
*/
@Conditional(NotCoreProfileCondition.class)
@SuppressWarnings("unused")
static class OnNotCoreProfile {
}

/**
* Condition implementation that checks if the 'buildagent' profile is active.
*/
static class BuildAgentProfileCondition implements Condition {

/**
* Evaluates whether the 'buildagent' profile is active.
*
* @param context the condition context
* @param metadata the metadata of the {@link Conditional} annotation
* @return true if 'buildagent' profile is active, false otherwise
*/
@Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
final Collection<String> activeProfiles = Arrays.asList(context.getEnvironment().getActiveProfiles());
return activeProfiles.contains(PROFILE_BUILDAGENT);
}
}

/**
* Condition implementation that checks if the 'core' profile is *not* active.
*/
static class NotCoreProfileCondition implements Condition {

/**
* Evaluates whether the 'core' profile is not active.
*
* @param context the condition context
* @param metadata the metadata of the {@link Conditional} annotation
* @return true if 'core' profile is not active, false otherwise
*/
@Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
final Collection<String> activeProfiles = Arrays.asList(context.getEnvironment().getActiveProfiles());
return !activeProfiles.contains(PROFILE_CORE);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ public HazelcastInstance hazelcastInstance(JHipsterProperties jHipsterProperties
config.getSerializationConfig().addSerializerConfig(createPathSerializerConfig());

if (registration == null) {
log.info("No discovery service is set up, Hazelcast cannot create a cluster.");
log.info("No discovery service is set up, Hazelcast cannot create a multi-node cluster.");
hazelcastBindOnlyOnInterface("127.0.0.1", config);
}
else {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package de.tum.in.www1.artemis.config;

import org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Conditional;
import org.springframework.context.annotation.Configuration;

/**
* Conditional configuration class that excludes {@link MetricsAutoConfiguration}
* when the application is running with the 'buildagent' profile active and the 'core' profile inactive.
* <p>
* This configuration ensures that the MetricsAutoConfiguration is not applied in environments where
* the application is being run by a build agent, which typically does not require metric collection,
* and where the 'core' profile, which might need metrics, is not active.
* </p>
*/
@Configuration
@Conditional(BuildAgentWithoutCoreCondition.class)
@EnableAutoConfiguration(exclude = MetricsAutoConfiguration.class)
public class ConditionalMetricsExclusionConfiguration {
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package de.tum.in.www1.artemis.config;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import org.springframework.boot.actuate.autoconfigure.endpoint.condition.ConditionalOnAvailableEndpoint;
Expand All @@ -23,7 +22,7 @@
*/
public class CustomMetricsExtensionConfiguration {

@Profile({ PROFILE_CORE, PROFILE_BUILDAGENT })
@Profile(PROFILE_CORE)
@Configuration
@ConditionalOnClass(Timed.class)
@AutoConfigureAfter(JHipsterMetricsEndpointConfiguration.class)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package de.tum.in.www1.artemis.config;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import org.springframework.context.annotation.Bean;
Expand All @@ -12,7 +11,7 @@
import de.tum.in.www1.artemis.aop.logging.LoggingAspect;
import tech.jhipster.config.JHipsterConstants;

@Profile({ PROFILE_CORE, PROFILE_BUILDAGENT })
@Profile(PROFILE_CORE)
@Configuration
@EnableAspectJAutoProxy
public class LoggingAspectConfiguration {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
import de.tum.in.www1.artemis.security.SecurityUtils;
import de.tum.in.www1.artemis.service.ProfileService;
import de.tum.in.www1.artemis.service.connectors.localci.SharedQueueManagementService;
import de.tum.in.www1.artemis.service.connectors.localci.dto.LocalCIBuildAgentInformation;
import de.tum.in.www1.artemis.service.connectors.localci.dto.BuildAgentInformation;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.MultiGauge;
Expand Down Expand Up @@ -301,7 +301,7 @@ private static int extractBuildAgents(Optional<SharedQueueManagementService> sha

private static int extractMaxConcurrentBuilds(Optional<SharedQueueManagementService> sharedQueueManagementService) {
return sharedQueueManagementService.map(queueManagementService -> queueManagementService.getBuildAgentInformation().stream()
.map(LocalCIBuildAgentInformation::maxNumberOfConcurrentBuildJobs).reduce(0, Integer::sum)).orElse(0);
.map(BuildAgentInformation::maxNumberOfConcurrentBuildJobs).reduce(0, Integer::sum)).orElse(0);
}

private void registerWebsocketMetrics() {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package de.tum.in.www1.artemis.config;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import org.apache.catalina.Context;
Expand All @@ -11,7 +10,7 @@
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

@Profile({ PROFILE_CORE, PROFILE_BUILDAGENT })
@Profile(PROFILE_CORE)
@Configuration
public class TomcatConfiguration {

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package de.tum.in.www1.artemis.config.audit;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;

import java.util.ArrayList;
Expand All @@ -18,7 +17,7 @@

import de.tum.in.www1.artemis.domain.PersistentAuditEvent;

@Profile({ PROFILE_CORE, PROFILE_BUILDAGENT })
@Profile(PROFILE_CORE)
@Component
public class AuditEventConverter {

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci;
package de.tum.in.www1.artemis.config.icl;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;

Expand Down Expand Up @@ -36,11 +36,11 @@
*/
@Configuration
@Profile(PROFILE_BUILDAGENT)
public class LocalCIConfiguration {
public class BuildAgentConfiguration {

private final ProgrammingLanguageConfiguration programmingLanguageConfiguration;

private static final Logger log = LoggerFactory.getLogger(LocalCIConfiguration.class);
private static final Logger log = LoggerFactory.getLogger(BuildAgentConfiguration.class);

@Value("${artemis.continuous-integration.docker-connection-uri}")
String dockerConnectionUri;
Expand All @@ -51,7 +51,7 @@ public class LocalCIConfiguration {
@Value("${artemis.continuous-integration.specify-concurrent-builds:false}")
boolean specifyConcurrentBuilds;

public LocalCIConfiguration(ProgrammingLanguageConfiguration programmingLanguageConfiguration) {
public BuildAgentConfiguration(ProgrammingLanguageConfiguration programmingLanguageConfiguration) {
this.programmingLanguageConfiguration = programmingLanguageConfiguration;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci;
package de.tum.in.www1.artemis.config.icl;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_LOCALVC;

Expand All @@ -10,7 +10,7 @@
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

import de.tum.in.www1.artemis.config.localvcci.service.ArtemisGitServletService;
import de.tum.in.www1.artemis.service.icl.ArtemisGitServletService;

/**
* Configuration of the JGit Servlet that handles fetch and push requests for local Version Control.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci.ssh;
package de.tum.in.www1.artemis.config.icl.ssh;

import java.security.PublicKey;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci.ssh;
package de.tum.in.www1.artemis.config.icl.ssh;

import java.io.IOException;
import java.io.OutputStream;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci.ssh;
package de.tum.in.www1.artemis.config.icl.ssh;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_LOCALVC;

Expand All @@ -18,9 +18,9 @@
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

import de.tum.in.www1.artemis.config.localvcci.ssh.service.GitPublickeyAuthenticatorService;
import de.tum.in.www1.artemis.config.localvcci.ssh.service.SshGitCommandFactoryService;
import de.tum.in.www1.artemis.config.localvcci.ssh.service.SshGitLocationResolverService;
import de.tum.in.www1.artemis.service.icl.GitPublickeyAuthenticatorService;
import de.tum.in.www1.artemis.service.icl.SshGitCommandFactoryService;
import de.tum.in.www1.artemis.service.icl.SshGitLocationResolverService;

@Profile(PROFILE_LOCALVC)
@Configuration
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci.ssh;
package de.tum.in.www1.artemis.config.icl.ssh;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_LOCALVC;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package de.tum.in.www1.artemis.config.localvcci.ssh;
package de.tum.in.www1.artemis.config.icl.ssh;

import java.io.IOException;
import java.nio.file.Path;
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/de/tum/in/www1/artemis/domain/BuildJob.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

import de.tum.in.www1.artemis.domain.enumeration.BuildStatus;
import de.tum.in.www1.artemis.domain.enumeration.RepositoryType;
import de.tum.in.www1.artemis.service.connectors.localci.dto.LocalCIBuildJobQueueItem;
import de.tum.in.www1.artemis.service.connectors.localci.dto.BuildJobQueueItem;

@Entity
@Table(name = "build_job")
Expand Down Expand Up @@ -80,7 +80,7 @@ public class BuildJob extends DomainObject {
public BuildJob() {
}

public BuildJob(LocalCIBuildJobQueueItem queueItem, BuildStatus buildStatus, Result result) {
public BuildJob(BuildJobQueueItem queueItem, BuildStatus buildStatus, Result result) {
this.buildJobId = queueItem.id();
this.name = queueItem.name();
this.exerciseId = queueItem.exerciseId();
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package de.tum.in.www1.artemis.service;

import static de.tum.in.www1.artemis.config.Constants.PROFILE_BUILDAGENT;
import static de.tum.in.www1.artemis.config.Constants.PROFILE_CORE;
import static java.nio.charset.StandardCharsets.UTF_8;

Expand Down Expand Up @@ -69,7 +68,7 @@
import de.tum.in.www1.artemis.web.rest.errors.BadRequestAlertException;
import de.tum.in.www1.artemis.web.rest.errors.InternalServerErrorException;

@Profile({ PROFILE_CORE, PROFILE_BUILDAGENT })
@Profile(PROFILE_CORE)
@Service
public class FileService implements DisposableBean {

Expand Down
Loading
Loading