Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(databasechange): implement the new interface and flow of multiple databases change #2275

Merged
merged 58 commits into from
May 20, 2024
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
69a8aa8
complete the controller and dao layers
zijiacj Mar 11, 2024
c2c47c7
complete the controller and dao layers
zijiacj Mar 11, 2024
ad863e7
Complete the code for the template section
zijiacj Mar 18, 2024
7cb866c
create multiple database change flowinstance
zijiacj Apr 9, 2024
9cf764d
Added View the details of multiple database change flowinstance
zijiacj Apr 12, 2024
5821aff
modify code format
zijiacj Apr 12, 2024
f53e892
Merge branch 'dev/4.3.x' into feat/zijia_multi_database_change_servic…
zijiacj Apr 14, 2024
7df1de2
Merge branch 'dev/4.3.x' into feat/zijia_multi_database_change_servic…
zijiacj Apr 15, 2024
230a17d
Change the version number to avoid duplication
zijiacj Apr 15, 2024
75fd52b
add sql check,task log and synchronization of the tasks' progress
zijiacj Apr 17, 2024
c9d1272
modify code format
zijiacj Apr 17, 2024
ff1d605
Merge remote-tracking branch 'origin/feat/zijia_multi_database_change…
zijiacj Apr 19, 2024
1d101dc
Delete obsolete files
zijiacj Apr 19, 2024
9227843
implement the servic of multiple databases change template
zijiacj Apr 22, 2024
e2595d7
implement the paging query and multiple databases sqlcheck
zijiacj Apr 22, 2024
0861b68
implement the paging query and multiple databases sqlcheck
zijiacj Apr 22, 2024
85a521e
Merge remote-tracking branch 'origin/dev/4.3.x' into feat/zijia_multi…
zijiacj Apr 23, 2024
6443186
Added the duplicate name verification interface and unit test of the …
zijiacj Apr 24, 2024
44d1b71
modify code format
zijiacj Apr 24, 2024
23227fd
builds: fix failed to install pnpm (#2248)
HSunboy Apr 18, 2024
133e2a4
Remove redundant code
zijiacj Apr 25, 2024
b02f5d4
Add name for multiple database change flow instance
zijiacj Apr 25, 2024
dab77f8
modify the code format
zijiacj Apr 25, 2024
9c61090
modify view the template list、batch access database、set the risk leve…
zijiacj Apr 28, 2024
d58ef27
Add environment objects to viewing multi-database flow's response a…
zijiacj Apr 29, 2024
38eed4c
Add a single pre-check node and flow description for multiple databases
zijiacj Apr 30, 2024
f1e14b3
add the function of viewing multi-database pre-check nodes
zijiacj May 6, 2024
a4ad4f7
Refine the execution of recorded responses
zijiacj May 6, 2024
d995faf
Add environment elements to multi-databases response results
zijiacj May 6, 2024
e877f41
Modify code format
zijiacj May 7, 2024
47ad5b0
Limit the number of databases and do not allow duplicate databases
zijiacj May 9, 2024
15ec6c4
Fixed error after sql check when the input is '#'
zijiacj May 9, 2024
c6825df
Solve the problem that multiple database flow cannot be initiated rep…
zijiacj May 10, 2024
d456ff8
Fixed NullPointerException when transferring sql files
zijiacj May 10, 2024
9f1fe7e
fix execution record display when failure occurs
zijiacj May 10, 2024
7efa93a
Merge remote-tracking branch 'origin/dev/4.3.x' into feat/zijia_multi…
zijiacj May 11, 2024
9164a56
modify code format
zijiacj May 11, 2024
42a8f97
enhancement(jpa): add json type (#2211)
ungreat Apr 22, 2024
669b238
Modify code format and optimize performance
zijiacj May 12, 2024
5bff76a
Merge remote-tracking branch 'origin/dev/4.3.x' into feat/zijia_multi…
zijiacj May 13, 2024
723da48
Finished ignoring error execution in automatic mode and fixed the iss…
zijiacj May 13, 2024
4e40a21
fix template create and update
zijiacj May 14, 2024
dfa37f9
Complete various strategy for creating multi-database workflow
zijiacj May 14, 2024
7ff931d
Reduce redundant code
zijiacj May 14, 2024
4bd3113
Modify the code to conform to odc specifications
zijiacj May 15, 2024
9399574
Remove redundant code
zijiacj May 15, 2024
4648913
Merge remote-tracking branch 'origin/dev/4.3.x' into feat/zijia_multi…
zijiacj May 15, 2024
82eb4aa
Added a scheduled template task to fix bugs
zijiacj May 16, 2024
1c6c30c
modify the code format
zijiacj May 17, 2024
c4104dc
response to cr comments
zijiacj May 17, 2024
1d38ffc
Optimize persistence and response objects
zijiacj May 20, 2024
b2c7f83
Remove redundant code
zijiacj May 20, 2024
8be8d2e
Serialize the databaseList
zijiacj May 20, 2024
471cf87
modify code format
zijiacj May 20, 2024
7b83339
delete the redundant code
zijiacj May 20, 2024
3e892d2
modify the code format
zijiacj May 20, 2024
01e0d35
response to cr
zijiacj May 20, 2024
3a1fcd5
response to cr
zijiacj May 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/*
* Copyright (c) 2023 OceanBase.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.oceanbase.odc.metadb.databasechange;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;

import com.oceanbase.odc.ServiceTestEnv;

/**
* @author: zijia.cj
* @date: 2024/4/24
*/
public class DatabaseChangeChangingOrderTemplateRepositoryTest extends ServiceTestEnv {

private static final Long PROJECT_ID = 1L;
private static final Long CURRENT_USER_ID = 1L;

private static final Long ORGANIZATION_ID = 1L;
private static final String TEMPLATE_NAME = "template";

@Autowired
private DatabaseChangeChangingOrderTemplateRepository templateRepository;

@Before
public void setUp() {
templateRepository.deleteAll();
}

@After
public void clear() {
templateRepository.deleteAll();
}


@Test
public void existsByNameAndProjectId_checkTemplateExist_succeed() {
create();
Boolean result =
templateRepository.existsByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID);
assertTrue(result);
}

@Test
public void findByNameAndProjectId_getTemplate_succeed() {
DatabaseChangeChangingOrderTemplateEntity databaseChangeChangingOrderTemplateEntity = create();
Optional<DatabaseChangeChangingOrderTemplateEntity> result =
templateRepository.findByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID);
assertNotNull(result.get());
assertEquals(databaseChangeChangingOrderTemplateEntity, result.get());
}

private DatabaseChangeChangingOrderTemplateEntity create() {
DatabaseChangeChangingOrderTemplateEntity databaseChangeChangingOrderTemplateEntity =
new DatabaseChangeChangingOrderTemplateEntity();
databaseChangeChangingOrderTemplateEntity.setName(TEMPLATE_NAME);
databaseChangeChangingOrderTemplateEntity.setProjectId(PROJECT_ID);
databaseChangeChangingOrderTemplateEntity.setOrganizationId(ORGANIZATION_ID);
databaseChangeChangingOrderTemplateEntity.setCreatorId(CURRENT_USER_ID);
List<List<Long>> orders = new ArrayList<>();
orders.add(Arrays.asList(1L, 2L));
orders.add(Arrays.asList(3L, 4L));
databaseChangeChangingOrderTemplateEntity.setDatabaseSequences(orders);
databaseChangeChangingOrderTemplateEntity.setEnabled(true);
return templateRepository.save(
databaseChangeChangingOrderTemplateEntity);
}
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,238 @@
/*
* Copyright (c) 2023 OceanBase.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.oceanbase.odc.service.databasechange;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.Mockito.when;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;

import com.oceanbase.odc.ServiceTestEnv;
import com.oceanbase.odc.core.shared.exception.BadArgumentException;
import com.oceanbase.odc.core.shared.exception.BadRequestException;
import com.oceanbase.odc.core.shared.exception.NotFoundException;
import com.oceanbase.odc.metadb.collaboration.ProjectRepository;
import com.oceanbase.odc.metadb.connection.DatabaseEntity;
import com.oceanbase.odc.metadb.connection.DatabaseRepository;
import com.oceanbase.odc.metadb.databasechange.DatabaseChangeChangingOrderTemplateEntity;
import com.oceanbase.odc.metadb.databasechange.DatabaseChangeChangingOrderTemplateRepository;
import com.oceanbase.odc.service.databasechange.model.CreateDatabaseChangeChangingOrderReq;
import com.oceanbase.odc.service.databasechange.model.QueryDatabaseChangeChangingOrderParams;
import com.oceanbase.odc.service.databasechange.model.QueryDatabaseChangeChangingOrderResp;
import com.oceanbase.odc.service.databasechange.model.UpdateDatabaseChangeChangingOrderReq;
import com.oceanbase.odc.service.iam.ProjectPermissionValidator;
import com.oceanbase.odc.service.iam.auth.AuthenticationFacade;

/**
* @author: zijia.cj
* @date: 2024/4/23
*/
public class DatabaseChangeChangingOrderTemplateServiceTest extends ServiceTestEnv {

private static final Long PROJECT_ID = 1L;
private static final Long CURRENT_USER_ID = 1L;

private static final Long ORGANIZATION_ID = 1L;
private static final String TEMPLATE_NAME = "template";
private static final String TEMPLATE_RENAME = "template_rename";

@Autowired
private DatabaseChangeChangingOrderTemplateService templateService;
@Autowired
private DatabaseChangeChangingOrderTemplateRepository templateRepository;
@MockBean
private AuthenticationFacade authenticationFacade;
@MockBean
private DatabaseRepository databaseRepository;
@MockBean
private ProjectRepository projectRepository;
@MockBean
private ProjectPermissionValidator projectPermissionValidator;

@Before
public void setUp() {
templateRepository.deleteAll();
when(authenticationFacade.currentUserId()).thenReturn(CURRENT_USER_ID);
when(authenticationFacade.currentOrganizationId()).thenReturn(ORGANIZATION_ID);
when(projectRepository.existsById(any())).thenReturn(true);
when(projectPermissionValidator.hasProjectRole(anyLong(), any())).thenReturn(true);
}

@After
public void clear() {
templateRepository.deleteAll();
}

@Test
public void createDatabaseChangingOrderTemplate_saveEntity_succeed() {
CreateDatabaseChangeChangingOrderReq req = new CreateDatabaseChangeChangingOrderReq();
req.setProjectId(PROJECT_ID);
req.setName(TEMPLATE_NAME);
List<List<Long>> orders = new ArrayList<>();
orders.add(Arrays.asList(1L, 2L));
orders.add(Arrays.asList(3L, 4L));
req.setOrders(orders);
DatabaseChangeChangingOrderTemplateEntity templateEntity = templateService.create(
req);
int size = templateRepository.findAll().size();
Assert.assertEquals(TEMPLATE_NAME, templateEntity.getName());
Assert.assertEquals(PROJECT_ID, templateEntity.getProjectId());
Assert.assertEquals(1, size);
}

@Test(expected = BadRequestException.class)
public void createDatabaseChangingOrderTemplate_tempaltNameIsDuplicate_throwIllegalArgumentException() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
createDatabaseChangingOrderTemplate_saveEntity_succeed();
}


@Test(expected = NotFoundException.class)
public void createDatabaseChangingOrderTemplate_projectIsNotExist_throwIllegalArgumentException() {
CreateDatabaseChangeChangingOrderReq req = new CreateDatabaseChangeChangingOrderReq();
req.setProjectId(PROJECT_ID);
req.setName(TEMPLATE_NAME);
List<List<Long>> orders = new ArrayList<>();
List<Long> list = Arrays.asList(1L, 2L);
orders.add(list);
req.setOrders(orders);
when(projectRepository.existsById(any())).thenReturn(false);
templateService.create(req);
}

@Test(expected = BadArgumentException.class)
public void createDatabaseChangingOrderTemplate_databaseNotBelongToProject_throwIllegalArgumentException() {
CreateDatabaseChangeChangingOrderReq req = new CreateDatabaseChangeChangingOrderReq();
req.setProjectId(PROJECT_ID);
req.setName(TEMPLATE_NAME);
List<List<Long>> orders = new ArrayList<>();
List<Long> list = Arrays.asList(1L, 2L);
orders.add(list);
req.setOrders(orders);
List<DatabaseEntity> databases = new ArrayList<>();
DatabaseEntity database = new DatabaseEntity();
database.setProjectId(2L);
databases.add(database);
when(projectRepository.existsById(any())).thenReturn(true);
when(databaseRepository.findByIdIn(any())).thenReturn(databases);
templateService.create(req);
}

@Test
public void modifyDatabaseChangingOrderTemplate_modifyTemplate_succeed() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
DatabaseChangeChangingOrderTemplateEntity byNameAndProjectId =
templateRepository.findByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID).get();
UpdateDatabaseChangeChangingOrderReq req = new UpdateDatabaseChangeChangingOrderReq();
req.setProjectId(PROJECT_ID);
req.setName(TEMPLATE_RENAME);
List<List<Long>> orders = new ArrayList<>();
orders.add(Arrays.asList(1L, 2L));
orders.add(Arrays.asList(3L, 4L));
DatabaseChangeChangingOrderTemplateEntity update = templateService
.update(byNameAndProjectId.getId(), req);
assertEquals(TEMPLATE_RENAME, update.getName());
Optional<DatabaseChangeChangingOrderTemplateEntity> byId =
templateRepository.findById(byNameAndProjectId.getId());
assertEquals(TEMPLATE_RENAME, byId.get().getName());

}

@Test(expected = NotFoundException.class)
public void modifyDatabaseChangingOrderTemplate_notFoundTemplate_throwIllegalArgumentException() {
UpdateDatabaseChangeChangingOrderReq req = new UpdateDatabaseChangeChangingOrderReq();
req.setProjectId(PROJECT_ID);
req.setName(TEMPLATE_RENAME);
List<List<Long>> orders = new ArrayList<>();
List<Long> list = Arrays.asList(1L, 2L);
orders.add(list);
when(authenticationFacade.currentUserId()).thenReturn(1L);
when(authenticationFacade.currentOrganizationId()).thenReturn(1L);
templateService.update(1L, req);
}

@Test
public void modifyDatabaseChangingOrderTemplate_projectNotExists_throwNotFoundException() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
DatabaseChangeChangingOrderTemplateEntity byNameAndProjectId =
templateRepository.findByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID).get();
UpdateDatabaseChangeChangingOrderReq req = new UpdateDatabaseChangeChangingOrderReq();
req.setName(TEMPLATE_RENAME);
req.setProjectId(2L);
when(projectRepository.existsById(2L)).thenReturn(false);
assertThrows(NotFoundException.class, () -> {
templateService.update(byNameAndProjectId.getId(),
req);
});
}

@Test
public void queryDatabaseChangingOrderTemplateById_findExistingTemplate_succeed() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
DatabaseChangeChangingOrderTemplateEntity byNameAndProjectId =
templateRepository.findByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID).get();
when(databaseRepository.findByIdIn(anyList())).thenReturn(Arrays.asList(new DatabaseEntity()));
QueryDatabaseChangeChangingOrderResp queryDatabaseChangeChangingOrderResp =
templateService.detail(
byNameAndProjectId.getId());
assertEquals(PROJECT_ID, queryDatabaseChangeChangingOrderResp.getProjectId());
assertEquals(TEMPLATE_NAME, queryDatabaseChangeChangingOrderResp.getName());
assertEquals(PROJECT_ID, queryDatabaseChangeChangingOrderResp.getProjectId());

}

@Test
public void listDatabaseChangingOrderTemplates_useQueryCondition_succeed() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
Pageable pageable = Pageable.unpaged();
QueryDatabaseChangeChangingOrderParams params = QueryDatabaseChangeChangingOrderParams.builder()
.projectId(PROJECT_ID).creatorId(CURRENT_USER_ID).name(TEMPLATE_NAME).build();
Page<DatabaseChangeChangingOrderTemplateEntity> result =
templateService.listTemplates(pageable, params);
Assert.assertNotNull(result);
Assert.assertEquals(1, result.getContent().size());
}


@Test
public void deleteDatabaseChangingOrderTemplateById_deleteExistingTemplate_succeed() {
createDatabaseChangingOrderTemplate_saveEntity_succeed();
DatabaseChangeChangingOrderTemplateEntity databaseChangeChangingOrderTemplateEntity =
templateRepository.findByNameAndProjectId(TEMPLATE_NAME, PROJECT_ID).get();
DatabaseChangeChangingOrderTemplateEntity delete = templateService.delete(
databaseChangeChangingOrderTemplateEntity.getId());
int size = templateRepository.findAll().size();
assertEquals(0, size);
}
}


Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ public enum ResourceType implements Translatable {
ODC_NOTIFICATION_POLICY,
ODC_NOTIFICATION_MESSAGE,

ODC_DATABASE_CHANGE_ORDER_TEMPLATE,


/**
* OB Resources, with 'OB_' prefix
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@
* @date 2021/3/15
*/
public enum TaskType implements Translatable {
/**
* Multiple database change
*/
MULTIPLE_ASYNC,
/**
* Database change
*/
Expand Down Expand Up @@ -107,7 +111,8 @@ public String getLocalizedMessage() {
}

public boolean needsPreCheck() {
return this == ASYNC || this == ONLINE_SCHEMA_CHANGE || this == ALTER_SCHEDULE || this == EXPORT_RESULT_SET;
return this == ASYNC || this == ONLINE_SCHEMA_CHANGE || this == ALTER_SCHEDULE || this == EXPORT_RESULT_SET
|| this == MULTIPLE_ASYNC;
}

public boolean needForExecutionStrategy() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ com.oceanbase.odc.ResourceType.ODC_NOTIFICATION_CHANNEL=Notification Channel
com.oceanbase.odc.ResourceType.ODC_NOTIFICATION_POLICY=Notification Policy
com.oceanbase.odc.ResourceType.ODC_NOTIFICATION_MESSAGE=Notification Message
com.oceanbase.odc.ResourceType.ODC_STRUCTURE_COMPARISON_TASK=Structure Comparison Task
com.oceanbase.odc.ResourceType.ODC_DATABASE_CHANGE_ORDER_TEMPLATE=Database change order template

#
# Batch Import
Expand Down Expand Up @@ -308,6 +309,7 @@ com.oceanbase.odc.AuditEventType.RESOURCE_GROUP_MANAGEMENT=Resource group manage
com.oceanbase.odc.AuditEventType.MEMBER_MANAGEMENT=Member management
com.oceanbase.odc.AuditEventType.AUDIT_EVENT=Audit event
com.oceanbase.odc.AuditEventType.FLOW_CONFIG=Flow config
com.oceanbase.odc.AuditEventType.MULTIPLE_ASYNC=Multiple Database Change
com.oceanbase.odc.AuditEventType.ASYNC=Database Change
com.oceanbase.odc.AuditEventType.MOCKDATA=Mock data
com.oceanbase.odc.AuditEventType.IMPORT=Import
Expand Down Expand Up @@ -793,6 +795,7 @@ com.oceanbase.odc.OBInstanceType.ORACLE_TENANT=Oracle Tenant
#
# TaskType
#
com.oceanbase.odc.TaskType.MULTIPLE_ASYNC=Multiple Database Change
com.oceanbase.odc.TaskType.ASYNC=Database Change
com.oceanbase.odc.TaskType.IMPORT=Import
com.oceanbase.odc.TaskType.EXPORT=Export
Expand Down
Loading