Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test to show how automatic migration handles deprecated definitions #4655

Merged
merged 4 commits into from
Jul 12, 2021
Merged

test to show how automatic migration handles deprecated definitions #4655

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2e2e1f0
test to show definitions not present in latest seed would be deleted …
subodh1810 Jul 9, 2021
d5cfbd3
format
subodh1810 Jul 9, 2021
bd45464
Merge branch 'master' into old-definition-should-be-deleted
subodh1810 Jul 12, 2021
ceddd4f
add deprecated config being used scenario
subodh1810 Jul 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions airbyte-server/src/test/java/io/airbyte/server/migration/RunMigrationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,10 @@ public void testRunMigration() {
resourceToBeCleanedUp.add(configRoot.toFile());
final JobPersistence jobPersistence = getJobPersistence(stubAirbyteDB.getDatabase(), file, INITIAL_VERSION);
assertDatabaseVersion(jobPersistence, INITIAL_VERSION);
ConfigRepository configRepository = new ConfigRepository(FileSystemConfigPersistence.createWithValidation(configRoot));
Map<String, StandardSourceDefinition> sourceDefinitionsBeforeMigration = configRepository.listStandardSources().stream()
.collect(Collectors.toMap(c -> c.getSourceDefinitionId().toString(), c -> c));
assertTrue(sourceDefinitionsBeforeMigration.containsKey("d2147be5-fa36-4936-977e-f031affa5895"));

runMigration(jobPersistence, configRoot);

Expand Down Expand Up @@ -137,6 +141,8 @@ private void assertSourceDefinitions(ConfigRepository configRepository) throws J
.stream()
.collect(Collectors.toMap(c -> c.getSourceDefinitionId().toString(), c -> c));
assertTrue(sourceDefinitions.size() >= 59);
// the definition is not present in latest seeds so it should be deleted
assertFalse(sourceDefinitions.containsKey("d2147be5-fa36-4936-977e-f031affa5895"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a bit that also shows old definition with a usage will not be deleted?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following assertion makes sure that the source that is being used stays on the old version and doesnt get upgraded

final StandardSourceDefinition mysqlDefinition = sourceDefinitions.get("435bb9a5-7887-4809-aa58-28c27df0d7ad");
    assertEquals("0.2.0", mysqlDefinition.getDockerImageTag());
    assertEquals("MySQL", mysqlDefinition.getName());

But there is no assertion for the scenario where a connector existed in the past and is being used and should not get deleted

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add one, should not be too tough

final StandardSourceDefinition mysqlDefinition = sourceDefinitions.get("435bb9a5-7887-4809-aa58-28c27df0d7ad");
assertEquals("0.2.0", mysqlDefinition.getDockerImageTag());
assertEquals("MySQL", mysqlDefinition.getName());
Expand Down
7 changes: 7 additions & 0 deletions ...on/dummy_data/config/STANDARD_SOURCE_DEFINITION/d2147be5-fa36-4936-977e-f031affa5895.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"sourceDefinitionId": "d2147be5-fa36-4936-977e-f031affa5895",
"name": "Old Connector",
"dockerRepository": "airbyte/source-appstore-singer",
"dockerImageTag": "0.2.0",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-appstore-singer"
}