support export import config by env #3947

lepdou · 2021-09-04T13:41:18Z

What's the purpose of this PR

Which issue(s) this PR fixes:

Brief changelog

Follow this checklist to help us incorporate your contribution quickly and easily:

Read the Contributing Guide before making this pull request.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit tests to verify the code.
Run mvn clean test to make sure this pull request doesn't break anything.
Update the CHANGES log.

codecov-commenter · 2021-09-04T13:45:47Z

Codecov Report

Merging #3947 (8cf9d55) into master (729e1fd) will increase coverage by 0.79%.
The diff coverage is 59.21%.

@@             Coverage Diff              @@
##             master    #3947      +/-   ##
============================================
+ Coverage     51.65%   52.44%   +0.79%     
- Complexity     2547     2611      +64     
============================================
  Files           484      484              
  Lines         14844    15203     +359     
  Branches       1536     1572      +36     
============================================
+ Hits           7667     7973     +306     
- Misses         6645     6675      +30     
- Partials        532      555      +23

Impacted Files	Coverage Δ
...apollo/adminservice/controller/ItemController.java	`7.36% <0.00%> (-1.17%)`	⬇️
...trip/framework/apollo/biz/service/ItemService.java	`12.94% <0.00%> (-1.93%)`	⬇️
...ip/framework/apollo/common/constants/GsonType.java	`0.00% <0.00%> (ø)`
...p/framework/apollo/portal/api/AdminServiceAPI.java	`12.36% <0.00%> (-0.07%)`	⬇️
...llo/portal/controller/ConfigsImportController.java	`14.28% <0.00%> (-19.05%)`	⬇️
...k/apollo/portal/controller/InstanceController.java	`35.71% <0.00%> (ø)`
...ork/apollo/portal/service/AppNamespaceService.java	`66.00% <0.00%> (-9.00%)`	⬇️
...ip/framework/apollo/portal/service/AppService.java	`15.11% <0.00%> (-2.70%)`	⬇️
...amework/apollo/portal/service/InstanceService.java	`42.85% <ø> (ø)`
...llo/portal/controller/ConfigsExportController.java	`18.51% <11.11%> (-3.23%)`	⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 729e1fd...8cf9d55. Read the comment docs.

apollo-portal/src/main/config/application-github.properties

Anilople · 2021-09-07T11:46:37Z

Is there an example of file xxx.zip exported?

Anilople

Great feature!

Will be used by financial company's Operations Engineer widely in the feature.

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ConfigsExportService.java

...rtal/src/main/java/com/ctrip/framework/apollo/portal/controller/ConfigsImportController.java

nobodyiam

I tried to export and import again, it looks the value isn't the same with special characters(highlighted below)

apollo-portal/src/main/config/application-github.properties

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ConfigsImportService.java

nobodyiam · 2021-09-08T11:59:52Z

Please also update the CHANGES.md

lepdou · 2021-09-09T13:22:44Z

@nobodyiam

the bug of imported item not exactly same wtih before had fixed , please verify again.

lepdou · 2021-09-09T14:47:09Z

I will implement export/import by namespace with UI at weekend.

JaredTan95

LGTM, just one lint.

CHANGES.md

nobodyiam · 2021-09-11T02:01:21Z

@nobodyiam

the bug of imported item not exactly same wtih before had fixed , please verify again.

I tried to export and import the sample data(access code: c3x4), but many errors occurred during import phase.

item not found for itemId 0

It looks like the itemid and namespace id of the exported data is 0, however, they are not 0 in the actual database.

2021-09-11 09:27:59.548 ERROR 91795 --- [onPool-worker-5] c.c.f.a.p.c.RetryableRestTemplate        : Http request failed, uri: apps/lxj-apollo/clusters/default/namespaces/application/items/0, method: PUT

org.springframework.web.client.HttpClientErrorException$NotFound: 404 : [{"exception":"com.ctrip.framework.apollo.common.exception.NotFoundException","message":"item not found for itemId 0","status":404,"timestamp":"2021-09-11T09:27:59.548"}]
	at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
	at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:818)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:710)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:601)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.doExecute(RetryableRestTemplate.java:293)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.execute(RetryableRestTemplate.java:138)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.put(RetryableRestTemplate.java:114)
	at com.ctrip.framework.apollo.portal.api.AdminServiceAPI$ItemAPI.updateItem(AdminServiceAPI.java:185)
	at com.ctrip.framework.apollo.portal.service.ItemService.updateItem(ItemService.java:142)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.lambda$importItems$6(ConfigsImportService.java:389)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importItems(ConfigsImportService.java:378)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importNamespace(ConfigsImportService.java:371)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importNamespaceFromText(ConfigsImportService.java:340)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.lambda$doImport$3(ConfigsImportService.java:230)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

item already exists

The itemid and namespace id of the exported data is also 0.

2021-09-11 09:27:59.537 ERROR 91795 --- [onPool-worker-0] c.c.f.a.p.service.ConfigsImportService   : import namespace error. namespace = NamespaceImportData{env=FAT, fileName='abc123+default+application.properties', content='[{"id":0,"namespaceId":0,"key":"jdbc.url","value":"jdbc:mysql://127.0.0.1/ofs","lineNum":1,"dataChangeCreatedBy":"apollo","dataChangeLastModifiedBy":"apollo","dataChangeCreatedByDisplayName":"apollo","dataChangeLastModifiedByDisplayName":"apollo","dataChangeCreatedTime":"Feb 19, 2019 2:32:47 PM","dataChangeLastModifiedTime":"Feb 19, 2019 2:32:47 PM"}]', ignoreConflictNamespace=false}

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : [{"exception":"com.ctrip.framework.apollo.common.exception.BadRequestException","message":"item already exists","status":400,"timestamp":"2021-09-11T09:27:59.537"}]
	at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
	at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:818)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:710)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:601)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.doExecute(RetryableRestTemplate.java:293)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.execute(RetryableRestTemplate.java:138)
	at com.ctrip.framework.apollo.portal.component.RetryableRestTemplate.post(RetryableRestTemplate.java:110)
	at com.ctrip.framework.apollo.portal.api.AdminServiceAPI$ItemAPI.createItem(AdminServiceAPI.java:191)
	at com.ctrip.framework.apollo.portal.service.ItemService.createItem(ItemService.java:136)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.lambda$importItems$6(ConfigsImportService.java:392)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importItems(ConfigsImportService.java:378)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importNamespace(ConfigsImportService.java:371)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.importNamespaceFromText(ConfigsImportService.java:340)
	at com.ctrip.framework.apollo.portal.service.ConfigsImportService.lambda$doImport$3(ConfigsImportService.java:230)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

lepdou · 2021-09-11T06:45:30Z

@nobodyiam
the bug is not set existed item id when updating item. I have fixed.

nobodyiam · 2021-09-11T12:01:23Z

@lepdou thanks, it works now.

zouyx

There are some conficts

...rtal/src/main/java/com/ctrip/framework/apollo/portal/controller/ConfigsImportController.java

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ItemService.java

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ConfigsImportService.java

nobodyiam · 2021-09-25T01:52:16Z

I will implement export/import by namespace with UI at weekend.

export/import by namespace is a nice feature, will we implement it via this pr?

lepdou · 2021-10-31T10:43:40Z

export/import by namespace is a nice feature, will we implement it via this pr?

Updated.

nobodyiam

LGTM

nobodyiam reviewed Sep 5, 2021

View reviewed changes

apollo-portal/src/main/config/application-github.properties Outdated Show resolved Hide resolved

nobodyiam requested review from Anilople and JaredTan95 September 5, 2021 08:27

JaredTan95 added area/portal apollo-portal feature Categorizes issue as related to a new feature. labels Sep 6, 2021

JaredTan95 added this to the 1.10.0 milestone Sep 6, 2021

Anilople reviewed Sep 7, 2021

View reviewed changes

nobodyiam requested changes Sep 8, 2021

View reviewed changes

apollo-portal/src/main/config/application-github.properties Outdated Show resolved Hide resolved

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ConfigsImportService.java Show resolved Hide resolved

JaredTan95 reviewed Sep 10, 2021

View reviewed changes

CHANGES.md Outdated Show resolved Hide resolved

lepdou force-pushed the dev/lepdou/1.10.0_export_config branch from b995911 to fd8854c Compare September 11, 2021 10:23

zouyx reviewed Sep 19, 2021

View reviewed changes

lepdou force-pushed the dev/lepdou/1.10.0_export_config branch from b9b76fd to f709e33 Compare September 22, 2021 11:08

nobodyiam reviewed Sep 25, 2021

View reviewed changes

apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/ConfigsImportService.java Outdated Show resolved Hide resolved

lepdou force-pushed the dev/lepdou/1.10.0_export_config branch 2 times, most recently from fba2de8 to ad86f8b Compare October 31, 2021 09:14

support export/import configs

7d57d1b

lepdou force-pushed the dev/lepdou/1.10.0_export_config branch from ad86f8b to 7d57d1b Compare October 31, 2021 10:02

Merge branch 'master' into dev/lepdou/1.10.0_export_config

8cf9d55

nobodyiam approved these changes Nov 6, 2021

View reviewed changes

nobodyiam merged commit 0e14333 into apolloconfig:master Nov 6, 2021

github-actions bot locked and limited conversation to collaborators Nov 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support export import config by env #3947

support export import config by env #3947

lepdou commented Sep 4, 2021

codecov-commenter commented Sep 4, 2021 •

edited

Loading

Anilople commented Sep 7, 2021

Anilople left a comment

nobodyiam left a comment

nobodyiam commented Sep 8, 2021

lepdou commented Sep 9, 2021

lepdou commented Sep 9, 2021

JaredTan95 left a comment

nobodyiam commented Sep 11, 2021

lepdou commented Sep 11, 2021

nobodyiam commented Sep 11, 2021

zouyx left a comment

nobodyiam commented Sep 25, 2021

lepdou commented Oct 31, 2021

nobodyiam left a comment

support export import config by env #3947

support export import config by env #3947

Conversation

lepdou commented Sep 4, 2021

What's the purpose of this PR

Which issue(s) this PR fixes:

Brief changelog

codecov-commenter commented Sep 4, 2021 • edited Loading

Codecov Report

Anilople commented Sep 7, 2021

Anilople left a comment

Choose a reason for hiding this comment

nobodyiam left a comment

Choose a reason for hiding this comment

nobodyiam commented Sep 8, 2021

lepdou commented Sep 9, 2021

lepdou commented Sep 9, 2021

JaredTan95 left a comment

Choose a reason for hiding this comment

nobodyiam commented Sep 11, 2021

lepdou commented Sep 11, 2021

nobodyiam commented Sep 11, 2021

zouyx left a comment

Choose a reason for hiding this comment

nobodyiam commented Sep 25, 2021

lepdou commented Oct 31, 2021

nobodyiam left a comment

Choose a reason for hiding this comment

codecov-commenter commented Sep 4, 2021 •

edited

Loading