feat: Add apollo audit log common solution backend #4985

BlackBear2003 · 2023-09-25T03:45:35Z

What's the purpose of this PR

add audit log module, PR for code review and further discussion

Which issue(s) this PR fixes:

Fixes #

#3505

Brief changelog

Add Apollo-Audit module which contains 4 modules of annotation, api, impl, springbootstarter

Add basic front-end pages

Follow this checklist to help us incorporate your contribution quickly and easily:

Read the Contributing Guide before making this pull request.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit tests to verify the code.
Run mvn clean test to make sure this pull request doesn't break anything.
Update the CHANGES log.

@bean

add ApolloAuditLogApi for manually logging needs. add ApolloAuditLogDataInfluenceTableId (maybe multi-primary-key in the future). add ApolloAuditController by @bean registered

- add @DomainEvents to BaseEntity.java - change the way that data influences append - add annotations and codes to make app audited - change some classes' name - make api to query and record api - add event and listener to catch data influences

- try to reduce the affection of audit module on other modules

# Conflicts: # apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/AppService.java

…ServiceTest.class

codecov · 2023-09-25T03:49:40Z

Codecov Report

Merging #4985 (c670758) into master (a632cc1) will increase coverage by 0.41%.
The diff coverage is 56.00%.

❗ Current head c670758 differs from pull request most recent head f4de3b3. Consider uploading reports for the commit f4de3b3 to get more accurate results

@@             Coverage Diff              @@
##             master    #4985      +/-   ##
============================================
+ Coverage     48.93%   49.34%   +0.41%     
- Complexity     1782     1894     +112     
============================================
  Files           348      372      +24     
  Lines         10991    11538     +547     
  Branches       1095     1123      +28     
============================================
+ Hits           5378     5693     +315     
- Misses         5284     5507     +223     
- Partials        329      338       +9

Files	Coverage Δ
...lo/audit/component/ApolloAuditHttpInterceptor.java	`100.00% <100.00%> (ø)`
...ctrip/framework/apollo/biz/service/AppService.java	`48.71% <ø> (ø)`
.../com/ctrip/framework/apollo/common/entity/App.java	`0.00% <ø> (ø)`
...p/framework/apollo/portal/api/AdminServiceAPI.java	`13.19% <ø> (ø)`
...k/apollo/portal/component/RestTemplateFactory.java	`100.00% <100.00%> (ø)`
...mework/apollo/portal/controller/AppController.java	`29.33% <ø> (ø)`
...ork/apollo/portal/service/AppNamespaceService.java	`66.66% <ø> (ø)`
...ip/framework/apollo/portal/service/AppService.java	`49.46% <100.00%> (+9.90%)`	⬆️
.../apollo/audit/context/ApolloAuditTraceContext.java	`93.75% <93.75%> (ø)`
...p/framework/apollo/common/entity/AppNamespace.java	`0.00% <0.00%> (ø)`
... and 23 more

... and 1 file with indirect coverage changes

apollo-portal/src/main/resources/static/views/common/nav.html

@@ -65,6 +65,7 @@
                        <li><a href="{{ '/delete_app_cluster_namespace.html' | prefixPath }}" target="_blank">{{'Common.Nav.DeleteApp-Cluster-Namespace' | translate }}</a></li>
                        <li><a href="{{ '/system_info.html' | prefixPath }}" target="_blank">{{'Common.Nav.SystemInfo' | translate }}</a></li>
                        <li><a href="{{ '/config_export.html' | prefixPath }}" target="_blank">{{'Common.Nav.ConfigExport' | translate }}</a></li>
+                        <li><a href="{{ '/audit_log_menu.html' | prefixPath }}" target="_blank">{{'ApolloAuditLog.Title' | translate }}</a></li>


Anilople · 2023-10-18T13:26:11Z

When user disable the audit log, could we prompt user?

Audit Log disable already, you can xxxx to enable it, more detail please see the document

And hide the page too.

BlackBear2003 · 2023-10-18T17:56:05Z

After writing a series of unit tests about the context package, I summarized my trace-context model.

It is implemented based on the RequestContext provided by spring boot. In the same request, a Tracer instance is maintained in the context, which can be shared by all threads in this request.
This is in line with my original design intention. A request is regarded as a trace (or part of the trace). Tracer maintains only one active Scope and corresponding Span at the same time. This is also true in a trace. Should be the only one.
Tracer is request-independent. Different requests correspond to different Tracers, which represent different links.
Since the concurrency of Apollo is not high, after using JMeter to conduct a 50-concurrency test, the current audit logic has no impact. Therefore, I am not sure whether I need to maintain a critical section to ensure that only one thread can enter the critical section to operate Tracer at the same time.
Next, do I need to continue to complete the implementation of this feature in this PR (use the code of my audit module in other application modules)? Or do anything else? I'm looking forward to your guidance!

Could you try more concurrency in JMeter, for example 100, 200? And append the result in PR

The result is very charming!!! How could I post my jmeter-output-html on here?

BlackBear2003 · 2023-10-19T09:44:44Z

Audit Log disable already, you can xxxx to enable it, more detail please see the document

Added, but it may not be very beautiful

- add API for get isEnabled of audit log feature - fix sql problem and add it to delta - remove 2 classes

BlackBear2003 · 2023-10-19T14:54:04Z

About JMeter 200 concurrency test

Besides, the data increased in database are correct in counts and logic. It could be queried well in my audit-log-menu page.

It is my first time using JMeter, am I providing the necessary info?

BlackBear2003 · 2023-10-19T17:22:31Z

run https://www.apolloconfig.com/#/zh/development/apollo-release-guide?id=_422-%e6%89%93%e5%8c%85 to build project,

try to deploy it.

watch audit log work or not

change the .properties file, apollo.audit.log.enabled = false, watch the audit log close or not

finished deploy!

at http://luke.host:8070/

test account is (username: apollo/ password: admin)

BlackBear2003 · 2023-10-20T05:43:06Z

when I add "apollo.audit.log.enabled=false" to application-github.properties file and restart the portal, the feature is successfully turned off!

Anilople · 2023-10-24T13:04:58Z

...dit-annotation/src/main/java/com/ctrip/framework/apollo/audit/annotation/ApolloAuditLog.java

+
+@Target(ElementType.METHOD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface ApolloAuditLog {


need javadoc in all annotation and api

Anilople · 2023-10-24T13:07:41Z

...it-impl/src/main/java/com/ctrip/framework/apollo/audit/controller/ApolloAuditController.java

+
+  @GetMapping("/enabled")
+  public String isEnabled() {
+    return "{\"enabled\": "+ properties.isEnabled() +"}";


return ApolloAuditProperties directly?

Suggested change

return "{\"enabled\": "+ properties.isEnabled() +"}";

return properties;

Anilople · 2023-10-24T13:10:48Z

...impl/src/main/java/com/ctrip/framework/apollo/audit/repository/ApolloAuditLogRepository.java

+  List<ApolloAuditLog> findByOpName(String opName, Pageable page);
+
+  @Query("SELECT log FROM ApolloAuditLog log WHERE log.opName = :opName AND (log.dataChangeCreatedTime >= :startDate) AND (log.dataChangeCreatedTime <= :endDate)")
+  List<ApolloAuditLog> findByOpNameAndTime(@Param("opName") String opName,


use jpa's method name instead of write sql by self.

reference https://stackoverflow.com/questions/39784344/check-date-between-two-other-dates-spring-data-jpa

try use findByOpNameAndCreatedTimeGreaterThanEqualAndCreatedTimeGreaterLessThanEqual or another method name.

nobodyiam · 2023-10-25T00:29:25Z

Fantastic work, you've really put in the effort! Kudos to you! 🌟
A suggestion to make it even more awesome: consider enriching the README.md in the apollo-audit module with some instructions or sample code. This would be a huge help for other developers eager to make the most out of this audit function. Keep shining! ✨

- add javadoc - Update ApolloAuditController to return ApolloAuditProperties directly

Anilople · 2023-10-25T12:09:08Z

apollo-adminservice/src/main/scripts/startup.sh

@@ -16,7 +16,7 @@
 #
 SERVICE_NAME=apollo-adminservice
 ## Adjust log dir if necessary
-LOG_DIR=/opt/logs
+LOG_DIR=/home/luke/apollo/logs


cannot change here

oh sorry! I pushed this accidentally, I will change it back quickly.

Anilople · 2023-10-25T12:12:48Z

...dit-annotation/src/main/java/com/ctrip/framework/apollo/audit/annotation/ApolloAuditLog.java

+
+/**
+ * This annotation is mainly used for operations(mainly reflected through methods) that need to be
+ * audited.


Mark which method should be audit, add to controller or service's method

Anilople · 2023-10-25T12:12:57Z

...n/src/main/java/com/ctrip/framework/apollo/audit/annotation/ApolloAuditLogDataInfluence.java

+ * Example usage:
+ * <pre>
+ * {@code
+ * public App batchDeleteByAppId(


add ApolloAuditLog on the method too

Anilople · 2023-10-25T12:14:43Z

...n/src/main/java/com/ctrip/framework/apollo/audit/annotation/ApolloAuditLogDataInfluence.java

+import java.lang.annotation.Target;
+
+/**
+ * It is mainly used to mark the method's parameters that need to be audited.


Conbine with {@link ApolloAuditLog}, mark which method's parameter is audit log's data change.

- add README.md

BlackBear2003 · 2023-10-25T13:13:12Z

Fantastic work, you've really put in the effort! Kudos to you! 🌟 A suggestion to make it even more awesome: consider enriching the README.md in the apollo-audit module with some instructions or sample code. This would be a huge help for other developers eager to make the most out of this audit function. Keep shining! ✨

Thanks! I have added the README.md ~

If it weren't for my mentor who took great care of me, I wouldn't be able to complete this task. 🥹

Anilople · 2023-10-25T13:22:18Z

apollo-audit/README.md

@@ -0,0 +1,114 @@
+# Features: Apollo-Audit-Log
+
+This module mainly provides audit log functions for other Apollo modules.


Suggested change

This module mainly provides audit log functions for other Apollo modules.

This module provides audit log functions for other Apollo modules.

Only apolloconfig's developer need to read it,

apolloconfig's user doesn't need.

Anilople · 2023-10-25T13:22:31Z

apollo-audit/README.md

+
+This module mainly provides audit log functions for other Apollo modules.
+
+## How to switch 


Suggested change

## How to switch

## How to enable/disable

Anilople · 2023-10-25T13:23:45Z

apollo-audit/README.md

+apollo.audit.log.enabled = true
+```
+
+## How to use


Suggested change

## How to use

## How to generate audit log

Anilople · 2023-10-25T13:26:08Z

apollo-audit/README.md

+
+```java
+public App create() {
+  Autocloseable auditScope = api.appendAuditLog(type, name);


try with resources?

Anilople · 2023-10-25T13:26:09Z

apollo-audit/README.md

+
+```java
+public App create() {
+  Autocloseable auditScope = api.appendAuditLog(type, name);


try with resources?

Anilople · 2023-10-25T13:27:22Z

apollo-audit/README.md

+
+### Append an AuditLog
+
+We can do this by using annotations:


You may use 5W1H to describe it.

Anilople · 2023-10-25T13:28:00Z

apollo-audit/README.md

+
+### Append DataInfluence
+
+This function can also be implemented automatically and manually.


Sometimes there is not DataInfluence, we should tell developer about it.

Anilople · 2023-10-25T13:28:37Z

apollo-audit/README.md

+
+This function can also be implemented automatically and manually.
+
+#### Automatically


Suggested change

#### Automatically

#### Mark which data change

Anilople · 2023-10-25T13:29:25Z

apollo-audit/README.md

+ * Append DataInfluences by a list of entities needs to be audited, and their
+ * audit-bean-definition.
+ */
+ApolloAuditLogApi.appendDataInfluences(List<Object> entities, Class<?> beanDefinition);


Recomand use annotation instead of api, so we can left the api to the bottom of README.md?

Anilople · 2023-10-25T13:30:53Z

apollo-audit/README.md

+
+And this will call appendDataInfluences automatically by the listener.
+
+#### Manually


Maybe should describe how to verify the audit log work too...

For example, which table's data change, or how to check it by UI

Anilople

LGTM

BlackBear2003 and others added 27 commits July 4, 2023 00:42

7.3

4911b87

Merge branch 'apolloconfig:master' into add-audit-logs

b7b7b61

Merge branch 'apolloconfig:master' into add-audit-logs

687b73b

plc success except jpa part

d1b4574

successfully run by EntityScan annotation(not a good way)

57eb3d6

successfully run by add packages to bean registrar

83e7021

'app create' audit log and query poc success

d291b7d

App create/update/delete poc success.

bf4105d

add License

4031cc2

add License

85a2403

try to pass tests

b556305

try to pass tests

668a34e

fix ${revision} to ${project.version}

ff6fb0e

add no-op implement and no-op autoconfiguration.

ac42630

add ApolloAuditLogApi for manually logging needs. add ApolloAuditLogDataInfluenceTableId (maybe multi-primary-key in the future). add ApolloAuditController by @bean registered

Merge branch 'apolloconfig:master' into add-audit-logs

154a17b

rebase and add license

7608549

add audit controller

721cd4f

- change controller

50a1ee5

- add @DomainEvents to BaseEntity.java - change the way that data influences append - add annotations and codes to make app audited - change some classes' name - make api to query and record api - add event and listener to catch data influences

- make ApolloAuditHttpInterceptor uses Api while not tracer

a6e7b3c

- try to reduce the affection of audit module on other modules

- retry workflows

ca7469e

- change for code review

21b061f

- basically finish front-end part.

1da08c3

- fix for hound

7be4d34

- update

c43217e

- update

bae4aad

Merge remote-tracking branch 'origin/add-audit-logs' into add-audit-logs

39491ef

# Conflicts: # apollo-portal/src/main/java/com/ctrip/framework/apollo/portal/service/AppService.java

- add ApolloAuditLogApiNoOpImpl.class to @ContextConfiguration at App…

c345830

…ServiceTest.class

- add license and fix for hound

87ef435

github-advanced-security bot found potential problems Sep 25, 2023

View reviewed changes

BlackBear2003 added 2 commits October 19, 2023 17:57

- add prompt sheet for disabling audit log feature

add5f0d

- add API for get isEnabled of audit log feature - fix sql problem and add it to delta - remove 2 classes

- fix test bugs

a154f53

BlackBear2003 and others added 7 commits October 21, 2023 00:55

- change the way of getting op-type for data influences

62cbdc9

- enhance front-end design

b7a2c25

- enhance front-end design

08f6fb9

Merge branch 'master' into add-audit-logs

a6ff3a6

- add AuditLog Search dropdown

37a8148

- add any-match data influences for batch-delete operation auditing

ee5c271

- fix bugs

1e04137

Anilople requested changes Oct 24, 2023

View reviewed changes

BlackBear2003 added 3 commits October 25, 2023 15:53

- fix front-end bugs

93b2907

- add javadoc - Update ApolloAuditController to return ApolloAuditProperties directly

- fix test failure

36f0d75

- fix test failure

37c1458

Anilople requested changes Oct 25, 2023

View reviewed changes

- update javadoc

5430d0b

- add README.md

Anilople requested changes Oct 25, 2023

View reviewed changes

BlackBear2003 added 2 commits October 25, 2023 22:23

- update README

c670758

- update README

f4de3b3

Anilople approved these changes Oct 25, 2023

View reviewed changes

Anilople merged commit bc55ba6 into apolloconfig:master Oct 28, 2023
10 checks passed

github-actions bot locked and limited conversation to collaborators Oct 28, 2023

nobodyiam added this to the 2.2.0 milestone Nov 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add apollo audit log common solution backend #4985

feat: Add apollo audit log common solution backend #4985

BlackBear2003 commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

Anilople commented Oct 18, 2023

BlackBear2003 commented Oct 18, 2023

After writing a series of unit tests about the context package, I summarized my trace-context model.

BlackBear2003 commented Oct 19, 2023

BlackBear2003 commented Oct 19, 2023 •

edited

Loading

BlackBear2003 commented Oct 19, 2023 •

edited

Loading

BlackBear2003 commented Oct 20, 2023

Anilople Oct 24, 2023

Anilople Oct 24, 2023

Anilople Oct 24, 2023

nobodyiam commented Oct 25, 2023

Anilople Oct 25, 2023

BlackBear2003 Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

BlackBear2003 commented Oct 25, 2023 •

edited

Loading

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople Oct 25, 2023

Anilople left a comment

	return "{\"enabled\": "+ properties.isEnabled() +"}";
	return properties;

		@@ -0,0 +1,114 @@
		# Features: Apollo-Audit-Log

		This module mainly provides audit log functions for other Apollo modules.

-This module mainly provides audit log functions for other Apollo modules.
+This module provides audit log functions for other Apollo modules.
+Only apolloconfig's developer need to read it,
+apolloconfig's user doesn't need.


		This module mainly provides audit log functions for other Apollo modules.

		## How to switch


		### Append DataInfluence

		This function can also be implemented automatically and manually.


		This function can also be implemented automatically and manually.

		#### Automatically


		And this will call appendDataInfluences automatically by the listener.

		#### Manually

feat: Add apollo audit log common solution backend #4985

feat: Add apollo audit log common solution backend #4985

Conversation

BlackBear2003 commented Sep 25, 2023 • edited Loading

What's the purpose of this PR

Which issue(s) this PR fixes:

Brief changelog

codecov bot commented Sep 25, 2023 • edited Loading

Codecov Report

Anilople commented Oct 18, 2023

BlackBear2003 commented Oct 18, 2023

After writing a series of unit tests about the context package, I summarized my trace-context model.

BlackBear2003 commented Oct 19, 2023

BlackBear2003 commented Oct 19, 2023 • edited Loading

About JMeter 200 concurrency test

BlackBear2003 commented Oct 19, 2023 • edited Loading

BlackBear2003 commented Oct 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nobodyiam commented Oct 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlackBear2003 commented Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anilople left a comment

Choose a reason for hiding this comment

BlackBear2003 commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

BlackBear2003 commented Oct 19, 2023 •

edited

Loading

BlackBear2003 commented Oct 19, 2023 •

edited

Loading

BlackBear2003 commented Oct 25, 2023 •

edited

Loading