Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SWSS ACL/Mirror] Race condition in Everflow mirror session and ACL rule creation #3035

Closed
keboliu opened this issue Jun 19, 2019 · 3 comments
Closed

[SWSS ACL/Mirror] Race condition in Everflow mirror session and ACL rule creation #3035

keboliu opened this issue Jun 19, 2019 · 3 comments

Comments

@keboliu
Copy link
Collaborator

keboliu commented Jun 19, 2019

Description
I have observed several failures in recent 201811 daily regression Everflow test(testcase_name=everflow_testbed), which seems caused by a race condition in ACL rule creation and mirror session creation.
In this test, it will create a mirror session first and then add ACL rules, the mirror session creation command always issued prior to ACL rules,
But when it comes to orchagent, the mirror session could be created after ACL rules, in this case, ACL rules creation will be failed due to the session not exist yet.

Steps to reproduce the issue:
not easy to reproduce, start the everflow_tb test when the DUT is busying update route entries may can have this issue reproduced.

commands issued by ansible to create mirror session and ACL rules:

Jun 17 18:34:23.438954 arc-mtbc-1001 INFO ansible-<stdin>: Invoked with warn=True executable=None chdir=None _raw_params=config mirror_session add test_session_1 1.1.1.1 2.2.2.2 8 1 0x8949 0 removes=None creates=None _uses_shell=False
Jun 17 18:34:24.458763 arc-mtbc-1001 INFO ansible-<stdin>: Invoked with warn=True executable=None chdir=None _raw_params=acl-loader update full /home/admin/everflow_tests/acl_rule_persistent.json --session_name=test_session_1 removes=None creates=None _uses_shell=False

Describe the results you received:
ACL rules not successfully created:

Jun 17 18:34:37.307616 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.307616 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.307816 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.307930 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308043 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308248 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308363 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308438 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308553 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308654 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308753 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308832 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308832 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308832 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308832 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308857 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308889 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308889 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.308993 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : test_session_1'
Jun 17 18:34:37.308993 arc-mtbc-1001 ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Jun 17 18:34:37.309355 arc-mtbc-1001 NOTICE swss#orchagent: :- createEntry: Created mirror session test_session_1
Jun 17 18:34:37.309624 arc-mtbc-1001 NOTICE swss#orchagent: :- attach: route0.0.0.0/0
Jun 17 18:34:37.345357 arc-mtbc-1001 NOTICE swss#orchagent: :- attach: Attached next hop observer of route 0.0.0.0/0 for destination IP 2.2.2.2
Jun 17 18:34:37.345609 arc-mtbc-1001 NOTICE swss#orchagent: :- updateNextHop: Updating mirror session test_session_1 with route 0.0.0.0/0
Jun 17 18:34:37.345660 arc-mtbc-1001 NOTICE swss#orchagent: :- getNeighborInfo: Mirror session test_session_1 neighbor is Ethernet0
Jun 17 18:34:37.346631 arc-mtbc-1001 NOTICE swss#orchagent: :- activateSession: Activate mirror session test_session_1

Describe the results you expected:

Jun 18 03:13:02.675453 arc-mtbc-1001 NOTICE swss#orchagent: :- createEntry: Created mirror session test_session_1
Jun 18 03:13:02.675453 arc-mtbc-1001 NOTICE swss#orchagent: :- attach: route0.0.0.0/0
Jun 18 03:13:02.701161 arc-mtbc-1001 NOTICE swss#orchagent: :- attach: Attached next hop observer of route 0.0.0.0/0 for destination IP 2.2.2.2
Jun 18 03:13:02.701470 arc-mtbc-1001 NOTICE swss#orchagent: :- updateNextHop: Updating mirror session test_session_1 with route 0.0.0.0/0
Jun 18 03:13:02.701582 arc-mtbc-1001 NOTICE swss#orchagent: :- getNeighborInfo: Mirror session test_session_1 neighbor is Ethernet0
Jun 18 03:13:02.702240 arc-mtbc-1001 NOTICE swss#orchagent: :- activateSession: Activate mirror session test_session_1
Jun 18 03:13:17.635537 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_2 in table EVERFLOW
Jun 18 03:13:17.650056 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_10 in table EVERFLOW
Jun 18 03:13:17.650056 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_1 in table EVERFLOW
Jun 18 03:13:17.651068 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_3 in table EVERFLOW
Jun 18 03:13:17.652981 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_6 in table EVERFLOW
Jun 18 03:13:17.657960 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_7 in table EVERFLOW
Jun 18 03:13:17.660443 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_8 in table EVERFLOW
Jun 18 03:13:17.666497 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_9 in table EVERFLOW
Jun 18 03:13:17.668022 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_4 in table EVERFLOW
Jun 18 03:13:17.669392 arc-mtbc-1001 NOTICE swss#orchagent: :- add: Successfully created ACL rule RULE_5 in table EVERFLOW

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

```
SONiC Software Version: SONiC.HEAD.70-cf9adfe
Distribution: Debian 9.9
Kernel: 4.9.0-8-amd64
Build commit: cf9adfe
Build date: Sun Jun 16 23:28:12 UTC 2019
Built by: johnar@jenkins-worker-4



```

**Attach debug file `sudo generate_dump`:**

syslog.zip
(paste your output here)
@lguohan
Copy link
Collaborator

lguohan commented Aug 11, 2020

@daall can you check these acl issues? is it still be applicable?

@daall
Copy link
Contributor

daall commented Aug 11, 2020

I think we need to add retry logic if the ACL rule is created before the mirror session is ready.

@daall
Copy link
Contributor

daall commented Dec 11, 2020

fixed by sonic-net/sonic-swss#1486

@daall daall closed this as completed Dec 11, 2020
@stcheng stcheng removed their assignment Jan 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lguohan @stcheng @keboliu @daall