Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reclaim buffer] Common infrastructure update for reclaiming buffer #9133

Merged
merged 5 commits into from
Nov 24, 2021

Conversation

stephenxs
Copy link
Collaborator

@stephenxs stephenxs commented Nov 1, 2021

Why I did it

This is to update the common sonic-buildimage infra for reclaiming buffer.

How I did it

  • Render zero_profiles.j2 to zero_profiles.json for vendors that support reclaiming buffer
    The zero profiles will be referenced in PR [Reclaim buffer] Reclaim unused buffers by applying zero buffer profiles #8768 on Mellanox platforms and there will be test cases to verify the behavior there.
    Rendering is done here for passing azure pipeline.
  • Load zero_profiles.json when the dynamic buffer manager starts
  • Generate inactive port list to reclaim buffer

It depends on PR sonic-net/sonic-swss#1996 and advancing submodule #9288

How to verify it

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

@stephenxs
Copy link
Collaborator Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

@stephenxs
Copy link
Collaborator Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

@stephenxs
Copy link
Collaborator Author

/azpw run vs

@mssonicbld
Copy link
Collaborator

/AzurePipelines run vs

@azure-pipelines
Copy link

No pipelines are associated with this pull request.

@stephenxs
Copy link
Collaborator Author

/azpw run Azure.sonic-buildimage

@mssonicbld
Copy link
Collaborator

/AzurePipelines run Azure.sonic-buildimage

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

keboliu
keboliu previously approved these changes Nov 15, 2021
….sh accordingly

Signed-off-by: Stephen Sun <stephens@nvidia.com>
Signed-off-by: Stephen Sun <stephens@nvidia.com>
Signed-off-by: Stephen Sun <stephens@nvidia.com>
Signed-off-by: Stephen Sun <stephens@nvidia.com>
Signed-off-by: Stephen Sun <stephens@nvidia.com>
@stephenxs
Copy link
Collaborator Author

stephenxs commented Nov 17, 2021

More than one PRs of mine failed due to the following error.
Looks like it's a community issue.
@lguohan @qiluo-msft @yxieca @prsunny is there anyone from the community who can help? thanks

2021-11-16T14:04:36.8871692Z =================================== FAILURES ===================================
2021-11-16T14:04:36.8872321Z ___________________ Test_SonicYang.test_validate_yang_models ___________________
2021-11-16T14:04:36.8872612Z 
2021-11-16T14:04:36.8873051Z self = <test_sonic_yang.Test_SonicYang object at 0x7fdeaa38c790>
2021-11-16T14:04:36.8874476Z sonic_yang_data = {'syc': <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>, 'test_file': '../sonic-yang-models/tests/files/sample_config_db.json', 'yang_dir': '../sonic-yang-models/yang-models/'}
2021-11-16T14:04:36.8875089Z 
2021-11-16T14:04:36.8875578Z     def test_validate_yang_models(self, sonic_yang_data):
2021-11-16T14:04:36.8876463Z         '''
2021-11-16T14:04:36.8876915Z         In this test, we validate yang models
2021-11-16T14:04:36.8877439Z         a.) by converting the config as per RFC 7951 using YANG Models,
2021-11-16T14:04:36.8923505Z         b.) by creating data tree using new YANG models and
2021-11-16T14:04:36.8924186Z         c.) by validating config against YANG models.
2021-11-16T14:04:36.8924913Z         Successful execution of these steps can be treated as
2021-11-16T14:04:36.8925512Z         validation of new Yang models.
2021-11-16T14:04:36.8926839Z         '''
2021-11-16T14:04:36.8927520Z         test_file = sonic_yang_data['test_file']
2021-11-16T14:04:36.8928262Z         syc = sonic_yang_data['syc']
2021-11-16T14:04:36.8928740Z         # Currently only 3 YANG files are not directly related to config
2021-11-16T14:04:36.8929758Z         # which are: sonic-extension.yang, sonic-types.yang and sonic-bgp-common.yang. Hard coding
2021-11-16T14:04:36.8930312Z         # it right now.
2021-11-16T14:04:36.8930810Z         # If any more such helper yang files are added, we need to update here.
2021-11-16T14:04:36.8931320Z         NON_CONFIG_YANG_FILES = 3
2021-11-16T14:04:36.8931712Z         # read config
2021-11-16T14:04:36.8932615Z         jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
2021-11-16T14:04:36.8933477Z         jIn = json.loads(jIn)
2021-11-16T14:04:36.8933910Z         numTables = len(jIn)
2021-11-16T14:04:36.8934334Z         # load config and create Data tree
2021-11-16T14:04:36.8934769Z         syc.loadData(jIn)
2021-11-16T14:04:36.8935258Z         # check all tables are loaded and config related to all Yang Models is
2021-11-16T14:04:36.8935776Z         # loaded in Data tree.
2021-11-16T14:04:36.8936181Z >       assert len(syc.jIn) == numTables
2021-11-16T14:04:36.8936661Z E       AssertionError: assert 76 == 78
2021-11-16T14:04:36.8938415Z E        +  where 76 = len({'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...})
2021-11-16T14:04:36.8940896Z E        +    where {'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...} = <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>.jIn
2021-11-16T14:04:36.8941820Z 
2021-11-16T14:04:36.8942558Z tests/libyang-python-tests/test_sonic_yang.py:308: AssertionError
2021-11-16T14:04:36.8943075Z ----------------------------- Captured stdout call -----------------------------
2021-11-16T14:04:36.8943401Z  Read JSON Section: SAMPLE_CONFIG_DB_JSON
2021-11-16T14:04:36.8943856Z Note: Below table(s) have no YANG models:
2021-11-16T14:04:36.8944118Z ACL_RULE, ACL_TABLE, 
2021-11-16T14:04:36.8944827Z _____________________ Test_SonicYang.test_xlate_rev_xlate ______________________
2021-11-16T14:04:36.8945029Z 
2021-11-16T14:04:36.8945317Z self = <test_sonic_yang.Test_SonicYang object at 0x7fdeaa323ca0>
2021-11-16T14:04:36.8946222Z sonic_yang_data = {'syc': <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>, 'test_file': '../sonic-yang-models/tests/files/sample_config_db.json', 'yang_dir': '../sonic-yang-models/yang-models/'}
2021-11-16T14:04:36.8946653Z 
2021-11-16T14:04:36.8949843Z     def test_xlate_rev_xlate(self, sonic_yang_data):
2021-11-16T14:04:36.8950461Z         # In this test, xlation and revXlation is tested with latest Sonic
2021-11-16T14:04:36.8951334Z         # YANG model.
2021-11-16T14:04:36.8952397Z         test_file = sonic_yang_data['test_file']
2021-11-16T14:04:36.8953415Z         syc = sonic_yang_data['syc']
2021-11-16T14:04:36.8953819Z     
2021-11-16T14:04:36.8955049Z         jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
2021-11-16T14:04:36.8955735Z         jIn = json.loads(jIn)
2021-11-16T14:04:36.8956322Z         numTables = len(jIn)
2021-11-16T14:04:36.8956690Z     
2021-11-16T14:04:36.8957028Z         syc.loadData(jIn)
2021-11-16T14:04:36.8957866Z         # check all tables are loaded and no tables is without Yang Models
2021-11-16T14:04:36.8958555Z >       assert len(syc.jIn) == numTables
2021-11-16T14:04:36.8959043Z E       AssertionError: assert 76 == 78
2021-11-16T14:04:36.8960895Z E        +  where 76 = len({'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...})
2021-11-16T14:04:36.8963078Z E        +    where {'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...} = <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>.jIn
2021-11-16T14:04:36.8963683Z 
2021-11-16T14:04:36.8964324Z tests/libyang-python-tests/test_sonic_yang.py:334: AssertionError
2021-11-16T14:04:36.8965025Z ----------------------------- Captured stdout call -----------------------------
2021-11-16T14:04:36.8965369Z  Read JSON Section: SAMPLE_CONFIG_DB_JSON
2021-11-16T14:04:36.8965793Z Note: Below table(s) have no YANG models:
2021-11-16T14:04:36.8966031Z ACL_RULE, ACL_TABLE, 
2021-11-16T14:04:36.8966870Z =============================== warnings summary ===============================

@neethajohn
Copy link
Contributor

More than one PRs of mine failed due to the following error. Looks like it's a community issue. @lguohan @qiluo-msft @yxieca @prsunny is there anyone from the community who can help? thanks

2021-11-16T14:04:36.8871692Z =================================== FAILURES ===================================
2021-11-16T14:04:36.8872321Z ___________________ Test_SonicYang.test_validate_yang_models ___________________
2021-11-16T14:04:36.8872612Z 
2021-11-16T14:04:36.8873051Z self = <test_sonic_yang.Test_SonicYang object at 0x7fdeaa38c790>
2021-11-16T14:04:36.8874476Z sonic_yang_data = {'syc': <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>, 'test_file': '../sonic-yang-models/tests/files/sample_config_db.json', 'yang_dir': '../sonic-yang-models/yang-models/'}
2021-11-16T14:04:36.8875089Z 
2021-11-16T14:04:36.8875578Z     def test_validate_yang_models(self, sonic_yang_data):
2021-11-16T14:04:36.8876463Z         '''
2021-11-16T14:04:36.8876915Z         In this test, we validate yang models
2021-11-16T14:04:36.8877439Z         a.) by converting the config as per RFC 7951 using YANG Models,
2021-11-16T14:04:36.8923505Z         b.) by creating data tree using new YANG models and
2021-11-16T14:04:36.8924186Z         c.) by validating config against YANG models.
2021-11-16T14:04:36.8924913Z         Successful execution of these steps can be treated as
2021-11-16T14:04:36.8925512Z         validation of new Yang models.
2021-11-16T14:04:36.8926839Z         '''
2021-11-16T14:04:36.8927520Z         test_file = sonic_yang_data['test_file']
2021-11-16T14:04:36.8928262Z         syc = sonic_yang_data['syc']
2021-11-16T14:04:36.8928740Z         # Currently only 3 YANG files are not directly related to config
2021-11-16T14:04:36.8929758Z         # which are: sonic-extension.yang, sonic-types.yang and sonic-bgp-common.yang. Hard coding
2021-11-16T14:04:36.8930312Z         # it right now.
2021-11-16T14:04:36.8930810Z         # If any more such helper yang files are added, we need to update here.
2021-11-16T14:04:36.8931320Z         NON_CONFIG_YANG_FILES = 3
2021-11-16T14:04:36.8931712Z         # read config
2021-11-16T14:04:36.8932615Z         jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
2021-11-16T14:04:36.8933477Z         jIn = json.loads(jIn)
2021-11-16T14:04:36.8933910Z         numTables = len(jIn)
2021-11-16T14:04:36.8934334Z         # load config and create Data tree
2021-11-16T14:04:36.8934769Z         syc.loadData(jIn)
2021-11-16T14:04:36.8935258Z         # check all tables are loaded and config related to all Yang Models is
2021-11-16T14:04:36.8935776Z         # loaded in Data tree.
2021-11-16T14:04:36.8936181Z >       assert len(syc.jIn) == numTables
2021-11-16T14:04:36.8936661Z E       AssertionError: assert 76 == 78
2021-11-16T14:04:36.8938415Z E        +  where 76 = len({'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...})
2021-11-16T14:04:36.8940896Z E        +    where {'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...} = <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>.jIn
2021-11-16T14:04:36.8941820Z 
2021-11-16T14:04:36.8942558Z tests/libyang-python-tests/test_sonic_yang.py:308: AssertionError
2021-11-16T14:04:36.8943075Z ----------------------------- Captured stdout call -----------------------------
2021-11-16T14:04:36.8943401Z  Read JSON Section: SAMPLE_CONFIG_DB_JSON
2021-11-16T14:04:36.8943856Z Note: Below table(s) have no YANG models:
2021-11-16T14:04:36.8944118Z ACL_RULE, ACL_TABLE, 
2021-11-16T14:04:36.8944827Z _____________________ Test_SonicYang.test_xlate_rev_xlate ______________________
2021-11-16T14:04:36.8945029Z 
2021-11-16T14:04:36.8945317Z self = <test_sonic_yang.Test_SonicYang object at 0x7fdeaa323ca0>
2021-11-16T14:04:36.8946222Z sonic_yang_data = {'syc': <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>, 'test_file': '../sonic-yang-models/tests/files/sample_config_db.json', 'yang_dir': '../sonic-yang-models/yang-models/'}
2021-11-16T14:04:36.8946653Z 
2021-11-16T14:04:36.8949843Z     def test_xlate_rev_xlate(self, sonic_yang_data):
2021-11-16T14:04:36.8950461Z         # In this test, xlation and revXlation is tested with latest Sonic
2021-11-16T14:04:36.8951334Z         # YANG model.
2021-11-16T14:04:36.8952397Z         test_file = sonic_yang_data['test_file']
2021-11-16T14:04:36.8953415Z         syc = sonic_yang_data['syc']
2021-11-16T14:04:36.8953819Z     
2021-11-16T14:04:36.8955049Z         jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
2021-11-16T14:04:36.8955735Z         jIn = json.loads(jIn)
2021-11-16T14:04:36.8956322Z         numTables = len(jIn)
2021-11-16T14:04:36.8956690Z     
2021-11-16T14:04:36.8957028Z         syc.loadData(jIn)
2021-11-16T14:04:36.8957866Z         # check all tables are loaded and no tables is without Yang Models
2021-11-16T14:04:36.8958555Z >       assert len(syc.jIn) == numTables
2021-11-16T14:04:36.8959043Z E       AssertionError: assert 76 == 78
2021-11-16T14:04:36.8960895Z E        +  where 76 = len({'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...})
2021-11-16T14:04:36.8963078Z E        +    where {'AAA': {'accounting': {'login': 'local'}, 'authentication': {'login': 'local'}, 'authorization': {'login': 'local'}},...nterval': '600', 'state': 'enabled'}}, 'BGP_GLOBALS': {'default': {'local_asn': '65001', 'router_id': '5.5.5.5'}}, ...} = <sonic_yang.SonicYang object at 0x7fdeaadf3ac0>.jIn
2021-11-16T14:04:36.8963683Z 
2021-11-16T14:04:36.8964324Z tests/libyang-python-tests/test_sonic_yang.py:334: AssertionError
2021-11-16T14:04:36.8965025Z ----------------------------- Captured stdout call -----------------------------
2021-11-16T14:04:36.8965369Z  Read JSON Section: SAMPLE_CONFIG_DB_JSON
2021-11-16T14:04:36.8965793Z Note: Below table(s) have no YANG models:
2021-11-16T14:04:36.8966031Z ACL_RULE, ACL_TABLE, 
2021-11-16T14:04:36.8966870Z =============================== warnings summary ===============================

This is been looked into

Copy link
Contributor

@neethajohn neethajohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the description to specify that #8768 will have the associated test cases for the mellanox specific template rendering? Also mention if there are merge dependencies

dockers/docker-orchagent/buffermgrd.sh Show resolved Hide resolved
platform/vs/docker-sonic-vs/buffermgrd.sh Show resolved Hide resolved
@stephenxs
Copy link
Collaborator Author

Can you update the description to specify that #8768 will have the associated test cases for the mellanox specific template rendering? Also mention if there are merge dependencies

Done.

@stephenxs
Copy link
Collaborator Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

@stephenxs
Copy link
Collaborator Author

/azpw run azure.sonic-buildimage

@mssonicbld
Copy link
Collaborator

/AzurePipelines run azure.sonic-buildimage

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@stephenxs
Copy link
Collaborator Author

/azpw run azure.sonic-buildimage

@mssonicbld
Copy link
Collaborator

/AzurePipelines run azure.sonic-buildimage

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@stephenxs
Copy link
Collaborator Author

stephenxs commented Nov 20, 2021

It can be cherry-picked to 202106 and 202012 branches based on the following commit respectively. It depends on the following PR to be merged to 202012 and 202106 before cherry picking: sonic-net/sonic-swss#2038 which is to back port sonic-net/sonic-swss#1996
Labels added.
202106

commit 1ebe52847ac1655a002fe2e3eb716b80a4400ffd
Author: shlomibitton <60430976+shlomibitton@users.noreply.github.com>
Date:   Thu Nov 18 19:40:48 2021 +0200

    [DHCPv6 relay] [202106] Fix DHCPv6 design to support multiple VLANS (#9163)

202012

commit d6ab40970998a6c1025c7fcba51c3180dcec4d3c
Author: Prince Sunny <prince.sunny@microsoft.com>
Date:   Thu Nov 18 12:48:20 2021 -0800

    [202012] td2/td3 change cpu cos num to 10 (#9311)

    Cherry-pick from #9301

@liat-grozovik liat-grozovik merged commit b3ccef9 into sonic-net:master Nov 24, 2021
@stephenxs stephenxs deleted the zero-profiles-base branch November 24, 2021 13:17
qiluo-msft pushed a commit that referenced this pull request Dec 1, 2021
…9133)

- Why I did it
This is to update the common sonic-buildimage infra for reclaiming buffer.

- How I did it
Render zero_profiles.j2 to zero_profiles.json for vendors that support reclaiming buffer
The zero profiles will be referenced in PR [Reclaim buffer] Reclaim unused buffers by applying zero buffer profiles #8768 on Mellanox platforms and there will be test cases to verify the behavior there.
Rendering is done here for passing azure pipeline.
Load zero_profiles.json when the dynamic buffer manager starts
Generate inactive port list to reclaim buffer

Signed-off-by: Stephen Sun <stephens@nvidia.com>
judyjoseph pushed a commit that referenced this pull request Dec 1, 2021
…9133)

- Why I did it
This is to update the common sonic-buildimage infra for reclaiming buffer.

- How I did it
Render zero_profiles.j2 to zero_profiles.json for vendors that support reclaiming buffer
The zero profiles will be referenced in PR [Reclaim buffer] Reclaim unused buffers by applying zero buffer profiles #8768 on Mellanox platforms and there will be test cases to verify the behavior there.
Rendering is done here for passing azure pipeline.
Load zero_profiles.json when the dynamic buffer manager starts
Generate inactive port list to reclaim buffer

Signed-off-by: Stephen Sun <stephens@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants