Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[config qos] QoS and Buffer config genration for multi ASIC platforms #978

Merged
merged 8 commits into from
Aug 8, 2020

Conversation

smaheshm
Copy link
Contributor

@smaheshm smaheshm commented Jul 9, 2020

- What I did
To support buffer and QoS config generation for multi ASIC platform. In case of multi ASIC platforms each ASIC has its own buffer and qos configuration template which is used to populate corresponding ASIC's config DB.

Following commands were modified for this purpose:

# config qos reload
# config qos clear

In multi ASIC platforms the command runs on all namespaces. In case of single ASIC platform the command runs in global namespace.

admin@str-nSonic-acs-2:~$ sudo config qos reload --help                                                                                                                                
Usage: config qos reload [OPTIONS]                                                                                                                                                    

  Reload QoS configuration

Options:
  -?, -h, --help        Show this message and exit.
admin@str-nSonic-acs-2:~$ sudo config qos clear --help                                                                                                                                 
Usage: config qos clear [OPTIONS]

  Clear QoS configuration

Options:
  -?, -h, --help        Show this message and exit.         

- How I did it
Modified config generation script to generate configs for all ASICs in case of multi-ASIC platform.

- How to verify it

verified configs generated on both multi-ASIC and single ASIC platforms. Verified traffic with different priorities using queue counters.

Single ASIC Platform:

admin@sonic:/usr/local/lib/python2.7/dist-packages$ cd
admin@sonic:~$ show platform summary
Platform: x86_64-dell_s6000_s1220-r0
HwSKU: Force10-S6000
ASIC: broadcom
admin@sonic:~$ sudo config qos reload
Running command: /usr/local/bin/sonic-cfggen  -d -t /usr/share/sonic/device/x86_64-dell_s6000_s1220-r0/Force10-S6000/buffers.json.j2 > /tmp/buffers.json
Running command: /usr/local/bin/sonic-cfggen  -d -t /usr/share/sonic/device/x86_64-dell_s6000_s1220-r0/Force10-S6000/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos.json
Running command: /usr/local/bin/sonic-cfggen  -j /tmp/buffers.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen  -j /tmp/qos.json --write-to-db
admin@sonic:~$ sudo config qos clear

Multi ASIC Platform

Before generating configs:

admin@str-nSonic-acs-2:~$ docker exec -it database0 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
(empty list or set)
127.0.0.1:6379[4]>
admin@str-nSonic-acs-2:~$ docker exec -it database2 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
(empty list or set)
127.0.0.1:6379[4]>


admin@str-nSonic-acs-2:~$ sudo config qos reload
Running command: /usr/local/bin/sonic-cfggen -n asic0 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/0/buffers.json.j2 > /tmp/buffers0.json
Running command: /usr/local/bin/sonic-cfggen -n asic0 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/0/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos0.json
Running command: /usr/local/bin/sonic-cfggen -n asic0 -j /tmp/buffers0.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic0 -j /tmp/qos0.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic1 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/1/buffers.json.j2 > /tmp/buffers1.json
Running command: /usr/local/bin/sonic-cfggen -n asic1 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/1/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos1.json
Running command: /usr/local/bin/sonic-cfggen -n asic1 -j /tmp/buffers1.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic1 -j /tmp/qos1.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic2 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/2/buffers.json.j2 > /tmp/buffers2.json
Running command: /usr/local/bin/sonic-cfggen -n asic2 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/2/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos2.json
Running command: /usr/local/bin/sonic-cfggen -n asic2 -j /tmp/buffers2.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic2 -j /tmp/qos2.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic3 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/3/buffers.json.j2 > /tmp/buffers3.json
Running command: /usr/local/bin/sonic-cfggen -n asic3 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/3/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos3.json
Running command: /usr/local/bin/sonic-cfggen -n asic3 -j /tmp/buffers3.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic3 -j /tmp/qos3.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic4 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/4/buffers.json.j2 > /tmp/buffers4.json
Running command: /usr/local/bin/sonic-cfggen -n asic4 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/4/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos4.json
Running command: /usr/local/bin/sonic-cfggen -n asic4 -j /tmp/buffers4.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic4 -j /tmp/qos4.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic5 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/5/buffers.json.j2 > /tmp/buffers5.json
Running command: /usr/local/bin/sonic-cfggen -n asic5 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/5/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos5.json
Running command: /usr/local/bin/sonic-cfggen -n asic5 -j /tmp/buffers5.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic5 -j /tmp/qos5.json --write-to-db
admin@str-nSonic-acs-2:~$

After generating configs:

admin@str-nSonic-acs-2:/usr/local/lib/python2.7/dist-packages$ docker exec -it database2 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
  1) "BUFFER_PROFILE|pg_lossless_40000_300m_profile"
  2) "BUFFER_PG|Ethernet-BP184|3-4"
  3) "BUFFER_QUEUE|Ethernet148|0-2"
  4) "BUFFER_POOL|egress_lossless_pool"
  5) "BUFFER_QUEUE|Ethernet184|3-4"
  6) "BUFFER_QUEUE|Ethernet-BP172|3-4"
  7) "BUFFER_QUEUE|Ethernet-BP172|5-6"
  8) "BUFFER_PG|Ethernet140|3-4"
  9) "BUFFER_QUEUE|Ethernet168|3-4"
.
.


admin@str-nSonic-acs-2:~$ sudo ip netns exec asic0 show mmu
Pool: ingress_lossless_pool
----  --------
type  ingress
mode  dynamic
size  12766208
----  --------

Pool: egress_lossless_pool
----  --------
type  egress
mode  static
size  12766208
----  --------

Pool: egress_lossy_pool
----  -------
type  egress
mode  dynamic
size  7326924
----  -------

Profile: pg_lossless_40000_300m_profile
----------  -----------------------------------
xon_offset  2496
dynamic_th  -3
xon         18432
xoff        55120
pool        [BUFFER_POOL|ingress_lossless_pool]
size        56368
----------  -----------------------------------

Profile: pg_lossless_40000_5m_profile
----------  -----------------------------------
xon_offset  2496
dynamic_th  -3
xon         18432
xoff        55120
pool        [BUFFER_POOL|ingress_lossless_pool]
size        56368
----------  -----------------------------------

Profile: egress_lossy_profile
----------  -------------------------------
dynamic_th  3
pool        [BUFFER_POOL|egress_lossy_pool]
size        1518
----------  -------------------------------

Profile: ingress_lossy_profile
----------  -----------------------------------
dynamic_th  3
pool        [BUFFER_POOL|ingress_lossless_pool]
size        0
----------  -----------------------------------

Profile: egress_lossless_profile
---------  ----------------------------------
static_th  12766208
pool       [BUFFER_POOL|egress_lossless_pool]
size       0
---------  ----------------------------------


admin@str-nSonic-acs-2:~$ sudo ip netns exec asic0 show queue counters Ethernet-BP40
         Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
-------------  -----  --------------  ---------------  -----------  ------------
Ethernet-BP40    UC0             252            44240            0             0
Ethernet-BP40    UC1            3000           192000            0             0
Ethernet-BP40    UC2            5000           320000            0             0
Ethernet-BP40    UC3            1000            64000            0             0
Ethernet-BP40    UC4            1000            64000            0             0
Ethernet-BP40    UC5               0                0            0             0
Ethernet-BP40    UC6               0                0            0             0
Ethernet-BP40    UC7               0                0            0             0
Ethernet-BP40    MC8               0                0            0             0
Ethernet-BP40    MC9               0                0            0             0
Ethernet-BP40   MC10               0                0            0             0
Ethernet-BP40   MC11               0                0            0             0
Ethernet-BP40   MC12               0                0            0             0
Ethernet-BP40   MC13               0                0            0             0
Ethernet-BP40   MC14               0                0            0             0
Ethernet-BP40   MC15               0                0            0             0


admin@str-nSonic-acs-2:/etc/sonic$ grep "Ethernet-BP..|3-4" config_db0.json -A2
        "Ethernet-BP12|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
        "Ethernet-BP16|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
        "Ethernet-BP20|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
.
.

- Previous command output (if the output of a command-line utility has changed)

- New command output (if the output of a command-line utility has changed)

Copy link
Contributor

@arlakshm arlakshm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comments

config/main.py Outdated Show resolved Hide resolved
config/main.py Outdated Show resolved Hide resolved
@smaheshm
Copy link
Contributor Author

@arlakshm @neethajohn Let me know if I should add any other reviewer.

config/main.py Outdated Show resolved Hide resolved
config/main.py Outdated Show resolved Hide resolved
neethajohn
neethajohn previously approved these changes Jul 24, 2020
@smaheshm
Copy link
Contributor Author

@arlakshm Ok with the changes?

config/main.py Outdated Show resolved Hide resolved
format(ns),
fg='yellow'
)
raise click.Abort()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have sonic_device_util.get_all_namespaces() that provide the dictionary of front and back namespace with string as namespace value. Then no need to convert from id to string. We can see if this API can be used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need the ASIC ID. That API doesn't give the ASIC ID.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't we have directory name as "asic0" ? i feel that is better and correlate to namespace directory created by linux.
We can change port_config.ini to also be in similar folder

@judyjoseph @SuvarnaMeenakshi what do you think here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a temporary directory, it can be anything. I still need the ASIC ID to read, and matches that format.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

Copy link
Contributor Author

@smaheshm smaheshm Jul 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ASIC IDs and ASIC namespaces are different. The configuration is for each ASIC ID rather than namespace. I think it's OK to specify ASIC ID for configuration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A simple approach would be to get the num of asics present in the platform and do a for range loop. We are using the asic_index as asic_id's internally.
If we derive the asic_id from namespace name, could cause problems if we create a namespace for a different purpose in the switch ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using 'get_namespaces()' API, this should return the valid namespaces for ASICs. It's better to use common APIs so there's one place that calculates the ASIC ID and namespace. Using a range loop can be fragile if the ASIC ID allocation scheme changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a temporary directory, it can be anything. I still need the ASIC ID to read, and matches that format.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

@smaheshm why can't asic_id_suffix be namespace name directly ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the directories use 'asic ID'.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 3, 2020

retest this please

@smaheshm smaheshm requested a review from arlakshm August 3, 2020 18:14
@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 3, 2020

retest this please

1 similar comment
@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 4, 2020

retest this please

@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 4, 2020

retest this please

@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 5, 2020

retest this please

config/main.py Show resolved Hide resolved
@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 5, 2020

retest this please

2 similar comments
@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 6, 2020

retest this please

@smaheshm
Copy link
Contributor Author

smaheshm commented Aug 7, 2020

retest this please

Copy link
Contributor

@arlakshm arlakshm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@smaheshm smaheshm merged commit a80826d into sonic-net:master Aug 8, 2020
@abdosi
Copy link
Contributor

abdosi commented Sep 3, 2020

@smaheshm Create PR for 201911

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants