[Mellanox] Support one ingress pool mode #4686

stephenxs · 2020-06-02T07:23:52Z

Support one ingress pool mode.

Signed-off-by: Stephen Sun stephens@mellanox.com

- Why I did it
Support one ingress pool mode

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

liat-grozovik · 2020-06-09T15:51:55Z

retest vsimage please

baiwei0427

5570496 ingress shared buffer size for T1 still seems too small for me

neethajohn · 2020-07-09T20:30:19Z

According to the SAI spec, the buffer pool size includes the reserved buffer for all pg/queues. So that would mean we would have the same pool size specified for T1 and T0 templates and internally subtract the reserved space to get the actual shared pool size to allocate.
This is followed by other ASICs which do not include a headroom pool
https://github.com/opencomputeproject/SAI/blob/master/doc/QOS/SAI-Proposal-buffers-Ver4.docx

neethajohn

I see 'ingress_lossy_profile' being defined in the templates but not used anywhere. Is that intentional?

build_image.sh

files/image_config/secureboot/test-certs/signing.cert

files/image_config/secureboot/test-certs/ca.cert

files/image_config/secureboot/test-certs/signing.key

stephenxs · 2020-07-23T02:07:58Z

ingress_lossy_profile
Hi,
ingress_lossy_profile is referenced in BUFFER_PG|<port>|0. Please check sonic-buildimage/build_templates/buffers_config.j2

baiwei0427 · 2020-08-03T23:33:38Z

Do you have buffer templates for 2700 C32 sku?

baiwei0427

Why different SKUs get different Xoff sizes under the same link speed and cable length?

stephenxs · 2020-08-04T00:16:16Z

Why different SKUs get different Xoff sizes under the same link speed and cable length?

If they are based on different ASICs, they can have different xoff sizes because of different cell sizes.
If you're not meaning this, can you elaborate it a bit more?

stephenxs · 2020-08-04T00:27:58Z

Do you have buffer templates for 2700 C32 sku?

Yes. All the buffer templates of SKUs based 2700 share the same templates.
We can provide the specific template for C32 if you provide the cable lengths of ports for t0/t1.
Are they:

t0: 28 * server ports (5m) + 4 leaf ports (40m)
t1: 24 * TOR ports (40m) + 8 spine ports (300m)

?

baiwei0427 · 2020-08-04T01:16:23Z

Why different SKUs get different Xoff sizes under the same link speed and cable length?

If they are based on different ASICs, they can have different xoff sizes because of different cell sizes.
If you're not meaning this, can you elaborate it a bit more?

Got it! What are the cell sizes of 3800 and 4600?

baiwei0427 · 2020-08-04T01:21:39Z

Do you have buffer templates for 2700 C32 sku?

Yes. All the buffer templates of SKUs based 2700 share the same templates.
We can provide the specific template for C32 if you provide the cable lengths of ports for t0/t1.
Are they:

t0: 28 * server ports (5m) + 4 leaf ports (40m)

t1: 24 * TOR ports (40m) + 8 spine ports (300m)

?
Let's assume the worst case.
t0: 16 * server ports (100g, 5M) + 16 leaf ports (100g, 40m)
t1: 16 * TOR ports (100g, 40m) + 16 spine ports (100g, 300m)

stephenxs · 2020-08-04T02:06:00Z

Let's assume the worst case.
t0: 16 * server ports (100g, 5M) + 16 leaf ports (100g, 40m)
t1: 16 * TOR ports (100g, 40m) + 16 spine ports (100g, 300m)

Hi @liat-grozovik @lguohan ,
If we all agree on this setup I will calculate the buffer pool size for Mellanox-SN2700 based on this.

…lation Calculate pool size in t1 as 24 * downlink port + 8 * uplink port - Take both port and peer MTU into account when calculating headroom - Worst case factor is decreased to 50% - Mellanox-SN2700-C28D8 t0, assume 48 * 50G/5m + 8 * 100G/40m ports - Mellanox-SN2700 (C32) - t0: 16 * 100G/5m + 16 * 100G/40m - t1: 16 * 100G/40m + 16 * 100G/300m Signed-off-by: Stephen Sun <stephens@mellanox.com>

stephenxs · 2020-08-08T02:38:43Z

D48C8 and C32 should use different T1 templates

Done. Please review. @baiwei0427 @neethajohn

liat-grozovik · 2020-08-10T07:28:39Z

@baiwei0427 are we good to go with these values?

baiwei0427 · 2020-08-10T08:47:44Z

@baiwei0427 are we good to go with these values?

I will do calculations tomorrow and get back to you as soon as possible,

stephenxs · 2020-08-10T10:09:15Z

@baiwei0427 could you also provide the cable length information for calculating pool size for 3800 as well?
For 3800, we have the following SKUs:

Mellanox-SN3800-C64
Mellanox-SN3800-D112C8
Mellanox-SN3800-D24C52
Mellanox-SN3800-D28C50

thank you!

baiwei0427 · 2020-08-12T08:30:04Z

@baiwei0427 could you also provide the cable length information for calculating pool size for 3800 as well?
For 3800, we have the following SKUs:

Mellanox-SN3800-C64

Mellanox-SN3800-D112C8

Mellanox-SN3800-D24C52

Mellanox-SN3800-D28C50

thank you!

Server to T0: 5M
T0 to T1: 40M
T1 to T2: 300M

stephenxs · 2020-08-12T08:38:39Z

@baiwei0427

Server to T0: 5M
T0 to T1: 40M
T1 to T2: 300M

How many ports are connected to servers, TORs, leaf/spine routers respectively in each of the SKU?
Thanks

baiwei0427 · 2020-08-12T08:39:01Z

@liat-grozovik @stephenxs I did calculations for D48C8 T0, SN2700 T0, and SN2700 T1 and got similar results.

liat-grozovik · 2020-08-13T10:12:04Z

Note: Leaving aside new requirements for SN3800. this PR is for updating SN2700 SKUs.

abdosi · 2020-08-14T16:30:49Z

cherry-pick has conflict. Create PR for 201911

@liat-grozovik @rlhui

stephenxs · 2020-08-15T14:46:45Z

cherry-pick has conflict. Create PR for 201911

@liat-grozovik @rlhui

@abdosi Done PR #5194

…lation (sonic-net#4686) Calculate pool size in t1 as 24 * downlink port + 8 * uplink port - Take both port and peer MTU into account when calculating headroom - Worst case factor is decreased to 50% - Mellanox-SN2700-C28D8 t0, assume 48 * 50G/5m + 8 * 100G/40m ports - Mellanox-SN2700 (C32) - t0: 16 * 100G/5m + 16 * 100G/40m - t1: 16 * 100G/40m + 16 * 100G/300m Signed-off-by: Stephen Sun <stephens@mellanox.com> Co-authored-by: Stephen Sun <stephens@mellanox.com>

liat-grozovik requested review from lguohan and neethajohn June 9, 2020 15:52

liat-grozovik added Platform: Mellanox Enhancement ➕ labels Jun 9, 2020

neethajohn requested a review from baiwei0427 June 9, 2020 21:27

liat-grozovik added the Request for 201911 Branch label Jun 10, 2020

baiwei0427 reviewed Jun 15, 2020

View reviewed changes

stephenxs force-pushed the single-ingress-pool branch from 99e7f14 to 1010d2c Compare June 30, 2020 11:26

stephenxs marked this pull request as ready for review July 9, 2020 03:43

qiluo-msft force-pushed the master branch from 13769f9 to 39941f0 Compare July 12, 2020 18:13

neethajohn suggested changes Jul 22, 2020

View reviewed changes

stephenxs force-pushed the single-ingress-pool branch from 79f3287 to 323ef24 Compare July 29, 2020 09:42

stephenxs requested review from neethajohn and baiwei0427 August 1, 2020 09:14

baiwei0427 reviewed Aug 4, 2020

View reviewed changes

stephenxs closed this Aug 4, 2020

stephenxs reopened this Aug 4, 2020

stephenxs force-pushed the single-ingress-pool branch from e871991 to 4c0e8ae Compare August 8, 2020 02:37

stephenxs requested a review from baiwei0427 August 8, 2020 02:39

baiwei0427 approved these changes Aug 12, 2020

View reviewed changes

neethajohn approved these changes Aug 12, 2020

View reviewed changes

liat-grozovik merged commit 54fcdbb into sonic-net:master Aug 13, 2020

stephenxs deleted the single-ingress-pool branch August 14, 2020 23:02

stephenxs mentioned this pull request Aug 15, 2020

[Mellanox] Backport single pool to 201911 #5194

Merged

3 tasks

abdosi added the Included in 201911 Branch label Aug 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mellanox] Support one ingress pool mode #4686

[Mellanox] Support one ingress pool mode #4686

stephenxs commented Jun 2, 2020 •

edited

Loading

liat-grozovik commented Jun 9, 2020

baiwei0427 left a comment

neethajohn commented Jul 9, 2020

neethajohn left a comment

stephenxs commented Jul 23, 2020

baiwei0427 commented Aug 3, 2020

baiwei0427 left a comment

stephenxs commented Aug 4, 2020 •

edited

Loading

stephenxs commented Aug 4, 2020 •

edited

Loading

baiwei0427 commented Aug 4, 2020

baiwei0427 commented Aug 4, 2020

stephenxs commented Aug 4, 2020 •

edited

Loading

stephenxs commented Aug 8, 2020 •

edited

Loading

liat-grozovik commented Aug 10, 2020

baiwei0427 commented Aug 10, 2020

stephenxs commented Aug 10, 2020 •

edited

Loading

baiwei0427 commented Aug 12, 2020

stephenxs commented Aug 12, 2020 •

edited

Loading

baiwei0427 commented Aug 12, 2020

liat-grozovik commented Aug 13, 2020

abdosi commented Aug 14, 2020

stephenxs commented Aug 15, 2020

[Mellanox] Support one ingress pool mode #4686

[Mellanox] Support one ingress pool mode #4686

Conversation

stephenxs commented Jun 2, 2020 • edited Loading

liat-grozovik commented Jun 9, 2020

baiwei0427 left a comment

Choose a reason for hiding this comment

neethajohn commented Jul 9, 2020

neethajohn left a comment

Choose a reason for hiding this comment

stephenxs commented Jul 23, 2020

baiwei0427 commented Aug 3, 2020

baiwei0427 left a comment

Choose a reason for hiding this comment

stephenxs commented Aug 4, 2020 • edited Loading

stephenxs commented Aug 4, 2020 • edited Loading

baiwei0427 commented Aug 4, 2020

baiwei0427 commented Aug 4, 2020

stephenxs commented Aug 4, 2020 • edited Loading

stephenxs commented Aug 8, 2020 • edited Loading

liat-grozovik commented Aug 10, 2020

baiwei0427 commented Aug 10, 2020

stephenxs commented Aug 10, 2020 • edited Loading

baiwei0427 commented Aug 12, 2020

stephenxs commented Aug 12, 2020 • edited Loading

baiwei0427 commented Aug 12, 2020

liat-grozovik commented Aug 13, 2020

abdosi commented Aug 14, 2020

stephenxs commented Aug 15, 2020

stephenxs commented Jun 2, 2020 •

edited

Loading

stephenxs commented Aug 4, 2020 •

edited

Loading

stephenxs commented Aug 4, 2020 •

edited

Loading

stephenxs commented Aug 4, 2020 •

edited

Loading

stephenxs commented Aug 8, 2020 •

edited

Loading

stephenxs commented Aug 10, 2020 •

edited

Loading

stephenxs commented Aug 12, 2020 •

edited

Loading