Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NVIDIA Mellanox Technologies MT2894 Family Ethernet Controller ConnectX-6 Lx #358

Merged
merged 1 commit into from
Sep 13, 2022
Merged

Add NVIDIA Mellanox Technologies MT2894 Family Ethernet Controller ConnectX-6 Lx #358

merged 1 commit into from
Sep 13, 2022

Conversation

wizhaoredhat
Copy link
Contributor

Fixes #357

Signed-off-by: William Zhao wizhao@redhat.com

@wizhaoredhat
Add NVIDIA Mellanox Technologies MT2894 Family Ethernet Controller Co…
5fba3aa 
…nnectX-6 Lx

Fixes k8snetworkplumbingwg#357

Signed-off-by: William Zhao <wizhao@redhat.com>
@github-actions
Copy link

github-actions bot commented Sep 9, 2022

Thanks for your PR,
To run vendors CIs use one of:

  • /test-all: To run all tests for all vendors.
  • /test-e2e-all: To run all E2E tests for all vendors.
  • /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

  • /skip-all: To skip all tests for all vendors.
  • /skip-e2e-all: To skip all E2E tests for all vendors.
  • /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
    Best regards.

@wizhaoredhat
Copy link
Contributor Author

@adrianchiris FYI

@adrianchiris
Copy link
Collaborator

@wizhaoredhat did you run connectx6 lx with openshift ?

@coveralls
Copy link

Pull Request Test Coverage Report for Build 3025470418

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 2 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.04%) to 23.688%
Files with Coverage Reduction New Missed Lines %
controllers/sriovnetwork_controller.go 2 69.0%
Totals Coverage Status
Change from base Build 3014972728: 0.04%
Covered Lines: 1752
Relevant Lines: 7396
💛 - Coveralls

@wizhaoredhat
Copy link
Contributor Author

@wizhaoredhat did you run connectx6 lx with openshift ?

@adrianchiris
Yes, I did.

@adrianchiris
Copy link
Collaborator

super ! thx @wizhaoredhat

@wizhaoredhat
Copy link
Contributor Author

wizhaoredhat commented Sep 12, 2022 •
edited
Loading

@adrianchiris @zshi-redhat Could you merge this PR as well?

@SchSeba
Copy link
Collaborator

SchSeba commented Sep 12, 2022

/hold

@github-actions github-actions bot added the hold label Sep 12, 2022
@SchSeba
Copy link
Collaborator

SchSeba commented Sep 12, 2022

Hi @wizhaoredhat can you try to use the u/s sriov-config-daemon image?

ghcr.io/k8snetworkplumbingwg/sriov-network-operator-config-daemon:v1.2.0

If I remember the problem was centos stream don't have the right version of mstflint to support this card.

e0ne
e0ne approved these changes Sep 12, 2022
View reviewed changes
Copy link
Collaborator

@e0ne e0ne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@SchSeba
Copy link
Collaborator

SchSeba commented Sep 12, 2022

#317

@wizhaoredhat
Copy link
Contributor Author

wizhaoredhat commented Sep 12, 2022 •
edited
Loading

ghcr.io/k8snetworkplumbingwg/sriov-network-operator-config-daemon:v1.2.0

@SchSeba

Looks good to me:

[root@wsfd-advnetlab25 ~]# oc exec -it sriov-network-config-daemon-m9zh7 -n openshift-sriov-network-operator -c sriov-network-config-daemon -- /bin/bash
[root@worker-advnetlab23 /]# mstconfig -d 5e:00.0 q

Device #1:
----------

Device type:    ConnectX6LX     
Name:           MCX631102AN-ADA_Ax
Description:    ConnectX-6 Lx EN adapter card; 25GbE ; Dual-port SFP28; PCIe 4.0 x8; No Crypto
Device:         5e:00.0   
[root@worker-advnetlab23 /]# mstflint -v
mstflint, mstflint 4.19.0, built on Apr  7 2022, 23:09:23. Git SHA Hash: 2b02298
[root@worker-advnetlab23 /]# cat /etc/redhat-release 
CentOS Stream release 8
[root@wsfd-advnetlab25 ~]# oc describe pod sriov-network-config-daemon-m9zh7 -n openshift-sriov-network-operator 
...
  sriov-network-config-daemon:
    Container ID:  cri-o://2b5f7f3fd6b667d73ef10093852304eaa26cca98ff26b0c34b36789f9a91d178
    Image:         ghcr.io/k8snetworkplumbingwg/sriov-network-operator-config-daemon:v1.2.0
    Image ID:      ghcr.io/k8snetworkplumbingwg/sriov-network-operator-config-daemon@sha256:018fb75722a5287bcd74c4b6aac27e3480bad1c39465873d2ecc62cd04c92001
...
[root@wsfd-advnetlab25 ~]# oc logs sriov-network-config-daemon-m9zh7 -n openshift-sriov-network-operator -c sriov-network-config-daemon
...
I0912 19:07:59.445256 1624833 utils.go:749] RunCommand(): 
Device #1:
----------

Device type:    ConnectX6LX     
Name:           MCX631102AN-ADA_Ax
Description:    ConnectX-6 Lx EN adapter card; 25GbE ; Dual-port SFP28; PCIe 4.0 x8; No Crypto
Device:         0000:5e:00.0    

Configurations:                              Default         Current         Next Boot
         MEMIC_BAR_SIZE                      0               0               0               
         MEMIC_SIZE_LIMIT                    _256KB(1)       _256KB(1)       _256KB(1)       
         HOST_CHAINING_MODE                  DISABLED(0)     DISABLED(0)     DISABLED(0)     
         HOST_CHAINING_CACHE_DISABLE         False(0)        False(0)        False(0)        
         HOST_CHAINING_DESCRIPTORS           Array[0..7]     Array[0..7]     Array[0..7]     
         HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]     Array[0..7]     Array[0..7]     
         FLEX_PARSER_PROFILE_ENABLE          0               0               0               
         PROG_PARSE_GRAPH                    False(0)        False(0)        False(0)        
         FLEX_IPV4_OVER_VXLAN_PORT           0               0               0               
         ROCE_NEXT_PROTOCOL                  254             254             254             
         ESWITCH_HAIRPIN_DESCRIPTORS         Array[0..7]     Array[0..7]     Array[0..7]     
         ESWITCH_HAIRPIN_TOT_BUFFER_SIZE     Array[0..7]     Array[0..7]     Array[0..7]     
         PF_BAR2_SIZE                        0               0               0               
         NON_PREFETCHABLE_PF_BAR             False(0)        False(0)        False(0)        
         VF_VPD_ENABLE                       False(0)        False(0)        False(0)        
         PF_NUM_PF_MSIX_VALID                False(0)        False(0)        False(0)        
         PER_PF_NUM_SF                       False(0)        False(0)        False(0)        
         STRICT_VF_MSIX_NUM                  False(0)        False(0)        False(0)        
         VF_NODNIC_ENABLE                    False(0)        False(0)        False(0)        
         NUM_PF_MSIX_VALID                   True(1)         True(1)         True(1)         
*        NUM_OF_VFS                          8               11              11              
         NUM_OF_PF                           2               2               2               
         PF_BAR2_ENABLE                      False(0)        False(0)        False(0)        
         SRIOV_EN                            True(1)         True(1)         True(1)         
         PF_LOG_BAR_SIZE                     5               5               5               
         VF_LOG_BAR_SIZE                     0               0               0               
         NUM_PF_MSIX                         63              63              63              
         NUM_VF_MSIX                         11              11              11              
         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0)    AUTOMATIC(0)    AUTOMATIC(0)    
         PCIE_CREDIT_TOKEN_TIMEOUT           0               0               0               
         ACCURATE_TX_SCHEDULER               False(0)        False(0)        False(0)        
         PARTIAL_RESET_EN                    False(0)        False(0)        False(0)        
         RESET_WITH_HOST_ON_ERRORS           False(0)        False(0)        False(0)        
         PCI_DOWNSTREAM_PORT_OWNER           Array[0..15]    Array[0..15]    Array[0..15]    
         CQE_COMPRESSION                     BALANCED(0)     BALANCED(0)     BALANCED(0)     
         IP_OVER_VXLAN_EN                    False(0)        False(0)        False(0)        
         MKEY_BY_NAME                        False(0)        False(0)        False(0)        
         PRIO_TAG_REQUIRED_EN                False(0)        False(0)        False(0)        
         UCTX_EN                             True(1)         True(1)         True(1)         
         REAL_TIME_CLOCK_ENABLE              False(0)        False(0)        False(0)        
         RDMA_SELECTIVE_REPEAT_EN            False(0)        False(0)        False(0)        
         PCI_ATOMIC_MODE                     PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0) PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0) PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
         TUNNEL_ECN_COPY_DISABLE             False(0)        False(0)        False(0)        
         LRO_LOG_TIMEOUT0                    6               6               6               
         LRO_LOG_TIMEOUT1                    7               7               7               
         LRO_LOG_TIMEOUT2                    8               8               8               
         LRO_LOG_TIMEOUT3                    13              13              13              
         LOG_TX_PSN_WINDOW                   7               7               7               
         LOG_MAX_OUTSTANDING_WQE             7               7               7               
         TUNNEL_IP_PROTO_ENTROPY_DISABLE     False(0)        False(0)        False(0)        
         ICM_CACHE_MODE                      DEVICE_DEFAULT(0) DEVICE_DEFAULT(0) DEVICE_DEFAULT(0)
         TX_SCHEDULER_BURST                  0               0               0               
         LOG_DCR_HASH_TABLE_SIZE             11              11              11              
         DCR_LIFO_SIZE                       16384           16384           16384           
         ROCE_CC_PRIO_MASK_P1                255             255             255             
         ROCE_CC_PRIO_MASK_P2                255             255             255             
         CLAMP_TGT_RATE_AFTER_TIME_INC_P1    True(1)         True(1)         True(1)         
         CLAMP_TGT_RATE_P1                   False(0)        False(0)        False(0)        
         RPG_TIME_RESET_P1                   300             300             300             
         RPG_BYTE_RESET_P1                   32767           32767           32767           
         RPG_THRESHOLD_P1                    1               1               1               
         RPG_MAX_RATE_P1                     0               0               0               
         RPG_AI_RATE_P1                      5               5               5               
         RPG_HAI_RATE_P1                     50              50              50              
         RPG_GD_P1                           11              11              11              
         RPG_MIN_DEC_FAC_P1                  50              50              50              
         RPG_MIN_RATE_P1                     1               1               1               
         RATE_TO_SET_ON_FIRST_CNP_P1         0               0               0               
         DCE_TCP_G_P1                        1019            1019            1019            
         DCE_TCP_RTT_P1                      1               1               1               
         RATE_REDUCE_MONITOR_PERIOD_P1       4               4               4               
         INITIAL_ALPHA_VALUE_P1              1023            1023            1023            
         MIN_TIME_BETWEEN_CNPS_P1            4               4               4               
         CNP_802P_PRIO_P1                    6               6               6               
         CNP_DSCP_P1                         48              48              48              
         CLAMP_TGT_RATE_AFTER_TIME_INC_P2    True(1)         True(1)         True(1)         
         CLAMP_TGT_RATE_P2                   False(0)        False(0)        False(0)        
         RPG_TIME_RESET_P2                   300             300             300             
         RPG_BYTE_RESET_P2                   32767           32767           32767           
         RPG_THRESHOLD_P2                    1               1               1               
         RPG_MAX_RATE_P2                     0               0               0               
         RPG_AI_RATE_P2                      5               5               5               
         RPG_HAI_RATE_P2                     50              50              50              
         RPG_GD_P2                           11              11              11              
         RPG_MIN_DEC_FAC_P2                  50              50              50              
         RPG_MIN_RATE_P2                     1               1               1               
         RATE_TO_SET_ON_FIRST_CNP_P2         0               0               0               
         DCE_TCP_G_P2                        1019            1019            1019            
         DCE_TCP_RTT_P2                      1               1               1               
         RATE_REDUCE_MONITOR_PERIOD_P2       4               4               4               
         INITIAL_ALPHA_VALUE_P2              1023            1023            1023            
         MIN_TIME_BETWEEN_CNPS_P2            4               4               4               
         CNP_802P_PRIO_P2                    6               6               6               
         CNP_DSCP_P2                         48              48              48              
         LLDP_NB_DCBX_P1                     False(0)        False(0)        False(0)        
         LLDP_NB_RX_MODE_P1                  OFF(0)          OFF(0)          OFF(0)          
         LLDP_NB_TX_MODE_P1                  OFF(0)          OFF(0)          OFF(0)          
         LLDP_NB_DCBX_P2                     False(0)        False(0)        False(0)        
         LLDP_NB_RX_MODE_P2                  OFF(0)          OFF(0)          OFF(0)          
         LLDP_NB_TX_MODE_P2                  OFF(0)          OFF(0)          OFF(0)          
         DCBX_IEEE_P1                        True(1)         True(1)         True(1)         
         DCBX_CEE_P1                         True(1)         True(1)         True(1)         
         DCBX_WILLING_P1                     True(1)         True(1)         True(1)         
         DCBX_IEEE_P2                        True(1)         True(1)         True(1)         
         DCBX_CEE_P2                         True(1)         True(1)         True(1)         
         DCBX_WILLING_P2                     True(1)         True(1)         True(1)         
         KEEP_ETH_LINK_UP_P1                 True(1)         True(1)         True(1)         
         KEEP_IB_LINK_UP_P1                  False(0)        False(0)        False(0)        
         KEEP_LINK_UP_ON_BOOT_P1             False(0)        False(0)        False(0)        
         KEEP_LINK_UP_ON_STANDBY_P1          False(0)        False(0)        False(0)        
         DO_NOT_CLEAR_PORT_STATS_P1          False(0)        False(0)        False(0)        
         AUTO_POWER_SAVE_LINK_DOWN_P1        False(0)        False(0)        False(0)        
         KEEP_ETH_LINK_UP_P2                 True(1)         True(1)         True(1)         
         KEEP_IB_LINK_UP_P2                  False(0)        False(0)        False(0)        
         KEEP_LINK_UP_ON_BOOT_P2             False(0)        False(0)        False(0)        
         KEEP_LINK_UP_ON_STANDBY_P2          False(0)        False(0)        False(0)        
         DO_NOT_CLEAR_PORT_STATS_P2          False(0)        False(0)        False(0)        
         AUTO_POWER_SAVE_LINK_DOWN_P2        False(0)        False(0)        False(0)        
         NUM_OF_VL_P1                        _4_VLs(3)       _4_VLs(3)       _4_VLs(3)       
         NUM_OF_TC_P1                        _8_TCs(0)       _8_TCs(0)       _8_TCs(0)       
         NUM_OF_PFC_P1                       8               8               8               
         VL15_BUFFER_SIZE_P1                 0               0               0               
         NUM_OF_VL_P2                        _4_VLs(3)       _4_VLs(3)       _4_VLs(3)       
         NUM_OF_TC_P2                        _8_TCs(0)       _8_TCs(0)       _8_TCs(0)       
         NUM_OF_PFC_P2                       8               8               8               
         VL15_BUFFER_SIZE_P2                 0               0               0               
         DUP_MAC_ACTION_P1                   LAST_CFG(0)     LAST_CFG(0)     LAST_CFG(0)     
         UNKNOWN_UPLINK_MAC_FLOOD_P1         False(0)        False(0)        False(0)        
         SRIOV_IB_ROUTING_MODE_P1            LID(1)          LID(1)          LID(1)          
         IB_ROUTING_MODE_P1                  LID(1)          LID(1)          LID(1)          
         DUP_MAC_ACTION_P2                   LAST_CFG(0)     LAST_CFG(0)     LAST_CFG(0)     
         UNKNOWN_UPLINK_MAC_FLOOD_P2         False(0)        False(0)        False(0)        
         SRIOV_IB_ROUTING_MODE_P2            LID(1)          LID(1)          LID(1)          
         IB_ROUTING_MODE_P2                  LID(1)          LID(1)          LID(1)          
         PF_TOTAL_SF                         0               0               0               
         PF_SF_BAR_SIZE                      0               0               0               
         PF_NUM_PF_MSIX                      63              63              63              
*        ROCE_CONTROL                        ROCE_ENABLE(2)  DEVICE_DEFAULT(0) ROCE_ENABLE(2)  
         PCI_WR_ORDERING                     per_mkey(0)     per_mkey(0)     per_mkey(0)     
         MULTI_PORT_VHCA_EN                  False(0)        False(0)        False(0)        
         PORT_OWNER                          True(1)         True(1)         True(1)         
         ALLOW_RD_COUNTERS                   True(1)         True(1)         True(1)         
         RENEG_ON_CHANGE                     True(1)         True(1)         True(1)         
         TRACER_ENABLE                       True(1)         True(1)         True(1)         
         IP_VER                              IPv4(0)         IPv4(0)         IPv4(0)         
         BOOT_UNDI_NETWORK_WAIT              0               0               0               
         UEFI_HII_EN                         True(1)         True(1)         True(1)         
         BOOT_DBG_LOG                        False(0)        False(0)        False(0)        
         UEFI_LOGS                           DISABLED(0)     DISABLED(0)     DISABLED(0)     
         BOOT_VLAN                           1               1               1               
         LEGACY_BOOT_PROTOCOL                PXE(1)          PXE(1)          PXE(1)          
         BOOT_RETRY_CNT                      NONE(0)         NONE(0)         NONE(0)         
         BOOT_INTERRUPT_DIS                  False(0)        False(0)        False(0)        
         BOOT_LACP_DIS                       True(1)         True(1)         True(1)         
         BOOT_VLAN_EN                        False(0)        False(0)        False(0)        
         BOOT_PKEY                           0               0               0               
         P2P_ORDERING_MODE                   DEVICE_DEFAULT(0) DEVICE_DEFAULT(0) DEVICE_DEFAULT(0)
         DYNAMIC_VF_MSIX_TABLE               False(0)        False(0)        False(0)        
         EXP_ROM_UEFI_ARM_ENABLE             True(1)         True(1)         True(1)         
         EXP_ROM_UEFI_x86_ENABLE             True(1)         True(1)         True(1)         
         EXP_ROM_PXE_ENABLE                  True(1)         True(1)         True(1)         
         ADVANCED_PCI_SETTINGS               False(0)        False(0)        False(0)        
         SAFE_MODE_THRESHOLD                 10              10              10              
         SAFE_MODE_ENABLE                    True(1)         True(1)         True(1)         
The '*' shows parameters with next value different from default/current value.
, %!s(<nil>)
I0912 19:07:59.445320 1624833 mellanox_plugin.go:277] mellanox-plugin parseMstconfigOutput(): Attributes [NUM_OF_VFS SRIOV_EN LINK_TYPE_P1 LINK_TYPE_P2]
I0912 19:07:59.445397 1624833 mellanox_plugin.go:257] mellanox-plugin mlnxNicFromMap() map[NUM_OF_VFS:11 SRIOV_EN:True(1)]
I0912 19:07:59.445408 1624833 mellanox_plugin.go:306] mellanox-plugin getLinkType(): linkType 
W0912 19:07:59.445413 1624833 mellanox_plugin.go:315] mellanox-plugin getLinkType(): LINK_TYPE_P* attribute was not found
I0912 19:07:59.445417 1624833 mellanox_plugin.go:257] mellanox-plugin mlnxNicFromMap() map[NUM_OF_VFS:11 SRIOV_EN:True(1)]
I0912 19:07:59.445421 1624833 mellanox_plugin.go:306] mellanox-plugin getLinkType(): linkType 
W0912 19:07:59.445425 1624833 mellanox_plugin.go:315] mellanox-plugin getLinkType(): LINK_TYPE_P* attribute was not found
I0912 19:07:59.445428 1624833 mellanox_plugin.go:300] mellanox-plugin isDualPort(): pciAddress 0000:5e:00.0
I0912 19:07:59.445433 1624833 mellanox_plugin.go:341] mellanox-plugin getOtherPortSpec(): pciAddress 0000:5e:00.0
I0912 19:07:59.445438 1624833 mellanox_plugin.go:321] mellanox-plugin isLinkTypeRequireChange(): device 0000:5e:00.0
...

@adrianchiris
Copy link
Collaborator

mellanox-plugin getLinkType(): LINK_TYPE_P* attribute was not found

thats expected since ConnectX6 Lx is ethernet only

@SchSeba
Copy link
Collaborator

SchSeba commented Sep 13, 2022

/hold cancel

/lgtm

@SchSeba
Copy link
Collaborator

SchSeba commented Sep 13, 2022

@adrianchiris will this PR also close #317 ?

@github-actions github-actions bot added the lgtm label Sep 13, 2022
@SchSeba SchSeba removed the hold label Sep 13, 2022
@adrianchiris
Copy link
Collaborator

@SchSeba yep. seems like centos repos were updated.

@SchSeba SchSeba merged commit bdc6257 into k8snetworkplumbingwg:master Sep 13, 2022
@SchSeba SchSeba mentioned this pull request Sep 13, 2022
Closed
@wizhaoredhat wizhaoredhat deleted the add_Nvidia_mlx5_ConnectX-6_Lx_support branch March 14, 2023 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adrianchiris adrianchiris adrianchiris approved these changes

@SchSeba SchSeba SchSeba approved these changes

@e0ne e0ne e0ne approved these changes

Assignees
No one assigned
Labels
lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add NVIDIA Mellanox Technologies MT2894 Family Ethernet Controller ConnectX-6 Lx
5 participants
@wizhaoredhat @adrianchiris @coveralls @SchSeba @e0ne