Switchtec-nvme-cli

This project is a fork of the nvme-cli project with additional support for using NVMe drives in the Switchtec Gen4 PAX fabric.

Background

Switchtec Gen4 PAX is a variant of Microsemi's Switchtec PCIe switch product which enables sharing SR-IOV devices among multiple hosts. Common SR-IOV devices include NICs, HBAs, IB cards, etc. With the support of a Switchtec Gen4 PAX switch and this utility, an SR-IOV capable NVMe drive can be managed and configured by a host and then shared among multiple hosts.

By default, an NVMe drive in the PAX fabric cannot be enumerated by the host. Using the bind operation provided by the Switchtec Gen4 PAX switch, such a VF can be bound to and enumerated by the host.

Unlike some other SR-IOV devices such as NICs, NVMe drives need to be properly configured before they can be used. Generally, configuration includes creating a namespace, attaching the namespace to a specific VF (NVMe secondary controller), allocating VI and VQ resources for the VF, and bringing the VF online.

The configuration is done through Fabric Manager using MRPC commands. The MRPC interface is a way for a host to send configuration commands to the Switchtec firmware. Switchtec Gen4 PAX provides a special MRPC command to forward NVMe admin commands to the NVMe drives in the PAX fabric.

Switchtec-nvme-cli is responsible for creating the NVMe admin commands and delivering them to NVMe drives in the PAX fabric, using the special MRPC command. The special MRPC commands can be sent through one of these two channels: the inband PCIe channel, or the Ethernet channel. Switchtec-nvme-cli is currently based on v1.11.1 of the nvme-cli.

Refer to the Switchtec PAX device specification for further information on Switchtec PAX, including MRPC, Fabric Manager and other terminologies used in this document.

Features

Swichtec-nvme-cli supports all features of nvme-cli for drives connected directly to the host.

Swichtec-nvme-cli supports the following operations for NVMe drives behind PAX:

Listing all NVMe devices and namespaces
Creating a namespace
Deleting a namespace
Attaching a namespace to requested controller(s)
Deleting a namespace from the controller
Sending the NVMe Identify Controller command
Sending the NVMe Identify Namespace command, displaying structure
Sending the NVMe Identify List command, displaying structure
Sending the Virtualization Management command
Resetting a specific Virtual Function (VF)
Downloading a firmware image
Committing a firmware image
Sending a user-defined Admin command
Listing secondary controllers

Building and Installation

Switchtec-nvme-cli has the following dependencies:

libssl-dev
switchtec-user library
switchtec-kernel driver

Please build and install the switchtec-user library and switchtec-kernel driver before building this utility.

To build and install this utility, simply run the following commands:

#sudo make
#sudo make install

Examples

Here are some sample commands for managing NVMe drives in a PAX fabric. For the NVMe drives connected directly to the host, all commands from the original nvme-cli are supported without any change.

Fabric Manager over PCIe channel

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION

List controllers of an NVMe drive

#sudo ./switchtec-nvme list-ctrl 0x3300@/dev/switchtec0
[   0]:0x1
[   1]:0x2
[   2]:0x3
[   3]:0x4
[   4]:0x5
[   5]:0x6
[   6]:0x7
[   7]:0x8
[   8]:0x9
[   9]:0xa
[  10]:0xb
[  11]:0xc
[  12]:0xd
[  13]:0xe
[  14]:0xf
[  15]:0x21

List namespaces of an NVMe drive

#sudo ./switchtec-nvme list-ns 0x3300@/dev/switchtec0
[   0]:0x1
[   1]:0x2
[   2]:0x3
[   3]:0x4
[   4]:0x5

Create a 4GB namespace

#sudo ./switchtec-nvme  create-ns 0x3300@/dev/switchtec0 -c 1048576 -s 1048576 -f 2
create-ns: Success, created nsid:6

Attach a namespace to a controller

#sudo ./switchtec-nvme attach-ns 0x3300@/dev/switchtec0 -n 6 -c 1
attach-ns: Success, nsid:6

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             6           4.29  GB /   4.29  GB      4 KiB +  0 B   REVISION

Allocate VQ and VI resources for the controller

#sudo ./switchtec-nvme virt-mgmt 0x3300@/dev/switchtec0 -c1 -r0 -a8 -n2
success, Number of Resources allocated:0x2
#sudo ./switchtec-nvme virt-mgmt 0x3300@/dev/switchtec0 -c1 -r1 -a8 -n2
success, Number of Resources allocated:0x2

Reset the controller and then bring it online (note that 0x3300 is the ID for the primary controller, and 0x3301 is for the secondary controller 1)

#sudo ./switchtec-nvme switchtec vf-reset 0x3301@/dev/switchtec0 -f
#sudo ./switchtec-nvme virt-mgmt 0x3300@/dev/switchtec0 -c1 -r1 -a9
success, Number of Resources allocated:0

Now you can bind the controller to a PAX device and start using the namespace resource
Bring the controller offline

#sudo ./switchtec-nvme virt-mgmt 0x3300@/dev/switchtec0 -c1 -r1 -a7
success, Number of Resources allocated:0

Detach a namespace from a controller

#sudo ./switchtec-nvme detach-ns 0x3300@/dev/switchtec0 -n 6 -c 0x21
detach-ns: Success, nsid:6

Delete a namespace

#sudo ./switchtec-nvme delete-ns 0x3300@/dev/switchtec0 -n 6
delete-ns: Success, deleted nsid:6

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@/dev/switchtec0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION

Fabric Manager over Ethernet channel

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list -r 10.188.117.80:0
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION

List controllers of an NVMe drive

#sudo ./switchtec-nvme list-ctrl 0x3300@10.188.117.80:0
[   0]:0x1
[   1]:0x2
[   2]:0x3
[   3]:0x4
[   4]:0x5
[   5]:0x6
[   6]:0x7
[   7]:0x8
[   8]:0x9
[   9]:0xa
[  10]:0xb
[  11]:0xc
[  12]:0xd
[  13]:0xe
[  14]:0xf
[  15]:0x21

List namespaces of an NVMe drive

#sudo ./switchtec-nvme list-ns 0x3300@10.188.117.80:0
[   0]:0x1
[   1]:0x2
[   2]:0x3
[   3]:0x4
[   4]:0x5

Create a 4GB namespace

#sudo ./switchtec-nvme  create-ns 0x3300@10.188.117.80:0 -c 1048576 -s 1048576 -f 2
create-ns: Success, created nsid:6

Attach a namespace to a controller

#sudo ./switchtec-nvme attach-ns 0x3300@10.188.117.80:0 -n 6 -c 0x21
attach-ns: Success, nsid:6

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list -r 10.188.117.80:0
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             6           4.29  GB /   4.29  GB      4 KiB +  0 B   REVISION

Allocate VQ and VI resources for the controller

#sudo ./switchtec-nvme virt-mgmt 0x3300@10.188.117.80 -c1 -r0 -a8 -n2
success, Number of Resources allocated:0x2
#sudo ./switchtec-nvme virt-mgmt 0x3300@10.188.117.80 -c1 -r1 -a8 -n2
success, Number of Resources allocated:0x2

Reset the controller and then bring it online (note that 0x3300 is the ID for the primary controller, and 0x3301 is for the secondary controller 1)

#sudo ./switchtec-nvme switchtec vf-reset 0x3301@10.188.117.80 -f
#sudo ./switchtec-nvme virt-mgmt 0x3300@10.188.117.80 -c1 -r1 -a9
success, Number of Resources allocated:0

Now you can bind the controller to a PAX device and start using the namespace resource
Bring the controller offline

#sudo ./switchtec-nvme virt-mgmt 0x3300@10.188.117.80 -c1 -r1 -a7
success, Number of Resources allocated:0

Detach a namespace from a controller

#sudo ./switchtec-nvme detach-ns 0x3300@10.188.117.80:0 -n 6 -c 0x21
detach-ns: Success, nsid:6

Delete a namespace

#sudo ./switchtec-nvme delete-ns 0x3300@10.188.117.80:0 -n 6
delete-ns: Success, deleted nsid:6

List all NVMe devices and namespaces

#sudo ./switchtec-nvme switchtec list -r 10.188.117.80:0
Node                       SN                   Model                                    Namespace Usage                      Format           FW Rev
-------------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             1           6.55  MB /   6.55  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             2          13.11  MB /  13.11  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             3         131.07  MB / 131.07  MB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             4           2.15  GB /   2.15  GB    512   B +  0 B   REVISION
0x3300@10.188.117.80:0   SERIALNUMBER         VENDOR MODEL                             5           2.15  GB /   2.15  GB    512   B +  0 B   REVISION

Name		Name	Last commit message	Last commit date
Latest commit History 1,608 Commits
Documentation		Documentation
completions		completions
etc		etc
linux		linux
nvmf-autoconnect		nvmf-autoconnect
plugins		plugins
scripts		scripts
tests		tests
util		util
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
NVME-VERSION-GEN		NVME-VERSION-GEN
README.md		README.md
README_nvme_cli.md		README_nvme_cli.md
cmd.h		cmd.h
cmd_handler.h		cmd_handler.h
common.h		common.h
define_cmd.h		define_cmd.h
fabrics.c		fabrics.c
fabrics.h		fabrics.h
nvme-builtin.h		nvme-builtin.h
nvme-device.h		nvme-device.h
nvme-filters.c		nvme-filters.c
nvme-ioctl.c		nvme-ioctl.c
nvme-ioctl.h		nvme-ioctl.h
nvme-lightnvm.c		nvme-lightnvm.c
nvme-lightnvm.h		nvme-lightnvm.h
nvme-models.c		nvme-models.c
nvme-models.h		nvme-models.h
nvme-print.c		nvme-print.c
nvme-print.h		nvme-print.h
nvme-status.c		nvme-status.c
nvme-status.h		nvme-status.h
nvme-topology.c		nvme-topology.c
nvme.c		nvme.c
nvme.control.in		nvme.control.in
nvme.h		nvme.h
nvme.spec.in		nvme.spec.in
plugin.c		plugin.c
plugin.h		plugin.h
regress		regress

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Switchtec-nvme-cli

Background

Features

Building and Installation

Examples

Fabric Manager over PCIe channel

Fabric Manager over Ethernet channel

About

Releases 3

Packages

Contributors 120

Languages

License

Microsemi/switchtec-nvme-cli

Folders and files

Latest commit

History

Repository files navigation

Switchtec-nvme-cli

Background

Features

Building and Installation

Examples

Fabric Manager over PCIe channel

Fabric Manager over Ethernet channel

About

Resources

License

Stars

Watchers

Forks

Releases 3

Packages 0

Contributors 120

Languages

Packages