Collected changes for Jan 2022 release: Octarina #1463
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Partial implementation of the initialization procedure.
Complete debug messages have become a little overwhelming now that we are allocating thousands of pages of memory for the adapter. Just for the moment disabling the hexdumps is the more sensible default. More fine-grained debug logging is likely needed.
Refactored the error checking to always be done when posting a command to the command queue. Previously this was a manual step for each command and that seems more error prone.
Now successfully:
- Providing boot memory to the adapter (6 pages)
- Querying adapter capabilities (current and maximum)
- Setting adapter capabilities (keep current)
- Providing init memory to the adapter (4232 pages !)
The output from the init sequence looks like this:
TRACE Read the initialization segment
TRACE Write the physical location of the command queues to the init segment.
TRACE Wait for the 'initializing' field to clear
fw_rev 14 12 1220
cmd_interface_rev 5
cmdq_phy_addr cdata<void *>: 0x1f000000
log_cmdq_size 5
log_cmdq_stride 6
ready true
nic_interface_supported true
internal_timer 2.0108995831647e+14
health_syndrome 0
Command: ENABLE_HCA
Command: QUERY_ISSI
cur_issi = 0
sup_issi =
01
Command: QUERY_PAGES
query_pages'boot' 6
Command: MANAGE_PAGES
Command: QUERY_HCA_CAP
Command: QUERY_HCA_CAP
Capabilities - current and (maximum):
eth_net_offloads = 0 (0)
end_pad = 1 (1)
cq_eq_remap = 1 (1)
device_frequency_mhz = 275 (275)
log_max_vlan_list = 12 (12)
log_min_stride_sz_rq = 0 (0)
log_max_klm_list_size = 16 (16)
log_max_rqt = 0 (0)
log_max_l2_table = 16 (16)
log_max_current_uc_list = 10 (10)
log_min_stride_sz_sq = 0 (0)
log_uar_page_sz = 0 (8)
log_max_wq_sz = 0 (0)
log_max_current_mc_list = 14 (14)
log_max_msg = 30 (30)
log_max_stride_sz_rq = 0 (0)
max_flow_counter = 0 (0)
log_max_eq_sz = 22 (22)
log_max_rqt_size = 0 (0)
basic_cyclic_rcv_wqe = 0 (0)
cache_line_128byte = 0 (0)
max_tc = 0 (0)
cmdif_checksum = 0 (3)
driver_version = 0 (0)
log_max_tis = 0 (0)
port_type = 1 (1)
wq_signature = 1 (1)
log_max_tir = 0 (0)
max_indirection = 4 (4)
log_max_rq = 0 (0)
cq_resize = 1 (1)
cq_oi = 1 (1)
cq_moderation = 1 (1)
log_max_pd = 24 (24)
log_max_mkey = 24 (24)
log_max_transport_domain = 0 (0)
rc = 1 (1)
num_ports = 1 (1)
bf = 1 (1)
vport_counters = 1 (1)
log_max_eq = 8 (8)
pad_tx_eth_packet = 0 (0)
log_pg_sz = 12 (12)
uar_sz = 5 (5)
cq_period_start_from_cqe = 1 (1)
uc = 1 (1)
log_max_mrw_sz = 64 (64)
log_max_cq = 24 (24)
vport_group_manager = 1 (1)
log_max_tis_per_sq = 0 (0)
start_pad = 0 (0)
log_max_cq_sz = 22 (22)
nic_flow_table = 0 (0)
scqe_break_moderation = 1 (1)
ud = 1 (1)
log_max_sq = 0 (0)
cqe_version = 0 (0)
log_bf_reg_size = 9 (9)
sctr_data_cqe = 1 (1)
log_max_rmp = 0 (0)
cqe_version = 0 (0)
log_bf_reg_size = 9 (9)
sctr_data_cqe = 1 (1)
log_max_rmp = 0 (0)
log_max_stride_sz_sq = 0 (0)
imaicl = 0 (0)
xrc = 1 (1)
Command: SET_HCA_CAP
Command: QUERY_PAGES
query_pages'init' 4232
Command: MANAGE_PAGES
Command: INIT_HCA
Pulls in more of the initialization procedure. See especially commit 4575dc7.
Added QUERY_VPORT_STATE, MODIFY_VPORT_STATE, QUERY_NIC_VPORT_CONTEXT. Note: I am not sure that these commands are actually needed since we are not using SR-IOV. The PRM mandates using some VPORT commands but I don't see them in the trace from the Linux mlx5 driver. So we may be able to remove this code.
Completed Mellanox initialization sequence
Required argument for new code merged from master in v2016.06. Request exclusive lock on the device.
This commit introduces a clean and working version of the device initialization.
The Mellanox driver is now split into two apps: 'ConnectX4' to setup the NIC and 'IO' to attach to a queue pair.
Previously a single IO app instances could be supported with multiple send and receive queues. The intention was to share traffic amongst the queues in order to overcome NIC hardware limitations on per-queue packet rate. The problem with that approach is that splitting traffic across queues potentially changes the relative ordering of packets and this potentially creates problems for the user. Better to avoid "DWIM" and have the user arrange multiple IO apps in an application-appropriate way if indeed they do need to overcome the ~16Mpps per-queue limit on the ConnectX-4 (100G).
Now also explicitly sets promiscuous mode on the ethernet port.
The selftest function, while still very simplistic, now presents its
output more neatly and checks that counter values match expectations.
The test now uses unicast traffic (need add additional tests to check
multicast and broadcast and ...).
Example output:
selftest: waiting for both links up
Links up. Sending 10,000,000 packets.
hardware counter 02:00.0 82:00.0
---------------- -------------------- --------------------
rx_bcast_octets 0 0
rx_bcast_packets 0 0
rx_error_octets 0 0
rx_error_packets 0 0
rx_mcast_octets 0 0
rx_mcast_packets 0 0
rx_ucast_octets 1,000,000,000 1,000,000,000
rx_ucast_packets 10,000,000 10,000,000
tx_bcast_octets 0 0
tx_bcast_packets 0 0
tx_error_octets 0 0
tx_error_packets 0 0
tx_mcast_octets 0 0
tx_mcast_packets 0 0
tx_ucast_octets 1,000,000,000 1,000,000,000
tx_ucast_packets 10,000,000 10,000,000
selftest: ok
Enabled "overflow ignore" on the receive completion queue so that we do not need to acknowledge completion entries. This should be safe because the completion queue and the receive queue are the same size i.e. each receive queue entry will have a separate completion entry with no collisions.
Fix the opcode for DISABLE_HCA (currently unused), add assert to alloc_pages() method.
Add the same set of counters supported by the Intel driver. Add a pull() method to the ConnextX4 object to periodically sync the stats so the "control" app can be run as a regular Snabb app.
Polling for command completion in sync_stats() creates too much latency and causes packet drops. This is mitigated by creating separtate HCAs for each command that reads a statistics register and perfom polling for completion only in sync_stats().
...and multiple switch() configs
# Conflicts: # src/apps/xdp/xdp.lua
(for new ICMP, ARP, and NDP counters)
eugeneia
changed the title
lwAFTR introduced a way to run applications with vmprofile disabled, in a way that branched on engine.vmprofile_enabled on the fast path. This patch cleans this up to only disable the profiler timer, and run setvmprofile unconditionally in both cases. Also removes the duplicate/unused/ineffectual profile option from lib.ptree.worker (profiling can be disabled via lib.scheduling).
Fixes for regressions in max-next headed for Octarina release
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR to collect changes to be included in the upcoming Octarina release.
Included: