Snabb 2024.08 “Garmin”

What's Changed

• Remove historic Snabb programs by @eugeneia in #1525
• Merge RaptorJIT changes (Aug 2024) by @eugeneia in #1526

Full Changelog: v2024.06...v2024.08

Snabb 2024.06 “Faye”

This small release comes with some bug fixes and minor improvements. Notably, we accidentally introduced a performance regression when upgrading RaptorJIT in the previous release, which is fixed in #1523.

What's Changed

• apps.mellanox.connectx: alloc_pages(): remove assertion by @alexandergall in #1518
• Update HTTP/HTTPS IPFIX templates by @alexandergall in #1522
• Revert raptorjit to use bundled dlmalloc again by @eugeneia in #1523

Full Changelog: v2023.10...v2024.06

Snabb 2023.10 “Enigma”

This release pulls in recent improvements and bug fixes from LuaJIT upstream, refines engine pacing, adds a snabb config shutdown command, and adds support for ConnectX-7 NICs.

What's Changed

• apps.rss, apps.ipfix: add timeline events by @eugeneia in #1508
• Fix engine.Hz mode by @eugeneia in #1509
• apps.mellanox: add support for ConnectX-7 by @alexandergall in #1511
• apps.mellanox.connectx: add support for PCAM register by @alexandergall in #1512
• Collected changes for October 2023 release “Enigma” by @eugeneia in #1515
• Implement shutdown command for the config program by @alexandergall in #1517
• Timed pace breathing by @eugeneia in #1510
• Sync with raptorjit August-2023 by @eugeneia in #1514
• lib.hardware.pci: fix pcall in shutdown() by @alexandergall in #1516

Full Changelog: v2023.06...v2023.10

Snabb 2023.06 “Davion”

This small summer release contains various improvements for Snabbflow (snabb ipfix) and a bug fix for Snabb lwAFTR.

What's Changed

• rss: fix missing local declaration in BPF filter construction by @alexandergall in #1504
• Collected changes for June 2023 release: Davion by @eugeneia in #1507
• lib.ipv4.reassemble: fix missing argument in call to reassembly_error by @eugeneia in #1500
• apps.ipfix.maps: re-create map ctable if source file has changed by @alexandergall in #1501
• rss.metadata: store the metadate in the packet headroom area by @alexandergall in #1502
• ipfix: add acquire-cpu option to YANG schema by @alexandergall in #1503
• ipfix: add HTTPS templates to extract the TLS SNI by @alexandergall in #1506

Full Changelog: v2023.04...v2023.06

Snabb 2023.04 “Caspar”

This release finally merges @lukego's timeline: a probabilistic high-frequency event log that is useful to understand and debug runtime latencies. It can be enabled via the environment variable SNABB_TIMELINE_ENABLE. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Snabbflow integration (timeline, group freelist and interlink sizes) by @eugeneia in #1497
• Timeline 2022 by @eugeneia in #1486
• core.group_freelist: fix a deadlock by @eugeneia in #1494
• lib.poptrie: fix bug where address bits where interpreted in reverse order by @eugeneia in #1495
• apps.rss: remove 59 (no-next-header) from list of IPv6 extension headers by @alexandergall in #1496
• lib.yang.list: fix bug in remove_obsolete_nodes by @eugeneia in #1498
• Collected changes for April 2023 release: Caspar by @eugeneia in #1499
• [sketch] Timeline redux by @lukego in #1098

Full Changelog: v2022.12...v2023.04

Contributors

Alexander Gall (1):

• 8e7be8e: apps.rss.metadata: remove 59 (no-next-header) from list of IPv6 extension headers

Luke Gorrie (10):

• 3e4c447: dasm_x86.lua: Add support for RDTSCP instruction
• 95872b1: core.timeline: New module for "timeline" logs
• 930f294: Makefile: Embed "*.events" timeline specs
• aa9980a: engine: Add engine.events, app.events, link.events
• 8530942: core.link: Add dropped bytes counter
• 9b252f0: engine: Instrument with timeline events
• 40ceef6: core.timeline: Switch to double-float on disk
• 563ae7a: engine: update timeline level probabilities
• ffb7379: engine: Randomize timeline log level with math
• 16a6bb2: engine: Remove timeline packet payload sampling

Max Rottenkolber (51):

• 1bb4108: core.app: remove remains from 16a6bb2 (timeline packet sampling)
• 0e52b24: core.timeline: make timeline log available to other core modules
• e834845: core.packet: record packet allocation events to timeline
• a6def7b: Revert "core.timeline: make timeline log available to other core modules"
• 8201d1d: lib.ptree.worker: reformulate to use engine.main with done func
• debd26c: engine.main: fix memory leak due to latency histogram creation
• edaad6c: core.app: set timeline log level at the very end of breathe loop
• 9c4c804: core.packet: reorder group_fl released/reclaimed/unlocked events
• 004f5bb: core.timeline: decouple log level from event rate
• 8e68109: core.app: initialize timeline log rate to 7 (no logging)
• 8f0c6bf: core.timeline: add switch to disable timeline logging
• 30cafdc: core.app: add timeline events for app network configuration
• e5faba0: Fix some wrong assumptions about the event rate
• 872eaab: app.events: reduce tick/ticked event rate
• 0252254: engine.events: fix breath_ticked description
• 8080451: core.app: log breath_ticked only when tick()=true
• cd43d69: engine.events: increase rate of commited_counters
• 8091531: Revert "app.events: reduce tick/ticked event rate"
• 3e0fee7: core.app: fix bug introduced in merge with master
• cb11a26: app.events: reduce tick/ticked event rate
• bfa3aa6: lib.ptree: fix bugs introduced by merge with master
• 8c34646: timeline: cleaup merge
• 840e1e5: timeline: cleanup
• db3c5cb: engine: add timeline_enabled toggle (on by default)
• 7b00bcf: Revert "engine: add timeline_enabled toggle (on by default)"
• 9896b4b: timeline: disable by default, add lib.scheduling toggle
• 21e2017: timeline: rework group freelist events
• b7a8af9: timeline: remove unused arguments to tick events
• 068b4d5: timeline: remove linkstats for push events
• f41f7a1: timeline: fixup rework group freelist events
• bcca06e: core.sync: add load primitive
• 45d3ac7: core.group_freelist: fix a deadlock
• b916131: group_freelist: revert panic
• f8c977c: group_freelist: add deadlock detection
• 2fbc94b: lib.poptrie: fix bug where address bits where interpreted in reverse order
• 67b3d9c: lib.poptrie: optimize x86 lookup
• a382b1d: core.timeline: fix selftest
• 00287bb: engine: allow timely timeline initialization
• a6fe6dc: snabb ipfix probe: add --timeline option to CLI
• 66f0eb3: lib.scheduling: add group_freelist_size setting
• f906776: lib.interlink: fix shm printing bug
• 50cad00: ipfix probe: make group freelist and interlink sizes configurable via schema
• 81a9e19: timeline: enable via environment variable
• f805e81: Revert "snabb ipfix probe: add --timeline option to CLI"
• 0916c24: Revert "engine: allow timely timeline initialization"
• 559c80d: core.sync: load64, cas64
• f47228b: core.group_freelist: use 64 bit sequence
• b6c130d: core.group_freelist: ensure finish is atomic
• e8eb4fd: Revert "group_freelist: add deadlock detection"
• f3905b3: core.group_freelist: cleanup
• 7472b83: lib.yang.list: fix bug in remove_obsolete_nodes

Snabb 2022.12 “Balaclava”

This release introduces a new data structure to represent YANG lists, allowing us to expose lists with the attribute ordered-by: user as a configuration interface. Additionally we have made previously hardcoded freelist and interlink buffer sizes configurable at runtime to enable tuning of those buffers to fit application demands. Finally, ipfix probe has been optimized to operate without software RSS when the configuration allows it. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Configurable interlink queue, freelist, and group freelist sizes by @eugeneia in #1489
• Add lib.yang.list hash trie to store YANG lists by @eugeneia in #1490
• ipfix probe: cleanup code, optimize operation without software rss by @eugeneia in #1492
• Collected changes for December 2022 release: Balaclava by @eugeneia in #1493

Full Changelog: v2022.10...v2022.12

Contributors

Max Rottenkolber (100):

• 4b58a3c: heap 1
• 536c929: heap 2
• 9ea357e: list 1
• 1afa4cd: heap no overflow, simplify, fix bugs
• c325cd9: heap collisions
• 62d4ae4: remove, iterator
• e451f5e: enable LUA52COMPAT
• 00a6169: fix bugs
• 0f8580e: ListMeta
• 2075954: add_or_update
• 2313cc0: fix heap _ref bug
• 559faf3: refactor
• b3692f5: refactor
• b60b918: ip bench
• 9c6a29e: refactor
• a6492c0: funny
• d8e53fe: ListMeta fixes
• cc7b972: Make keys/members specs extendable
• 89a67ac: todo: deterministic struct field order
• ddabaa3: lib.yang.list: support optional members
• 64331e4: lib.yang.list: sort fields in build_type
• 62696b7: lib.yang.list: more builtin types
• 95a9d31: lib.yang.list: add lvalue type (for arbitrary Lua values)
• abc0f72: lib.yang.list: add ipv4/ipv6-address types
• 147557f: lib.yang.list: single key meta, object() getter
• 8e1da16: lib.yang.list: load/save
• e36d84e: lib.yang.list: object() functions a is-list? predicate
• 4c90e38: lib.yang.list: add from_table constructor
• 4eca2e7: core.shm: pass through extra sizeof args in map
• f948fa6: lib.interlink: make queue size configurable
• f1577c9: core.shm: allow mapping of partial objects
• c7bb164: core.group_freelist: configurable group freelist size
• 7b6ee58: core.packet: configurable freelist size
• 1ac0362: core.packet: fix freelist shm display bugs
• 07dc242: lib.interlink: revert from typeof to cdef for struct interlink
• 9400c62: lib.scheduling: add max_packets parameter
• 690bd3e: lib.yang: integrate list into data
• 5d20d5a: lib.yang.list: clean up lvalues, load, save
• d728c32: lib.yang: integrate list into binary
• 86a8a32: lib.yang.list: fix bug wrt lvalues/optional
• 96ce235: lib.yang.binary: test list lvalues and fix bug
• dbc6f5e: lib.yang.list: add optional lvalue test case
• c5609e9: lib.yang.data: cleanup
• d5e508a: lib.yang: update README
• 1db6430: lib.yang.path_data: wip
• a66ee30: lib.yang.path_data: wip
• de2acbc: lib.yang.path: cleanup
• c3423ec: libya.ng.path_data: wip
• 63dde84: lib.yang.path: fix bugs
• d267f22: lib.yang.list: add identityref and leafref types
• 275da2a: lib.yang.path_data: wip
• 97a5899: lib.yang.path: add unparse_path
• ee4af97: lib.yang.path_data: wip passing
• adb40ef: lib.yang.path_data: fix bugs, extend tests
• e8c9725: lib.yang.list: integrate state, ptree support
• 6ff9f33: lib.yang.list: add struct support
• ab96b5d: list lwaftr wip
• 00ba121: lib.lua.StackTracePlus: pull upstream
• 5eebd6f: core.main: re-enable StackTracePlus
• 4bf08f8: lib.yang: factor out lib.yang.ctype
• bb9eca2: lib.yang.list: use lib.yang.ctype
• bc571d3: apps.lwaftr.binding_table: fixup
• f3c8e18: lib.yang.binary: serialize struct ts for list
• 8c8c25c: program.lwaftr.tests.probased.genyang: integrate
• a3c0e39: core.main: fallback to debug.traceback when STP fails
• 27a5ec9: integrate lwaftr wip fixup
• 990ab59: lib.yang.list: fix empty iterator bug
• 3cd57c2: lib.yang.state: fixup
• c764412: lib.yang.path/path_data: handle rpc schema
• 231381d: lib.yang: only ever handle list objects for list productions
• 34c4324: lib.yang.path_data: add grammar_for_schema_by_name
• 4fc54c2: integrate lwaftr wip
• 7ae62e7: lib.yang.list: use ctype if available
• 1145275: integrate lwaftr fix bugs
• 264bd6a: lib.equal: do not compare tables with metatables
• ad5c3a5: lwaftr integrate fix bugs
• c2261f2: lib.yang.data: test ordered-by
• 3858999: lib.yang.list: fix ordering bugs
• 308e56c: lib.yang.path_data: fix setter and remover bugs
• b04a261: lib.yang.list: fix bug in field_order
• 04c79b6: lib.yang.data: avoid redundantly optional list members
• 0059c0b: lib.yang.state: muffle warnings
• 4e56332: lwaftr: remove legacy schema and migrations
• b0ea121: lwaftr: supply mandatory padding value in native_binding_table_from_ietf
• 5bd3087: Integrate snabbflow
• 1896a2c: snabb ipfix: remove class order leaf
• 78f1a11: lib.yang.binary: support serializing leafrefs
• 68cd6c4: snabb-snabbflow-v1.yang: use leafrefs where sensible
• 1553134: lib.ptree: fix a bug where bogus PID was logged
• 5923add: ipfix probe: cleanup code, optimize operation without software rss
• 48a1747: program.ipfix.tests.test: tweak tolerance
• 004b6b8: lib.yang.binary: bumb version
• 052d8c2: lib.interlink: fix queue size race condition
• 37628fc: core.shm: test parameterized types
• acf5d80: apps.lwaftr.binding_table: keep ctable defaults originally used in lib.yang.data
• 84d59cd: lib.interlink: move size to same cache line as state
• 0183beb: lib.interlink: keep local copies of mask
• 3b3c3de: core.group_freelist: keep local copies of mask
• 176b126: lib.interlink: edits
• 58be8ee: core.group_freelist: edits

Snabb 2022.10 “Alpina”

This release introduces a YANG schema for the multi-core scalable IPFIX probe developed by @alexandergall at SWITCH, “Snabbflow”. Additionally we have written a tutorial for getting up and running with Snabb lwAFTR. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Fix lwAFTR trace performance issues (for release Alpina) by @eugeneia in #1488
• WIP: YANG schema for snabb ipfix (“Snabbflow”) by @eugeneia in #1481
• lwAFTR 100G related fixes by @eugeneia in #1482
• Tutorial: Getting Started with Snabb lwAFTR by @eugeneia in #1483
• apps.mellanox: use default enable_counters setting by @eugeneia in #1484
• core.app: fix a bug related to getting app zone name by @eugeneia in #1485
• Collected changes for October 2022 release: Alpina by @eugeneia in #1487

Full Changelog: v2022.07...v2022.10

Contributors

Alexander Gall (3):

• 101911a: interlink: fix shmem names in receiver/transmitter shutdown
• 56cdc1d: Adapt ipfix stats program to work with probe_ptree
• 89a218f: map_pci_memory(): set file path, increase timeout

Max Rottenkolber (97):

• a03d3c5: lib.yang: support empty leaf-lists
• 0ffd14c: lib.yang: fix bug where native number keys caused parsing to c(l)tables
• c501f78: lib.yang: fix bug where value_ctype was used in native_key tables
• 2f86ddf: Revert "lib.yang: fix bug where value_ctype was used in native_key tables"
• 402ecdf: lib.ptree.action_codec: support toplevel app classes and nil app args
• bbbea95: program.ipfix.lib: factor pure configure functions
• 507173f: sketch: ipfix probe_ptree
• ea636f4: apps.tap: continuously scan MAC and add overwrite_src_mac option
• 8a5d3e2: apps.tap: add forwarding option
• 11bbb49: program.ipfix: use Tap app features for setting exporter src MAC and sysctls
• c6ab421: App apps.snmp.iftable
• e3f4d77: program.ipfix: use apps.snmp to setup ifmibs
• 4bbd11e: lib.yang.data: do not derive bogus ctype for leaf-lists
• 247761a: lib.cpuset: fix assertion in add, add contains/remove/list methods
• d42c38f: lib.ptree: scan for 'pciaddress' field in acquire_cpu_for_worker
• 6cf6e2f: program.ipfix.probe_ptree: pin to cpus specified in config
• 8d04cd2: ipfix probe_ptree: update example config
• 069264f: ipfix probe_ptree: start with name
• c814c43: lib.ptree: add rudimentary worker-restarting facility
• f3fd7e7: ipfix probe_ptree: use lib.ptree worker restarts
• a757a8e: ipfix probe_ptree: log_level=INFO, rename schema variable
• 0996502: lib.ptree: manage worker restarts per worker configued via worker_opts
• 97bba23: ipfix probe_ptree: only restart dedicated exporter processes
• 2d61cfb: apps.tap: overwrite_dst_mac not overwrite_src_mac
• d585c2f: ljsyscall: fix bug in S.sysctl where we wrote at position ~= 0
• ab14ac3: ipfix probe_ptree: canonicalize singular list names in schema/config
• 88312f5: ipfoc probe_ptree: more schema cleanup
• 42a58ee: ipfix: probe_ptree: [wip] schema documentation
• 3a1055e: core.app: setvmprofile in tick
• aec36e6: ipfix probe_ptree: remove use-maps from schema
• 5490392: ipfix probe_ptree: remove instance weight leaf from schema
• 2fe881b: ipfix probe_ptree: add scan-protection/enable leaf
• 6635812: ipfix probe_tree: raise exporter MTU default to 1500
• 8169068: ipfix probe_tree: remove add-packet-metadata option
• 9a5d18e: ipfix probe_tree: more schema documentation
• c3ce608: ipfix probe_tree: allow maximum ConnectX recvq_size in schema
• 3630cf7: core.lib: equal: check object identity early
• 4571335: apps.ipfix: allow full reconfig while keeping flow sets intact
• 749ce28: ipfix probe_ptree: support configs without software RSS classes
• 7d7b0ed: apps.ipfix: fix bugs in counter initialization
• cfc5095: lwaftr: fix bug in migrate_device_on_config
• 192a0a3: lwaftr: tutorial wip
• b1d7eb9: lwaftr: tutorial wip 2
• 285f3f3: lwaftr: tutorial wip3
• b969b9d: lwaftr tutorial: flowchart
• a3a8029: lwaftr tutorial: edit lwaftr-veth-env.sh
• c76a58f: lib.ptree.inotify: turn warnings into debug messages behind flag
• 8e94ad8: lwaftr tutorial: supporting programs
• a5523ea: lib.ptree: support scalar value app configs in action_codec
• ff74f68: ipfix probe_ptree: test wip
• 4fea4ea: Revert "ipfix probe_ptree: test wip"
• ebc7413: ipfix probe_ptree: add state counters
• 3e5ac19: wip
• a416c42: wip
• b23fe45: apps.ipfix: assert templates shm doesn’t already exist
• 413cf19: wip
• 680a28b: wip get.state
• b26d015: lib.ptree: add acquire_cpu worker option
• ef3a002: ipfix probe_ptree: do not acquire cpu core for mellanox control worker
• 6eb81db: lib.scheduling/lib.ptree.worker: make jit.opt configurable
• 00112e1: ipfix probe_ptree: run workers with custom jit options and engine.busywait=false
• de3a225: ipfix probe_ptree: key rss groups / instances in snabbflow-state with id instead of pid
• 02dcd33: ipfix probe_ptree: device queue stats for rss groups
• acf52dd: ipfix probe_ptree: fix bug introduced in de3a225
• 1fd8e51: apps.connectx: fix stop() bugs
• f346467: ipfix probe_ptree: normalize device, app, and input link names
• 29dfa83: ipfix probe_ptree: guard against duplicate devices
• 5187a9e: ipfix probe_ptree: hide debug prints
• 89581cf: apps.mellanox.connectx: fix bug in IO:stop()
• 83b9f1e: apps.mellanox: use default enable_counters setting
• e2890f4: apps.mellanox: add test case for rxdrop and per-queue rxdrop counters
• c0a4963: core.app: fix a bug related to getting app zone name
• 00d2337: ipfix probe_ptree: warn on missing traffic class, throw error on missing exporter
• 61b67a3: Replace ipfix probe with probe_tree, remove probe_rss
• 22db66a: ipfix probe: add common scheduling CLI opts
• e930e4e: ipfix probe: enable VMProfile by default
• 7228010: apps.mellanox: make sync interval configurable, add :report()
• 5feed3d: packetblaster: support connectx, increase send desciptors in loadgen mode
• bfa68ba: program.config: do not exit on error in call_leader
• da48397: Add program/ipfix/tests/bench.snabb
• 81bd389: apps.basic: fix bug in Truncate
• 1ef5bee: packetblaster ipfix: generate flows from pcap on multiple cpu cores
• 2aae05b: ipfix probe: update bench.snabb to use packetblaster ipfix
• f2870ac: program/ipfix/tests: check in truncate_pcap script
• d64e23c: ipfix probe: add --rss-goups option to bench.snabb
• 97e9369: program.ipfix.lib: remove dead code
• c508b40: ipfix probe: add test.snabb
• 7c5ee4e: ipfix probe: update collector-test.sh
• a8c2f24: ipfix probe: remove generate_packets.py
• cdf1c3b: ipfix probe: cleanup tests
• 86e14ed: ipfix probe: update README.md
• 609716d: lwaftr tutorial: fix typos, reference schema instead of source
• 70785be: lwaftr tutorial: remove TODOs
• 5a4d0b1: apps.ipfix: fix bug in selftest
• 18d2631: snabb config listen: fix bug introduced in bfa68ba
• 5fe89b5: apps.lwaftr: do not flush lookups early
• 4706127: lib.ctable/lib.hash.siphash: optimize traces

Snabb 2022.07 “Parmigiano”

This release upstreams the multi-core scalable IPFIX probe developed by @alexandergall at SWITCH, “Snabbflow”. Additionally, we have extended the core.app interface to support the new tick and per-link push methods. Other notable changes include improved inter-process link scaling, and a new benchmark for our Connect-X driver to be used in CI. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Upstream ipfix / snabbflow into max-next by @eugeneia in #1479
• Use MPMC bounded queue for group freelist by @eugeneia in #1472
• core.app 3.0: tick method, advanced app linking features by @eugeneia in #1475
• map_pci_memory: workaround for kernel race condition by @eugeneia in #1476
• lwaftr bench: build pcaps in generate-configuration, print summary by @eugeneia in #1477
• Collected changes for July 2022 release: Parmigiano by @eugeneia in #1478
• apps.mellanox.connectx: benchmark matrix by @eugeneia in #1469
• ConnectX: various driver enhancements by @eugeneia in #1473
• default.nix: remove obsolete brute-force shebang patching (breaks wit… by @eugeneia in #1474
• Fixes for lwAFTR in many-worker configurations by @eugeneia in #1480

Full Changelog: v2022.01.1...v2022.07

Contributors

Alexander Gall (105):

• 0f5068d: lj_trace.c: unpatch blacklisted bytecodes when flushing traces
• 5df67e8: lj_trace.c: Add penalty to trace abort message
• b19d442: intel_mp: fix range of array registers for i210
• 51214ea: intel_mp: add sanity checks for the number of rx/tx descriptors
• d4a9871: intel_mp: add queue enable counters, fix rxdrop counter
• fe709ed: lib.protocol.ipv4: avoid allocation for pseudo-header
• d15cbff: socket.raw: don't use select()
• 54780c2: ctable: fix power-of-2 issue
• 3002614: Add optional callback to the CTable:resize() method
• 0cda3d1: lib: disable tail-call optimization for endian conversions
• 62d0e72: Avoid factor-of-2 headroom in ctable
• 4c15c4e: Check bound of values in lib/lpm/lpm4_248.lua
• d4f8914: lib.lpm4_248: implement search in pure Lua
• 30b1e6f: Flush JIT traces after ctable resize
• d59aaff: Export flow before resetting fields
• 5ec2a64: Properly count records for IPFIX and NetFlow v9
• f4e93cd: Send full-sized export packets
• 556bc58: Get flow octet counter from IP header
• 246655c: Add tcpControlBits to flow sets, factorize transport processing
• 132f5b9: Allow multiple files as input for make_ipfix_element_map()
• e2587ae: Delegate exporting packets to FlowSets
• f40c1b1: Add counters
• 605dac0: Add sub-program to print stats for running ipfix processes
• a25d353: Add take_all() method to token bucket
• e0d93bc: Make table scan time configurable
• 30b7c4f: Support VLAN-tagged traffic, IPv6 extension headers
• b36e45b: Support PEN, strings and octetArray in templates
• d72d42d: Add optional state variables to flow cache value
• 46263a7: Add optional template-specific stats counters
• 57b71a0: Refactor template definitions
• 045f48e: Add templates for HTTP
• 2575d94: Add templates for DNS
• 9e6d48b: Add options to ipfix probe program, support routed tap interface
• 463022d: Add extended template
• 7f6c241: Switch to packet-metadata module from RSS
• 24a5063: Allow multiple IPFIX instances in the same process
• 8178577: Add probe_rss ipfix sub-program
• 84b486a: Disable map logging in extended template by default
• 7a9e3c8: Add SNMP Interface MIB for PCI devices
• 5b9e56e: program.ipfix: redesign RSS framework
• a15f4f3: program.ipfix: support hardware RSS in probe
• 2c05b4c: program.ipfix: fix stats sub-program for redesign
• 95ff0ca: program.ipfix: integrate mellanox driver
• 5151c96: program.ipfix: include PCI stats from control app
• 560e5b1: program.ipfix: increase receive buffers for mellanox driver
• a9e9f7f: program.ipfix: adapt to interlink semantics of v2018.06
• 28cca93: apps.ipfix: tune log messages
• 84cbf0d: apps.ipfix: fix selftests
• a2097f3: core.app: skip array indices when traversing input links
• 395a4f8: core.app: assert that links are connected only once
• 2c7929c: core.app: support link-specific push methods
• 565d7e6: apps.basic: apply link push methods to basic apps
• ea076d8: VlanMux: optimize tag pop/push
• bc36ddb: VlanMux: apply specialized push methods
• 1cfadb3: core.app: only call push() if the link is not empty
• 5fb9816: Add housekeeping() method to apps
• 233c894: Adapt to new versions of logger and token bucket
• be1f65c: apps.vhost: suppress -Wstringop-truncation
• 33eefac: lib.hardware.pci: add new device to supported Mellanox cards
• b655643: apps.ipfix, apps.rss: adapt to specialized push link semantics
• 40373df: program.iptix.probe_rss: add JIT configuration for RSS workers
• 14e32eb: apps.rss: mitigate unbiased branches in metadata handling
• c113670: apps.rss: explicitely unroll the loop over packet queues
• 744d6d5: apps.rss.rss: Preserve order in classes_active, fix explicite loop unrolling
• 3647611: program.ipfix.probe_rss: make receive queue size configurable for mellanox
• 72f5482: program.ipfix.stats: use global counters when per-queue stats are not available
• c24d55c: lib.interlink: increase queue size to allow bursts for ipfix
• b279790: Make be1f65c conditional for GCC >= 8
• 9bfdaae: program.ipfix: add IF MIB for routed tap interfaces
• 670c411: apps.rss.rss: use specialized function to copy addresses for hashing
• 54409f8: core.packet: increase maximum number of allocated packets
• a1780b3: Fix transitions in CXQ state machine
• 90c5762: map_pci_memory: wait for kernel driver to release the device
• 87c1a0e: apps.mellanox.connectx4: unbind device in IO:new()
• 08f25ca: apps.intel_mp.intel_mp: fix device unbinding for Intel driver
• b47b38d: apps.intel_mp.intel_mp: remove disable bus master cleanup
• 8f1a480: apps.ipfix.ipfix: move record expiration to housekeeping
• da76c64: apps.mellanox.connectx4: fix mcast counters
• 5f05dab: lib.hardware.pci.map_pci_memory: only wait if first call failed
• 15ef82b: program.ipfix.probe_rss: use pools to assign collector address and port
• afbe2bd: apps.rss.rss: Add VLAN match capability
• 6d5bafa: program.ipfix: increase logging rate
• eb659ba: program.ipfix.probe_rss: add observation domain to log output
• 58ce70e: program.ipfix: disable date in log messages by default, fix READMEs
• 432f38a: Add scan-protection feature
• 541c0d8: interlink: fix shmem names in receiver/transmitter shutdown
• b9c499e: program.ipfix.probe_rss: restart dead workers
• 429fa21: program.ipfix: make maximum load-factors of hash tables configurable
• 276a26d: ipfix: restrict flow-rate tracker to signature flows
• aafd550: program.ipfix.probe_rss: scale scan-protection rates with number of processes
• 3620245: program.ipfix.probe_rss: add CPU pool and NUMA binding
• bc9211f: Untabify
• 5a47fde: program.ipfix.probe_rss: make CPU-pinning configurable
• 1f082b8: program.ipfix.probe_rss: increase logging rate limit for master process
• 4af5345: program.ipfix: rename ConnectX4 -> ConnectX
• 4296bfa: Don't export empty flows
• fc407c2: Use proper flow start time after active timeout
• caa11e1: lib.ctable: fix hugepages allocation size
• 64a62f2: apps.mellanox: Make RX/TX flow control configurable, default to off/off
• 4c1571f: apps.mellanox: Add per-queue drop stats
• 69a15b9: apps.mellanox: Use lib.parse in ConnectX:new() and IO:new()
• 3a5b5ae: apps.mellanox: Use consumer counter in cxq to simplify ownership handling
• be7a343: connectx.lua: Add basic event handling
• 7c269fa: map_pci_memory: wait for kernel driver to release the device
• 1c7497c: lib.hardware.pci.map_pci_memory: only wait if first call failed

Andy Wingo (2):

• 907ce81: Use binary search for all lookups
• b457b6e: Add a type cache, in case that's important

Luke Gorrie (5):

• b29666f: lj_trace.c: Clear penalty slot after successful trace
• b4060ef: lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
• 0a90764: fixup! lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
• c685d8b: lj_trace.c: Reset hotcount table after flush
• 5443940: lj_record.c: Relax heuristic for root trace meeting JIT loop

Max Rottenkolber (76):

• bd5d4ec: wip
• 2f6b983: wip 2
• e301b53: mellanox benchmark numa
• aaea69b: add Source software benchmark to rule it out as a bottleneck
• 7634573: measure rx at sink
• 7f967e4: apps.mellanox: packetblaster
• 9adad52: mellanox benchmark: use packetblaster
• 350aea9: apps.mellanox.connectx: cleanup and optimization
• 3d86feb: mellanox benchmark: use stats counters instead of querying HCA
• ae5121b: mellanox benchmark: cleanup output
• b1cf848: mellanox benchmarks example scripts
• 87207cf: apps.connectx: remove dead code from packetblaster
• 0ab555f: mellanox benchmark: add --opt option to supply driver config options
• 312b740: mellanox benchmark: add forwarding mode
• 28f249a: mellanox benchmark: add --nmacs, --nvlans paremetrs, remove source-sofftest
• 4fa1d01: mellanox benchmark: fix up previous commit
• 6c69c04: mellanox benchmark: add sink mode, sink halts
• 7195667: mellanox benchmark: IMIX source
• f799af2: apps.interlink: add instrumented latency benchmark
• fbc13ab: core.packet: factor packet reclaimation from group_fl into reclaim_ste...

Snabb 2022.01.1 “Octarina”

Fix up .version for Octarina release.

See Snabb 2022.01 “Octarina”

Snabb 2022.01 “Octarina”

This release contains the a new driver for Mellanox Connect-X series NICs, an improved Intel AVF driver with support for RSS, and an experimental XDP driver. As usual this release also brings many smaller improvements and bug fixes.

What's Changed

• Mellanox ConnectX-4 Driver (WIP) by @capr in #918
• Completed Mellanox initialization sequence by @lukego in #943
• Upstream lwAFTR 2022.01.x by @eugeneia in #1465
• Fixes for regressions in max-next headed for Octarina release by @eugeneia in #1466
• Collected changes for Jan 2022 release: Octarina by @eugeneia in #1463
• Snabb 2022.01 “Octarina” by @eugeneia in #1467
• XDP app: driver for Linux AF_XDP sockets by @eugeneia in #1450
• intel_avf: fix integration breakage and free resources on shutdown (stop()) by @eugeneia in #1451
• Various YANG fixes related to choice statements and the consistency checker by @eugeneia in #1452
• Add filter option to XDP app for punting packets to Linux by @eugeneia in #1454
• lib.numa: gracefully handle failure to read CPU performance governor by @eugeneia in #1455
• lib.ptree: cleanup obsolete aggregated stats by @eugeneia in #1456
• arch/checksum: fix bug in carry propagation by @eugeneia in #1457
• lib.ctable: fix hugepages allocation size by @alexandergall in #1459
• core.packet: fix account_free physical capacity formula by @eugeneia in #1460
• Driver for Mellanox Connect-X 4/5/6 cards by @eugeneia in #1461
• Updated Intel AVF driver with VLAN+RSS support by @eugeneia in #1462
• Tracing JIT stability improvements from lwAFTR by @eugeneia in #1464

Full Changelog: v2019.11...v2022.01

Contributors

Alexander Gall (24):

• 48f6080: Cherry-picked apps.melanox from lukego/debug100g
• 4474000: Uncommited changes from lukego
• 580cf9b: connectx4.lua: minor changes
• 10ceea4: connectx4.lua: fix alignment of HCA command queue
• 1b9cf2b: connectx4.lua: fix RSS
• 8332671: Register ConnectX5 device in lib.hardware.pci
• 263919a: connectx4.lua: identify driver for device_info
• d57c67a: connectx4.lua: add statistics counters
• 22af847: connectx4.lua: use async commands to read stats counters
• a949245: connectx4.lua: use a factory to create HCAs
• b4d9252: connectx4.lua: fix bug with async commands when debug_hexdump is enabled
• e2eecff: connectx4.lua: fix page sizes in CREATE_{C,R}Q
• 76c0c80: apps.vhost: suppress -Wstringop-truncation
• 56884a7: apps.mellanox.connectx4: add MT28800 as supported device
• a0e8585: apps.mellanox.connectx4: add per-queue drop counters for ConnectX5
• fc37be4: apps.mellanox.connectx4: unbind device in IO:new()
• adc7d3d: apps.mellanox.connectx4: fix mcast counters
• 2219748: Fix transitions in CXQ state machine
• 6cdcb40: apps.mellanox: free CXQ during process shutdown
• c579524: apps.mellanox: use correct WQE stride in RX queues
• 556e427: mellanox: complete renaming ConnectX4 -> ConnectX
• a348899: lib.ctable: fix hugepages allocation size
• a86c61e: connectex.lua: extend RSS hashing to non-TCP/UDP IP packets
• 5f10b3b: connectx.lua: convert tabs to spaces

Andy Wingo (5):

• 9a787b5: Fix snabb top interface view for multiple queues
• 7a0c6c8: Disable vmprofile on lwAFTR by default
• 648c67a: Add 2019.06.02 changelog entry
• 6524dec: fix typo
• 828fa4f: lwAFTR version 2019.06.02

Cosmin Apreutesei (6):

• 45d0696: stub module and readme
• 8720a60: pci init
• cd3a103: vendor/dev id
• dfbd705: firmware revision register
• 34929f1: print firmware revision maj and min
• fe8065f: init procedure until QUERY_HCA_CAP

Luke Gorrie (24):

• a6973da: lib.hardware.pci: Add reset_device()
• ccdf38e: mellanox: Added text/hexdump trace of Linux driver
• 276f4fa: connectx4: Progress on initialization routine
• 6a426bd: connectx4: Translate virtual to physical on demand
• f349c41: core.memory: Add 'align' argument to dma_alloc()
• 8c55f66: connectx4: Request alignment on memory.dma_alloc()
• 2c1a92d: connectx4: Add multi-mailbox command support
• 6abbd96: connectx4: Successful QUERY_HCA_CAP query
• 92ec410: connectx4: Command mailbox chaining fixes
• 13e6550: connectx4: Set debug = false
• cd8e30e: connectx4: Remove stray print statements
• a285999: connectx4: Move checkstatus() into post()
• 4575dc7: connectx4: Device initialization fixes
• 8c0be9d: connectx4: Comment on ommitted SET_DRIVER_VERSION
• f196ac7: connectx4: Added CREATE_EQ (event queue) command
• c1cee02: connectx4: Added initial VPORT commands
• 51d4d45: connectx4: Give 'lock' argument to map_pci_memory
• 7659eb6: connectx4: Clean driver initialization
• 21d0dc3: connectx4: Refactored for multiprocess operation
• f08da3c: connectx4: IO app now has only 1xSQ and 1xRQ
• ffe4a22: connectx4: Add 'mtu' paramter (default 9500)
• 06c43f0: connectx4: Cleaner (simplistic) selftest method
• e367b6b: connectx4: Fix recv completion queue wrap-around
• 42c2196: mellanox: Rename ConnectX4 -> ConnectX (because it works with 5+ too)

Max Rottenkolber (150):

• 77e548c: Squashed 'lib/ljsyscall/' changes from d78235c..e587f8c55
• 09b29d9: ljsyscall: add some XDP constants
• 3f82010: ljsyscall: fix a bug in getsockopt when querying for structs
• 1e176cb: ljsyscall: lookup strflags in linux bpf(2) wrappers
• 573a29b: ljsyscall: add BPF_MAP type XSKMAP
• 63f0cfe: apps.test.synth: add packets option
• f535525: apps.test.synth: emit all given packets
• 6174563: core.packet: expose constants for packet alignment and headroom
• 984d4a4: apps.xdp: driver for AF_XDP sockets
• e056fa1: Remove src/xdp.snabb (accidentally checked in xdp proof of concept)
• ee4e42d: core.packet: expose free_internal
• b677292: apps.xdp: fix free stats accounting, cleanup fill/completion mngmnt
• 1b3593f: apps.xdp: add duplex test case, fix random_v4_packets
• 22d754c: apps.xdp: cleanup bpf_ld_imm64/LD_MAP_FD construction
• 4e31856: apps.xdp: generate more descriptive errors on bind(2) failure
• e53f314: apps.xdp: fixup comment about engine breath stimulation
• d225e75: apps.xdp: make num_chunks configurable in snabb_enable_xdp(opt)
• d44fb1d: apps.xdp: do not leak packets in XDP:stop()
• f8cb80c: apps.xdp: munmap(2) rings in XDP:stop() (do not leak rings mappings)
• e06c82f: apps.intel_avf: use new PCI acccess API (fix integration breakage)
• 7722f0d: apps.intel_avf: free resources on shutdown (fix leak)
• 6c330ec: lj_syscall: add BPF_OBJ_* syscall wrappers, extend bpf_attr union
• ea955a8: apps.xdp: allow sharing interfaces between Snabb processes
• 01f11e9: pflua: do not cdef struct bpf_ins (collides with ljsyscall)
• ae35674: core.app: add clearvmprofiles() function
• 1124b80: lib.timers.ingress_drop_monitor: clear vmprofiles on flush
• 8f2a216: lib.timers.ingress_drop_monitor: reset last_value on drop-free period
• 63968f6: raptorjit: clear all hotcounts every second
• dfadac8: raptorjit: amend prev. commit, also clear trace exit hotcounts
• b7c50cb: raptorjit: amend prev. commit, do not clear SNAPCOUNT_DONE
• bd84224: apps.xdp: expose driver reference
• 1a9eba5: lib.scheduling: add enable_xdp option
• 18cc365: apps.xdp: kick tx queue if not empty
• 19072b4: lib.yang.data: reorder translated members before printing
• ae4f734: snabb config: use consistency checker in command parsing
• 05b495e: lib.yang.path_data: expand choices in consistency checker
• e96ae1a: lib.yang.data: rework 19072b4 (remove order parameter) add tests
• 95d3a74: lib.yang: remove leftover debug print
• 3fdf926: pflua: fix typo bug in native codegen (xor/or emission)
• 07acce2: pflua: make regalloc architecture independent
• cd4d773: apps.xdp.bpf: add basic disassembler, endiannes opcodes
• 48ff71c: apps.xdp.pf_ebpf_codegen: add eBPF backend for pflua
• c158ac8: apps.xdp: add filter option (offload non-matching packets to kernel)
• 6b5851a: lib.numa: gracefully handle failure to read CPU performance governor
• 16c99ed: lib.ptree: cleanup obsolete aggregated stats
• 0068df6: arch/checksum: fix bug in carry propagation
• 3461a09: apps.xdp: only run rxtx_match selftest if SNABB_XDP_NQUEUES=1
• 00cac36: apps.xdp: debug stop()
• 8d4ee13: apps.xdp: mark stop() NYI / not supported
• 2bcc644: lwaftr --xdp sketch
• dca3004: snabb-softwire-v2: add default for leaf error-rate-limiting/packets
• 8c2000b: apps/ipv4,6/reassemble,fragment: handle padded packets
• 561b52a: lwaftr/setup: fix interfaces for bump-in-the-wire Interfaces used in bump-in-the-wire configurations were swapped, possibly during a prior refactor. Undo this.
• 1cdc635: snabb-softwire-v2.yang: documentation edits
• 6bcce09: lib.numa: gracefully handle failure to read CPU performance governor
• 191581a: apps.lwaftr.ndp: send NS via solicited node multicast
• 4817ae8: lwaftr: support vlans in XDP mode
• 004587b: lwaftr: automate linux interface setup for XDP
• bce8579: snabb-softwire-v2: allow more than two queues
• 909dca7: lib.protocol.ethernet: add ptoi used by connectx driver
• 6bca98b: apps.mellanox.connectx: fixup lib.hardware.pci usage
• 832ea9d: apps.mellanox.connectx...