Skip to content

Conversation

@masaori335
Copy link
Contributor

@masaori335 masaori335 commented May 20, 2024

Address #5165.

Add a option (128) for SO_INCOMING_CPU to the proxy.config.net.sock_option_flag_in.

Background

There was a bug with SO_REUSEPORT since v4.6. But it's fixed by v6.2 and some distros backported it.
Now we can use SO_INCOMING_CPU, SO_REUSEPORT and CPU Affinity 4 for perfect locality.

torvalds/linux@b261eda

Ref.

    SO_INCOMING_CPU (gettable since Linux 3.19, settable since Linux 4.4)

          Sets or gets the CPU affinity of a socket.  Expects an
          integer flag.

              int cpu = 1;
              setsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu,
                         sizeof(cpu));

          Because all of the packets for a single stream (i.e., all
          packets for the same 4-tuple) arrive on the single RX
          queue that is associated with a particular CPU, the
          typical use case is to employ one listening process per RX
          queue, with the incoming flow being handled by a listener
          on the same CPU that is handling the RX queue.  This
          provides optimal NUMA behavior and keeps CPU caches hot.

https://www.man7.org/linux/man-pages/man7/socket.7.html

@masaori335 masaori335 self-assigned this May 20, 2024
@masaori335 masaori335 added this to the 10.1.0 milestone May 20, 2024
@masaori335 masaori335 changed the title Add SO_INCOMING_CPU support on net.sock_option_flag_in Add SO_INCOMING_CPU support to net.sock_option_flag_in May 20, 2024
@masaori335 masaori335 force-pushed the asf-master-so-incoming-cpu branch from 1c27b6e to 32217f8 Compare May 20, 2024 07:28
EThread *ethread = this_ethread();

#ifdef TS_USE_HWLOC
cpu = ethread->hwloc_obj->os_index;
Copy link
Contributor Author

@masaori335 masaori335 May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hwloc_obj::os_index looks "OS-provided physical index number" and it's working with exec_thread.affinity: 4 as expected. However, hwloc has any better API?
https://github.com/open-mpi/hwloc/blob/63a8288d31a1baf67a909466aba9a022c78ca7b1/include/hwloc.h#L491

@bryancall bryancall requested review from bneradt and bryancall May 20, 2024 22:11
@masaori335 masaori335 force-pushed the asf-master-so-incoming-cpu branch from 13e9467 to 348f03d Compare May 27, 2024 01:42
@masaori335 masaori335 merged commit 266b666 into apache:master Jun 9, 2024
@masaori335
Copy link
Contributor Author

@cmcfarlen I'd request to 10.0.x backport. But, this is a new feature, so we can wait for 10.1.x, if you want.

@cmcfarlen
Copy link
Contributor

Cherry-picked to v10.0.x

cmcfarlen pushed a commit that referenced this pull request Jun 10, 2024
* Add SO_INCOMING_CPU support to net.sock_option_flag_in

* Fix FreeBSD Build

(cherry picked from commit 266b666)
masaori335 added a commit to masaori335/trafficserver that referenced this pull request Dec 10, 2024
…e#779)

* Add SO_INCOMING_CPU support to net.sock_option_flag_in (apache#11366)

* Add SO_INCOMING_CPU support to net.sock_option_flag_in

* Fix FreeBSD Build

(cherry picked from commit 266b666)

Conflicts:
	include/iocore/net/NetVCOptions.h
	iocore/eventsystem/I_EThread.h
	iocore/net/Connection.cc

* Fix building without HWLOC (apache#11435)

Fixes apache#11434.

(cherry picked from commit 44a41cb)

---------

Co-authored-by: Masaori Koshiba <masaori@apache.org>
Co-authored-by: JosiahWI <41302989+JosiahWI@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: picked-10.0.0

Development

Successfully merging this pull request may close these issues.

3 participants