gnrc/ipv6/nib: don't queue packets on 6lo neighbors and drop/flush if…

[derMihai]

Contribution description

This following fix only applies for CONFIG_GNRC_IPV6_NIB_QUEUE_PKT == 1 (default config).

This PR is an alternative solution for #20781:

There is currently the case that a on-link neighbor's packet queue might never be emptied. Specifically: if a neighbor is unreachable, the packets queued to be later send (i.e. once the neighbor is resolved) will never be de-queued if the neighbor is never resolved. On a typical link (e.g. ethernet), this is less of a problem, as the NIB subsystem actively tries to resolve the host, and in case of failure the neighbor entry is removed and the packets in it's queue get released. However, on e.g. 6LoWPANs, there is no active discovery going on, so the code path for removing the neighbor never gets executed.

This PR adds following changes:

• packets are not enqueued on 6lo neighbors, as this is not required by specification: https://www.rfc-editor.org/rfc/rfc6775.html#section-5.6
• once a neighbor's status switches to UNREACHABLE, flush it's packet queue
• for UNREACHABLE neighbors: drop packets instead of queuing
• neighbor packet queue length is capped

Issues/PRs references

#20781

[derMihai]

Not sure if we should drop with ENOBUFS or silently.

[riot-ci]

Murdock results

❌ FAILED

3733209 fixup! dropped per-neighbor packet queue

Success	Failures	Total	Runtime
58	0	9195	01m:32s

Artifacts

[fabian18]

SHOULD always be smaller than @ref CONFIG_GNRC_IPV6_NIB_NUMOF

Why should that be?

[derMihai]

Because if it's >= CONFIG_GNRC_IPV6_NIB_NUMOF, then even for a single neighbor, you can't even get to drop old packets from it's queue because you can't allocate a new one in the first place. With multiple neighbors, this can happen anyway, so this is why I went for a SHOULD instead of MUST.

[fabian18]

Ah

#if IS_ACTIVE(CONFIG_GNRC_IPV6_NIB_QUEUE_PKT)
static gnrc_pktqueue_t _queue_pool[CONFIG_GNRC_IPV6_NIB_NUMOF];
#endif  /* CONFIG_GNRC_IPV6_NIB_QUEUE_PKT */

That is not just an array of queues, but rather every packet to be queued has to get a slot here.

[fabian18]

If CONFIG_GNRC_IPV6_NIB_QUEUE_PKT_CAP is the queue capacity per neighbor.
I would define this as 1 by default and change the upper code snipped to:

#if IS_ACTIVE(CONFIG_GNRC_IPV6_NIB_QUEUE_PKT)
static gnrc_pktqueue_t _queue_pool[CONFIG_GNRC_IPV6_NIB_NUMOF * CONFIG_GNRC_IPV6_NIB_QUEUE_PKT_CAP];
#endif  /* CONFIG_GNRC_IPV6_NIB_QUEUE_PKT */

[fabian18]

Hm but it would be bad if packets cannot be queued even if there are 15 available slots because only one slot per neighbor is allowed. The current definition nevertheless looks a bit strange to me.

What about a neighbor is not allowed to have more than halve of the slots?

[derMihai]

Sure we still need space in pktbuf to actually send out the neighbor solicitation (and receive the response), but that isn't deepening on the number of queued packets but their size - or do I misunderstand something here?

I think this is a different problem. What I'm trying to solve is the scenario where a neighbor gets flooded with packets and we run out of free queue entries. In that case, the neighbor(s) queue(s) will be filled with stale packets because we can't even _alloc_queue_entry() in order to drop the oldest packets.

Then one neighbor could take all the slots and at some point _alloc_queue_entry() will always fail.

AFAIK we should always drop the oldest, but that can't be done if we can't allocate in the first place.

The current capping solution doesn't guarantee this scenario won't happen, just makes it less probable.

[derMihai]

Also note that the default number of free queue entries is CONFIG_GNRC_IPV6_NIB_NUMOF == 16, which is rather small.

[fabian18]

You could as well drop the oldest packet, as soon as allocation fails and try to allocate again.
It could be that 16 slots are held by one neighbor and another neighbor for which allocation fails does not have an oldest packet to drop, so allocation fails again because one neighbor holds all the slots.

With your capacity limit per neighbor you want to make this case less likely, but also do not prevent this case.
Yes the idea is good, but at the same time you accept that a packet is not queued even though a slot could be allocated.

If I got that right I am not sure if this is really beneficial as long as we don´t note an issue that packets cannot be queued because one neighbor is taking most of the queue slots.
Did you observe this case?

[derMihai]

If I got that right I am not sure if this is really beneficial as long as we don´t note an issue that packets cannot be queued because one neighbor is taking most of the queue slots.
Did you observe this case?

No, I only had issues with one host.

What bothers me most isn't failed allocations, but stale packets in a neighbor's queue. This goes against the first part of be strict when sending and tolerant when receiving.

It could be that 16 slots are held by one neighbor and another neighbor for which allocation fails does not have an oldest packet to drop, so allocation fails again because one neighbor holds all the slots.

This is partially true. We have the static _nib_onl_entry_t _nodes[CONFIG_GNRC_IPV6_NIB_NUMOF]; . We could iterate through that and drop either from the first neighbor (faster) or the one with the largest queue (may be slower for large CONFIG_GNRC_IPV6_NIB_NUMOF). Anyway, I don't see iterating through that list as a performance problem. We already do so when searching for free packets, adding one more run isn't changing much.

[derMihai]

I removed the per-neighbor queue cap. Packet allocation now never fails, it just pops a packet from the neighbor with the most packets in its queue. I made the queue entry count one more than the neighbor count. That way, there must always be a neighbor with at least two packets in it's queue, so we never leave it packet-less.

[benpicco]

that drop is a bit strange. Up to 16 entries, each entry can queue only a single packet, but once there are 17 entries, we can queue 16 per entry?

How about

Suggested change

	#define CONFIG_GNRC_IPV6_NIB_QUEUE_PKT_CAP (CONFIG_GNRC_IPV6_NIB_NUMOF > 16 ? 16 : 1)
	#define CONFIG_GNRC_IPV6_NIB_QUEUE_PKT_CAP DIV_ROUND_UP(CONFIG_GNRC_IPV6_NIB_NUMOF, 2)

[derMihai]

I guess it's fine, I don't know how probable is resolution of multiple neighbors at the same time.