net_buf: support large size #97440

MarkWangChinese · 2025-10-13T08:36:13Z

change uint16_t as size_t to support large size of data.

The user case:
In USB, the net_buf is used. usb msc may transfer 64KB (65536) data in one net_buf. So the uint16_t can't handle 65536.

include/zephyr/net_buf.h

pdgendt

Please see https://github.com/zephyrproject-rtos/zephyr/pull/97440/files#r2425645621
Besides that, the following functions/structs need to be updated too:

uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
struct net_buf_simple_state;
static inline void net_buf_simple_save(const struct net_buf_simple *buf,
				       struct net_buf_simple_state *state);
struct net_buf;

pdgendt · 2025-10-13T09:55:06Z

Make sure the tests in tests/lib/net_buf work with this new option, and that this is included in CI.

include/zephyr/net_buf.h

MarkWangChinese · 2025-10-14T02:16:27Z

Make sure the tests in tests/lib/net_buf work with this new option, and that this is included in CI.

Tested tests\lib\net_buf\buf on one NXP platform (lpc55s69), the result is OK as follow. I tried to test tests\lib\net_buf\buf_simple too, but it can't be built successfully.

===================================================================
PROJECT EXECUTION SUCCESSFUL
*** Booting Zephyr OS build v4.2.0-5562-g3a3a6a6c7e65 ***
Running TESTSUITE net_buf_tests
===================================================================
START - test_net_buf_1
 PASS - test_net_buf_1 in 0.001 seconds
===================================================================
START - test_net_buf_2
 PASS - test_net_buf_2 in 0.001 seconds
===================================================================
START - test_net_buf_3
 PASS - test_net_buf_3 in 0.001 seconds
===================================================================
START - test_net_buf_4
 PASS - test_net_buf_4 in 0.001 seconds
===================================================================
START - test_net_buf_big_buf
 PASS - test_net_buf_big_buf in 0.001 seconds
===================================================================
START - test_net_buf_byte_order
 PASS - test_net_buf_byte_order in 0.001 seconds
===================================================================
START - test_net_buf_clone_no_ref_count
 PASS - test_net_buf_clone_no_ref_count in 0.001 seconds
===================================================================
START - test_net_buf_clone_ref_count
 PASS - test_net_buf_clone_ref_count in 0.001 seconds
===================================================================
START - test_net_buf_clone_reference_counted_zero_sized_buffer
 PASS - test_net_buf_clone_reference_counted_zero_sized_buffer in 0.001 seconds
===================================================================
START - test_net_buf_clone_user_data
 PASS - test_net_buf_clone_user_data in 0.001 seconds
===================================================================
START - test_net_buf_comparison
 PASS - test_net_buf_comparison in 0.001 seconds
===================================================================
START - test_net_buf_fixed_append
 PASS - test_net_buf_fixed_append in 0.001 seconds
===================================================================
START - test_net_buf_fixed_pool
 PASS - test_net_buf_fixed_pool in 0.001 seconds
===================================================================
START - test_net_buf_linearize
 PASS - test_net_buf_linearize in 0.001 seconds
===================================================================
START - test_net_buf_multi_frags
 PASS - test_net_buf_multi_frags in 0.001 seconds
===================================================================
START - test_net_buf_user_data
 PASS - test_net_buf_user_data in 0.001 seconds
===================================================================
START - test_net_buf_user_data_copy
 PASS - test_net_buf_user_data_copy in 0.001 seconds
===================================================================
START - test_net_buf_var_pool
 PASS - test_net_buf_var_pool in 0.001 seconds
===================================================================
START - test_net_buf_var_pool_aligned
 PASS - test_net_buf_var_pool_aligned in 0.001 seconds
===================================================================
TESTSUITE net_buf_tests succeeded

------ TESTSUITE SUMMARY START ------

SUITE PASS - 100.00% [net_buf_tests]: pass = 19, fail = 0, skip = 0, total = 19 duration = 0.019 seconds
 - PASS - [net_buf_tests.test_net_buf_1] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_2] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_3] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_4] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_big_buf] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_byte_order] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_no_ref_count] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_ref_count] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_reference_counted_zero_sized_buffer] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_user_data] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_comparison] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_fixed_append] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_fixed_pool] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_linearize] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_multi_frags] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_user_data] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_user_data_copy] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_var_pool] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_var_pool_aligned] duration = 0.001 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION SUCCESSFUL

pdgendt · 2025-10-14T06:35:50Z

include/zephyr/net_buf.h


+#if defined(CONFIG_NET_BUF_LARGE_SIZE)
+/** @brief net buf size type */
+typedef size_t net_buf_size_t;


If we look at #96903, should we prefer uint32_t?

Not blocking, just a question.

Comment addressed

jfischer-no · 2025-10-14T11:05:42Z

include/zephyr/net_buf.h

+#if defined(CONFIG_NET_BUF_LARGE_SIZE)
+/** @brief net buf size type */
+typedef size_t net_buf_size_t;
+#else
+/** @brief net buf size type */
+typedef uint16_t net_buf_size_t;
+#endif
+


I do not think that it makes any sense to have it configurable, nor use size_t. This will introduce a lot of mess because no one would know what actual maximal size is. Either it remains uint16_t, or we change it to uint32_t. And USB in Zephyr for sure does not require it to be larger than uint16_t.

And USB in Zephyr for sure does not require it to be larger than uint16_t.

Could you explain more about it?
Windows will write/read 64KiB data to/from usb disk. If the buffer of msc is configured as 64KiB and give the 64KiB buffer to controller at one time, then the controller can transfer all the 64 KiB bytes without host stack and msc class involved.

I do not think that it makes any sense to have it configurable, nor use size_t. This will introduce a lot of mess because no one would know what actual maximal size is.

@jfischer-no I agree about not using size_t, since it needs to be more explicit than that, however no code should be using the struct member types for doing any "maximal size" derminations. net_buf offers separate APIs like net_buf_max_len(), net_buf_headroom() and net_buf_tailroom() to determine how much storage space is available.

jhedberg · 2025-10-14T11:57:27Z

include/zephyr/net_buf.h

-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
+net_buf_size_t net_buf_simple_max_len(const struct net_buf_simple *buf);


I think this is an oversight in the API. The public API has strived to keep the uint16_t as much of an internal detail as possible, while using size_t for both input and return parameters. I think this one should probably just be changed to size_t to be consistent with the rest of the API.

As for the point @jfischer-no raised regarding the predictability of what the buffer can actually contain, I agree that internal (within the structure) a fixed size type (e.g. uint32_t) makes sense.

Thanks @jhedberg , I do the changes, please help to review whether it is your thought.

change uint16_t as uint32_t to support large size of data. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

sonarqubecloud · 2025-10-16T09:01:00Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

jhedberg · 2025-10-16T09:19:24Z

include/zephyr/net_buf.h

-	uint16_t len;
+	uint32_t len;

 	/** Amount of data that net_buf_simple#__buf can store. */
-	uint16_t size;
+	uint32_t size;


I think you misunderstood me a little bit. I wasn't proposing to change this unconditionally to uint32_t, rather you'd need some explicit enabling of the larger size, through Kconfig, like you had before. Otherwise you're causing memory overflows to platforms what are very constrained (we have at least some build configurations for 16k platforms where even just a few bytes increase in RAM consumption will cause the build to fail).

jhedberg · 2025-10-16T09:20:01Z

include/zephyr/net_buf.h

 * @return Number of bytes usable behind the net_buf_simple::data pointer.
 */
-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
+size_t net_buf_simple_max_len(const struct net_buf_simple *buf);


This should be in a separate commit, with the commit message explaining that it's just aligning this with the rest of the net_buf APIs.

jhedberg · 2025-10-16T09:20:21Z

include/zephyr/net_buf.h

-	uint16_t offset;
+	uint32_t offset;
 	/** Length of data */
-	uint16_t len;
+	uint32_t len;


Same here. This should be build-time conditional.

Got it, I will add back the net_buf_size_t too. Do you have concern about using typedef? IMO, both are OK.

#if defined(CONFIG_NET_BUF_LARGE_SIZE) /** @brief net buf size type */ typedef uint32_t net_buf_size_t; #else /** @brief net buf size type */ typedef uint16_t net_buf_size_t; #endif ... net_buf_size_t len; ...

or

... #if defined(CONFIG_NET_BUF_LARGE_SIZE) uint16_t len; #else uint32_t len; #endif ...

Considering that you need to have the correct type in multiple structs and multiple member variables, I think the typedef would be the simplest option.

That said, I'd appreciate some more thoughts from @jfischer-no, since he seemed to be against this. @jfischer-no changing unconditionally to a larger size will not work without also finding a solution for our memory constrained platforms that can't currently take any additional memory hit wrt to net_buf sizes.

jhedberg · 2025-10-16T09:20:34Z

include/zephyr/net_buf.h

-			uint16_t len;
+			uint32_t len;

 			/** Amount of data that this buffer can store. */
-			uint16_t size;
+			uint32_t size;


And here.

Initially I think we should make the larger size opt in. We can then discuss if eventually it'd be opt-out, and then we make the memory-constrained configurations explicitly select the smaller size.

jhedberg · 2025-10-16T09:21:34Z

include/zephyr/net_buf.h

 * @return Number of bytes usable behind the net_buf::data pointer.
 */
-static inline uint16_t net_buf_max_len(const struct net_buf *buf)
+static inline size_t net_buf_max_len(const struct net_buf *buf)


Separate commit.

jhedberg · 2025-10-16T09:21:46Z

lib/net_buf/buf_simple.c

 }

-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf)
+size_t net_buf_simple_max_len(const struct net_buf_simple *buf)


Separate commit

pdgendt reviewed Oct 13, 2025

View reviewed changes

include/zephyr/net_buf.h Outdated Show resolved Hide resolved

MarkWangChinese marked this pull request as ready for review October 13, 2025 09:13

zephyrbot added the area: Networking Buffers net_buf/net_buf_simple API & implementation label Oct 13, 2025

zephyrbot requested review from jhedberg, jukkar and rlubos October 13, 2025 09:15

zephyrbot assigned jhedberg Oct 13, 2025

MarkWangChinese requested review from jfischer-no and tmon-nordic October 13, 2025 09:19

pdgendt previously requested changes Oct 13, 2025

View reviewed changes

tmon-nordic reviewed Oct 13, 2025

View reviewed changes

include/zephyr/net_buf.h Outdated Show resolved Hide resolved

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 3a3a6a6 to 938fd0f Compare October 14, 2025 02:12

zephyrbot added the area: Tests Issues related to a particular existing or missing test label Oct 14, 2025

zephyrbot requested a review from nashif October 14, 2025 02:14

jukkar previously approved these changes Oct 14, 2025

View reviewed changes

pdgendt reviewed Oct 14, 2025

View reviewed changes

pdgendt previously approved these changes Oct 14, 2025

View reviewed changes

jfischer-no requested changes Oct 14, 2025

View reviewed changes

jhedberg reviewed Oct 14, 2025

View reviewed changes

net_buf: support large size

e78e3dc

change uint16_t as uint32_t to support large size of data. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

MarkWangChinese dismissed stale reviews from pdgendt and jukkar via e78e3dc October 16, 2025 08:49

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 938fd0f to e78e3dc Compare October 16, 2025 08:49

jhedberg requested changes Oct 16, 2025

View reviewed changes

		uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
		net_buf_size_t net_buf_simple_max_len(const struct net_buf_simple *buf);

net_buf: support large size #97440

Are you sure you want to change the base?

net_buf: support large size #97440

Conversation

MarkWangChinese commented Oct 13, 2025

Uh oh!

Uh oh!

pdgendt left a comment

Choose a reason for hiding this comment

Uh oh!

pdgendt commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MarkWangChinese commented Oct 14, 2025

Uh oh!

pdgendt Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Oct 16, 2025

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarkWangChinese Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pdgendt commented Oct 13, 2025 •

edited

Loading

pdgendt Oct 14, 2025 •

edited

Loading

MarkWangChinese Oct 16, 2025 •

edited

Loading