out-of-memory in allocator.c

[Rrooach]

Bug report

Required Info:

• Operating System:
  Ubuntu 22.04
• Installation type:
  source code
• Version or commit hash:
  humble
• DDS implementation:
  FastRTPS
• Client library (if applicable):
  RCLCPP

Steps to reproduce issue

Basically, I installed ros-humble on a remote server, compiled it with address sanitizer, And when I start to boot the ros2 system and try to testing it, I get the following asan report.

Expected behavior

The system start without error.

Actual behavior

System emit an asan report

Additional information

the following is the asan report:
https://pastebin.com/5DgN1X74

it seems that the program fails to allocate a large amount of memory region when serialize message, in allocator.c:

static void *
__default_reallocate(void * pointer, size_t size, void * state)
{
  RCUTILS_CAN_RETURN_WITH_ERROR_OF(NULL);

  RCUTILS_UNUSED(state);
  return realloc(pointer, size);
}

the following is the gdb output:

https://pastebin.com/NtjqZBtP

It would be so nice of you to review this bug report. Thank you again for taking the time. Any suggestions or feedback you can provide would be helpful.

[fujitatomoya]

this one of the bad_alloc test cases.

#3  0x00007ffff6f33dce in rclcpp::SerializedMessage::reserve (this=0x7ffffffefe60,
    capacity=18446744073709551615)
    at /data/ros2_humble/src/ros2/rclcpp/rclcpp/src/rclcpp/serialized_message.cpp:151

https://github.com/ros2/rclcpp/blob/9284d7cefa9e37ce3b6926a44e6177d6e4a4b62a/rclcpp/test/rclcpp/test_serialized_message.cpp#L150-L152

[clalancette]

After looking at this, I'm going to suggest that we just remove the test completely. Passing -1 there is invalid (it gets promoted to a size_t anyway), so this isn't really valid.

[fujitatomoya]

closing, in favor of ros2/rclcpp#2330