Add standard linux utils such as bash to ros-image-core

[Ryanf55]

Purpose

• ROS requires standard tools beyond what is supplied by busybox
• The ROS setup.sh script relies on bash internally still

Necessary Tests

• Source the ROS bash environment, and make sure less errors are reported.

Documentation Needed

• Augment the standard release testing process to check that the ROS 2 CLI works reasonably well.

References

• Create packagegroups or images based on REP 2001 #1068 (comment)
• humble-kirkstone Wiki Creation Questions #1093 (comment)

Backport

This should be backported to all maintained branches including kirkstone-next in support of the humble-kirkstone wiki

[Ryanf55]

As it stands, the recommendation on packages to add results in a build failure. When builtin-interfaces tries to find python3 using find_package, it fails. I'll try to chase this down.

ryan@B650-970:~/Dev/yocto_ws/poky/build$ bitbake ros-image-core
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (09:29:00.272521)
Loading cache: 100% |#######################################################################################################################################################################################################| Time: 0:00:01
Loaded 5956 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "2.7.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.3+snapshot-7f9dd77eaeaf586c88ee1b5f4f5ee0439612b964"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
DISTRO_NAME          = "Poky (Yocto Project Reference Distro)"
ROS_DISTRO           = "humble"
ROS_VERSION          = "2"
ROS_PYTHON_VERSION   = "3"
meta                 
meta-poky            
meta-yocto-bsp       = "master:7f9dd77eaeaf586c88ee1b5f4f5ee0439612b964"
meta-oe              
meta-python          = "master:9953ca1ac0bf5747aaa664d25a6c06f7f0ad2446"
meta-ros-common      
meta-ros2            
meta-ros2-humble     = "add-standard-linux-utils-to-core:8e11e0277630e4a17eca2d3216e2d5cd1c22ecea"

Sstate summary: Wanted 1938 Local 0 Mirrors 0 Missed 1938 Current 2070 (0% match, 51% complete)#########################################################################################################                    | ETA:  0:00:00
Initialising tasks: 100% |##################################################################################################################################################################################################| Time: 0:00:02
NOTE: Executing Tasks
WARNING: rcl-yaml-param-parser-5.3.5-1-r0 do_package_qa: QA Issue: File /usr/share/rcl_yaml_param_parser/cmake/rcl_yaml_param_parserExport.cmake in package rcl-yaml-param-parser-dev contains reference to TMPDIR [buildpaths]
ERROR: builtin-interfaces-1.2.1-1-r0 do_configure: ExecutionError('/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/temp/run.do_configure.2736277', 1, None, None)
ERROR: Logfile of failure stored in: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/temp/log.do_configure.2736277
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/fastcdr/fastcdr_1.0.24-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/python-cmake-module/python-cmake-module_0.10.0-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-defaults/rosidl-default-runtime_1.2.0-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport-fastrtps/fastrtps-cmake-module_2.2.1-1.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport-fastrtps/rosidl-typesupport-fastrtps-c_2.2.1-1.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport-fastrtps/rosidl-typesupport-fastrtps-cpp_2.2.1-1.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport/rosidl-typesupport-c_2.0.1-1.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport/rosidl-typesupport-cpp_2.0.1-1.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl/rosidl-generator-cpp_3.1.5-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl/rosidl-runtime-c_3.1.5-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl/rosidl-runtime-cpp_3.1.5-2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/recipes-devtools/cmake/cmake-native_3.22.3.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/poky/meta/recipes-core/glibc/glibc_2.38.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/gcc/gcc-cross_13.2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/gcc/gcc-runtime_13.2.bb:do_populate_sysroot', '/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros-common/recipes-devtools/python/python3-lark-parser_0.7.0.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/ament-cmake/ament-cmake_1.3.5-1.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-defaults/rosidl-default-generators_1.2.0-2.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport-fastrtps/rosidl-typesupport-fastrtps-c_2.2.1-1.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl-typesupport-fastrtps/rosidl-typesupport-fastrtps-cpp_2.2.1-1.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl/rosidl-adapter_3.1.5-2.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/meta-ros/meta-ros2-humble/generated-recipes/rosidl/rosidl-parser_3.1.5-2.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/ninja/ninja_1.11.1.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/python/python3-numpy_1.26.3.bb:do_populate_sysroot', 'virtual:native:/home/ryan/Dev/yocto_ws/poky/meta/recipes-devtools/python/python3_3.12.1.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['ament-index-python', 'fastcdr', 'python-cmake-module', 'rcpputils', 'rcutils', 'rmw', 'rosidl-default-runtime', 'rosidl-generator-py', 'fastrtps-cmake-module', 'rosidl-typesupport-fastrtps-c', 'rosidl-typesupport-fastrtps-cpp', 'rosidl-typesupport-c', 'rosidl-typesupport-cpp', 'rosidl-cli', 'rosidl-cmake', 'rosidl-generator-c', 'rosidl-generator-cpp', 'rosidl-runtime-c', 'rosidl-runtime-cpp', 'rosidl-typesupport-interface', 'rosidl-typesupport-introspection-c', 'rosidl-typesupport-introspection-cpp', 'cmake-native', 'openssl', 'expat', 'gettext-minimal-native', 'glibc', 'libxcrypt', 'ncurses', 'util-linux-libuuid', 'zlib', 'binutils-cross-x86_64', 'gcc-cross-x86_64', 'gcc-runtime', 'libgcc', 'libedit', 'libtool-native', 'm4-native', 'opkg-utils', 'python3-numpy', 'python3', 'quilt-native', 'bzip2', 'libnsl2', 'libtirpc', 'texinfo-dummy-native', 'xz', 'linux-libc-headers', 'gdbm', 'libffi', 'sqlite3', 'googletest-native', 'python3-argcomplete-native', 'python3-empy-native', 'python3-lark-parser-native', 'python3-catkin-pkg-native', 'ament-cmake-ros-native', 'domain-coordinator-native', 'ament-cmake-core-native', 'ament-cmake-export-definitions-native', 'ament-cmake-export-dependencies-native', 'ament-cmake-export-include-directories-native', 'ament-cmake-export-interfaces-native', 'ament-cmake-export-libraries-native', 'ament-cmake-export-link-flags-native', 'ament-cmake-export-targets-native', 'ament-cmake-gen-version-h-native', 'ament-cmake-gmock-native', 'ament-cmake-gtest-native', 'ament-cmake-include-directories-native', 'ament-cmake-libraries-native', 'ament-cmake-pytest-native', 'ament-cmake-python-native', 'ament-cmake-target-dependencies-native', 'ament-cmake-test-native', 'ament-cmake-version-native', 'ament-cmake-native', 'ament-index-python-native', 'ament-package-native', 'fastcdr-native', 'gmock-vendor-native', 'gtest-vendor-native', 'python-cmake-module-native', 'rcpputils-native', 'rcutils-native', 'rmw-native', 'rosidl-default-generators-native', 'rosidl-generator-py-native', 'fastrtps-cmake-module-native', 'rosidl-typesupport-fastrtps-c-native', 'rosidl-typesupport-fastrtps-cpp-native', 'rosidl-typesupport-c-native', 'rosidl-typesupport-cpp-native', 'rosidl-adapter-native', 'rosidl-cli-native', 'rosidl-cmake-native', 'rosidl-generator-c-native', 'rosidl-generator-cpp-native', 'rosidl-parser-native', 'rosidl-runtime-c-native', 'rosidl-runtime-cpp-native', 'rosidl-typesupport-interface-native', 'rosidl-typesupport-introspection-c-native', 'rosidl-typesupport-introspection-cpp-native', 'rpyutils-native', 'openssl-native', 'expat-native', 'ncurses-native', 'util-linux-libuuid-native', 'zlib-native', 'flex-native', 'gnu-config-native', 'libedit-native', 'make-native', 'ninja-native', 'patch-native', 'perl-native', 'pseudo-native', 'python3-atomicwrites-native', 'python3-attrs-native', 'python3-build-native', 'python3-calver-native', 'python3-cython-native', 'python3-editables-native', 'python3-flit-core-native', 'python3-hatch-fancy-pypi-readme-native', 'python3-hatch-vcs-native', 'python3-hatchling-native', 'python3-importlib-metadata-native', 'python3-iniconfig-native', 'python3-installer-native', 'python3-more-itertools-native', 'python3-numpy-native', 'python3-packaging-native', 'python3-pathlib2-native', 'python3-pathspec-native', 'python3-pip-native', 'python3-pluggy-native', 'python3-py-native', 'python3-pyparsing-native', 'python3-pyproject-hooks-native', 'python3-pytest-native', 'python3-setuptools-scm-native', 'python3-setuptools-native', 'python3-six-native', 'python3-toml-native', 'python3-tomli-native', 'python3-trove-classifiers-native', 'python3-typing-extensions-native', 'python3-wcwidth-native', 'python3-wheel-native', 'python3-zipp-native', 'python3-native', 'bzip2-native', 'libnsl2-native', 'libtirpc-native', 'unzip-native', 'xz-native', 'zstd-native', 'attr-native', 'curl-native', 'gdbm-native', 'gmp-native', 'libffi-native', 'libmpc-native', 'mpfr-native', 're2c-native', 'sqlite3-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_configure
| -- The C compiler identification is GNU 13.2.0
| -- The CXX compiler identification is GNU 13.2.0
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Check for working C compiler: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc - skipped
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Check for working CXX compiler: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++ - skipped
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Found ament_cmake: 1.3.5 (/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/ament_cmake/cmake)
| -- Found Python3: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/bin/python3-native/python3 (found version "3.12.1") found components: Interpreter
| -- Found rosidl_default_generators: 1.2.0 (/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/rosidl_default_generators/cmake)
| -- Using all available rosidl_typesupport_c: rosidl_typesupport_fastrtps_c;rosidl_typesupport_introspection_c
| -- Found rosidl_adapter: 3.1.5 (/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/rosidl_adapter/cmake)
| -- Using all available rosidl_typesupport_cpp: rosidl_typesupport_fastrtps_cpp;rosidl_typesupport_introspection_cpp
| -- Found ament_cmake_ros: 0.10.0 (/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/ament_cmake_ros/cmake)
| -- Using all available rosidl_typesupport_c: rosidl_typesupport_fastrtps_c;rosidl_typesupport_introspection_c
| -- Using all available rosidl_typesupport_cpp: rosidl_typesupport_fastrtps_cpp;rosidl_typesupport_introspection_cpp
| -- Found PythonInterp: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/bin/python3-native/python3 (found suitable version "3.12.1", minimum required is "3.6")
| -- Found python_cmake_module: 0.10.0 (/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/share/python_cmake_module/cmake)
| -- Found PythonLibs: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/lib/libpython3.12.so (found suitable version "3.12.1", minimum required is "3.5")
| -- Using PYTHON_EXECUTABLE: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/bin/python3-native/python3
| -- Using PYTHON_INCLUDE_DIRS: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/include/python3.12
| -- Using PYTHON_LIBRARIES: /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/lib/libpython3.12.so
| -- Found PythonExtra: .so
| CMake Error at /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
|   Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES
|   Development NumPy Development.Module Development.Embed) (found version
|   "3.12.1")
| Call Stack (most recent call first):
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/cmake-3.22/Modules/FindPython/Support.cmake:3166 (find_package_handle_standard_args)
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/cmake-3.22/Modules/FindPython3.cmake:490 (include)
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/share/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake:24 (find_package)
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot-native/usr/share/ament_cmake_core/cmake/core/ament_execute_extensions.cmake:48 (include)
|   /home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/recipe-sysroot/usr/share/rosidl_cmake/cmake/rosidl_generate_interfaces.cmake:286 (ament_execute_extensions)
|   CMakeLists.txt:16 (rosidl_generate_interfaces)
| 
| 
| -- Configuring incomplete, errors occurred!
| See also "/home/ryan/Dev/yocto_ws/poky/build/tmp/work/core2-64-poky-linux/builtin-interfaces/1.2.1-1/build/CMakeFiles/CMakeOutput.log".
| WARNING: exit code 1 from a shell command.

I first thought it was an ordering issue due to not using append as recommended, but that didn't fix it.

[Ryanf55]

All fixed up and working. I changed to just add bash. I would like to know from ROS where they list the required packages to be installed for all the tools before installing a bunch of other stuff.

Do you know how to change the default shell for the user from sh to bash if they run ros-image-core-humble?

[robwoolley]

Hi Ryan, thanks for pursing this. I think it may be good to use the packagegroups from oe-core to pull in the other expected packages as well.

There are the options in recipes-core (https://git.yoctoproject.org/poky/tree/meta/recipes-core/packagegroups), but I think recipes-extended (https://git.yoctoproject.org/poky/tree/meta/recipes-extended/packagegroups) may be what we are looking for.

I think "packagegroup-core-full-cmdline" might be a good default starting point for new developers because it has many of the tools that they would just expect to be there: https://git.yoctoproject.org/poky/tree/meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb

I suggest that we should get input from the others in the working group on this at the next meeting. They have good experience in this area and it would be good get their feedback.

Regards,
Rob

[moto-timo]

+1 for core-image-full-cmdline (it also does not suffer from as many disk image size constraints that affect core-image-minimal in my usage)

[Ryanf55]

Contributor

I've added a require directive on this. There were two options I found in poky:

../meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb
../meta/recipes-extended/images/core-image-full-cmdline.bb

I chose core-image-full-cmdline.

Next, with this PR, the images are split up, however, it doesn't seem like core-image-full-cmdline comes with bash.
which bash returns nothing.

1. Should we expect bash to be part of core-image-full-cmdline? I took a browse through the recipe info, but didn't see specifics of packages installed.
2. If core-image-full-cmdline doesn't have bash, should I just add that one dependency, or are there others to add?
3. Why would you use IMAGE_INSTALL:append = "bash" over something like include "${COREBASE}/meta/recipes-extended/bash/bash.inc"

Note: include "${COREBASE}/meta/recipes-extended/bash/bash.inc" results in an error that that's not a valid bitbake file even though the docs say a .inc file is fine. Perhaps COREBASE converts this to an absolute path. Anyways, using the suggested IMAGE_INSTALL method adds bash, but it's not the default shell. Perhaps it could be default on the new ros-image-core.

[Ryanf55]

Hey Rob,

I've ran out of time. Can you take over ownership for this?

[robwoolley]

Sure, I will take this on. Thanks for the heads up and the work.