🧑‍🌾 Segfaults in macOS Monterey (mac.five.monterey)

[Crola1702]

Environment

• OS Version: macOS Monterey
• Source gz-rendering6 gz-redndering3 homebrew build

Description

• Expected behavior: All tests should pass
• Actual behavior: 33 tests fail by segfaulting

Steps to reproduce

1. Run a build in gz-rendering6-homebrew or ignition_rendering-ci-ign-rendering3-homebrew with mac.five.monterey machine
2. See 33 test regressions that segfault.

Output

Reference build: https://build.osrfoundation.org/job/ignition_rendering-ci-ign-rendering6-homebrew-amd64/104/

Test regressions:

Fortress:

The following tests FAILED:
	  1 - UNIT_ArrowVisual_TEST (SEGFAULT)
	  3 - UNIT_AxisVisual_TEST (SEGFAULT)
	 11 - UNIT_Camera_TEST (SEGFAULT)
	 13 - UNIT_Capsule_TEST (SEGFAULT)
	 21 - UNIT_Grid_TEST (SEGFAULT)
	 23 - UNIT_Heightmap_TEST (SEGFAULT)
	 27 - UNIT_JointVisual_TEST (SEGFAULT)
	 29 - UNIT_LidarVisual_TEST (SEGFAULT)
	 31 - UNIT_LightVisual_TEST (SEGFAULT)
	 33 - UNIT_Light_TEST (SEGFAULT)
	 35 - UNIT_Marker_TEST (SEGFAULT)
	 37 - UNIT_Material_TEST (SEGFAULT)
	 41 - UNIT_Mesh_TEST (SEGFAULT)
	 43 - UNIT_MoveToHelper_TEST (SEGFAULT)
	 45 - UNIT_Node_TEST (SEGFAULT)
	 47 - UNIT_OrbitViewController_TEST (SEGFAULT)
	 49 - UNIT_OrthoViewController_TEST (SEGFAULT)
	 55 - UNIT_RayQuery_TEST (SEGFAULT)
	 63 - UNIT_RenderTarget_TEST (SEGFAULT)
	 65 - UNIT_RenderingIface_TEST (SEGFAULT)
	 67 - UNIT_Scene_TEST (SEGFAULT)
	 77 - UNIT_Text_TEST (SEGFAULT)
	 79 - UNIT_ThermalCamera_TEST (SEGFAULT)
	 81 - UNIT_TransformController_TEST (SEGFAULT)
	 85 - UNIT_Visual_TEST (SEGFAULT)
	 87 - UNIT_WireBox_TEST (SEGFAULT)
	 89 - INTEGRATION_gpu_rays (SEGFAULT)
	 93 - INTEGRATION_depth_camera (SEGFAULT)
	 95 - INTEGRATION_camera (SEGFAULT)
	 97 - INTEGRATION_render_pass (SEGFAULT)
	 99 - INTEGRATION_shadows (SEGFAULT)
	101 - INTEGRATION_scene (SEGFAULT)
	109 - INTEGRATION_lidar_visual (SEGFAULT)

Citadel:

• INTEGRATION_camera.test_ran
• INTEGRATION_depth_camera.test_ran
• INTEGRATION_render_pass.test_ran
• INTEGRATION_scene.test_ran
• INTEGRATION_shadows.test_ran
• INTEGRATION_thermal_camera.test_ran
• UNIT_ArrowVisual_TEST.test_ran
• UNIT_AxisVisual_TEST.test_ran
• UNIT_Camera_TEST.test_ran
• UNIT_Grid_TEST.test_ran
• UNIT_Light_TEST.test_ran
• UNIT_Marker_TEST.test_ran
• UNIT_Material_TEST.test_ran
• UNIT_Mesh_TEST.test_ran
  *UNIT_MoveToHelper_TEST.test_ran
• UNIT_Node_TEST.test_ran
• UNIT_OrbitViewController_TEST.test_ran
• UNIT_RayQuery_TEST.test_ran
• UNIT_RenderTarget_TEST.test_ran
• UNIT_RenderingIface_TEST.test_ran
• UNIT_Scene_TEST.test_ran
• UNIT_ThermalCamera_TEST.test_ran
• UNIT_TransformController_TEST.test_ran
• UNIT_Visual_TEST.test_ran

</details>

[Blast545]

👨‍🌾 This is still affecting our CI. Originally, it only affected builds running in MacOS Monterrey of our older buildfarm. Now it's affecting all of our CI nodes for homebrew on rendering3 and rendering6.

Reference build on rendering3: https://build.osrfoundation.org/job/gz_rendering-ci-ign-rendering3-homebrew-amd64/46/
Reference build on rendering6: https://build.osrfoundation.org/job/gz_rendering-ci-ign-rendering6-homebrew-amd64/51/

The problem manifests itself as the affected tests segfaulting with no extra output or information.

The name of the failing tests makes me think it's related to Ogre configuration on Homebrew, given that all the affected tests include a sort of Testname/ogre in their path. I tried looking to the Ogre configuration, and seems to be configured properly, and found in the CMake build stage.

I'm not particularly familiar with Gazebo configuration, but part of the console output of:
https://build.osrfoundation.org/job/gz_rendering-ci-ign-rendering6-homebrew-amd64/51/consoleFull

41: [Err] [OgreMesh.cc:325] Cannot assign material created by another render-engine
 41/114 Test  #41: UNIT_Mesh_TEST ..........................***Exception: SegFault  0.30 sec
test 42

Makes me think it may that tests are not well configured to run with ogre, and may be used other render engine.

[Blast545]

@scpeters Can I ask you help me taking a look to this one? I went as far as my detective skills got me, but I find it odd that Ogre seems to be setup correctly, but all those tests are failing.

[iche033]

Looked through ign-rendering6 mac failures. Not much info from the logs. The error:

[Err] [OgreMesh.cc:325] Cannot assign material created by another render-engine

is from failing to cast the material to the derived type but I think the cast is not the actual issue. I suspect the material that's being set is already null, which suggests ogre failed to create the material in the first place. So some ogre / rendering operations are failing on the machine. We would probably have to see the ~/.gz/rendering/ogre.log to get more info.

[Crola1702]

We would probably have to see the ~/.gz/rendering/ogre.log to get more info.

I think for this specific case we would need ~/.ignition (as we're talking of rendering6).

Here are both ogre.log files:

• rendering6#57 log file: ogre-rendering6.log
• rendering7#53 log file: ogre-rendering7.log

Let me know if you need anything else from the agent @iche033

[claraberendsen]

Looking at the logs for rendering 3 I see

: OGRE EXCEPTION(7:InternalErrorException): Cannot load GL vertex program Ogre/ShadowExtrudeDirLightFiniteDebug.  Line 0:
Error on line 12: undefined destination identifier (hint: 'R1') in Ogre/ShadowExtrudeDirLightFiniteDebug at /tmp/ogre1.9-20230823-74689-xcss2v/sinbad-ogre-108ab0bcc696/RenderSystems/GL/src/OgreGLGpuProgram.cpp (line 251

[claraberendsen]

@Crola1702 we can go ahead with disabling these tests on rendering3 and rendering6.