Only can spawn gate in scene, when changing to other prefab_id, still spawn the gate

[test-bai-cpu]

In the quadrotor and objects section in flightmare document(https://flightmare.readthedocs.io/en/latest/first_steps/quad_and_objects.html), it said: The model loads the prefab matching the prefab_id in the folder Assets/Resources

The code is

std::string prefab_id = "rpg_gate"; // Name of the prefab in the Assets/Resources folder
std::shared_ptr<StaticGate> gate = std::make_shared<StaticGate>(object_id, prefab_id);

When I change "rpg_gate" or other values, it will still spawn the gate. For example, I have tried with: Transparen_Cube.

The class StaticGate can be used with any object with prefab_id, it does not only work for gate.

Then I checked the Flightmare Unity Binary, the directory is flightmare/flightrender/RPG_Flightmare. This binary contains four scenes: industrial, warehouse, garage, natureforest. But I cannot find the origin unity project package with this binary. The package repo: (https://github.com/uzh-rpg/flightmare_unity) only contains scene Industrial. So I don't know if the prefabs in flightmare_unity/Assets/Resources, like Drone_red, Transparen_Cube, rpg_gate, transparent, also exist in the unity binary in flightmare.

So I tried to build binary using project flightmare_unity, and add prefab gameobject in the project. Replace the content of flightmare/flightrender/RPG_Flightmare, and run the simulation. It works. I can see my changes in the scene, but still only spawn the gate, no matter how I set the prefab_id to.

I tried to search how is flightmare use prefab_id to match the object to spawn, but failed.

So How can I spawn a static object in the scene, except for rpg_gate?

Thanks!

[yun-long]

1. all the environments we used in the binary are paid Assets, unfortunately, we cannot opensource it. If you want to modify them, you would have to buy these assents from the unity store. they are not expensive.
2. Yes, you can spawn any static objects in the scene, but you would have to first make such a prefab and then associate it also on the Unity side.

[test-bai-cpu]

Thanks!

Yes, with the second part, I think it should work to spawn any static object if the prefab exist.

But when I follow the example in the document: https://flightmare.readthedocs.io/en/latest/first_steps/quad_and_objects.html

In this part:

// Initialize gates
std::string object_id = "unity_gate"; // Unique name
std::string prefab_id = "rpg_gate"; // Name of the prefab in the Assets/Resources folder

When I change the prefab_id to any random string, like "aaa", "bbb", or try with "Transparen_Cube", it will still spawn the rpg_gate.

If there is only one prefab available (like only have rpg_gate associated) , other strings should return an error, but not the rpg_gate. So I am confused about how to spawn objects.

[yun-long]

I see. Sorry, I guess we didn't associate the object on the unity site since we currently only have one prefab for the static object.

You have to check this file and make changes according,
https://github.com/uzh-rpg/flightmare_unity/blob/main/Assets/Flightmare/Scripts/CameraController.cs

[test-bai-cpu]

Thanks! I changed the file and build the scene. It works now.

[francofusco]

Hi! ☺️ I would like as well be able to insert other object types into the scene. @test-bai-cpu, since you said you managed to do it, could you please share the steps you followed? Unfortunately I have no experience with Unity and I am not yet familiar with C#, so I am a little lost. If you find the time to give me some pointers, it would be very useful 👍 If you do not have time, do not worry, I will try to figure it out and eventually post the steps here for future reference.

Alternatively, @yun-long is there a documentation on how to customize and re-build flightmare_unity?

Thanks in advance to both of you! 😃

[test-bai-cpu]

Yes.
About customize and re-build the flightmare_unity:

The project is https://github.com/uzh-rpg/flightmare_unity. Open this project in Unity, modify the project, build the scene. The scene will be built and several files will be generated. Put these generated files in flightmare/flightrender and replace the original files.

About insert other objects, my temp solution is:

Open flightmare_unity project in Unity, add your own prefabs, and modify this file: Assets/Flightmare/Scripts/CameraController.cs.
In the function instantiateObjects(), I remove line 707(the line with gate_template) and add(uncomment) the

GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;

And instantiate the objects in update function updateObjectPositions() :
remove line 565(the line with gate_template) and add

GameObject other_obj = Instantiate(Resources.Load(obj_state.prefabID, typeof(GameObject))) as GameObject;

Then rebuild the scene.

With this solution, I can dynamically add any prefab in the unity scene, but fail to update the position of each object.

[francofusco]

Thanks @test-bai-cpu, your feedback was very useful 🎉 I could focus on the relevant parts and I finally got a rough idea of what's going on and what should be changed.

I think I have also managed to make the code insert custom objects while allowing to update their position. I am not sure enough - as I said, I am a C# noob and thus I have to understand if what I did is actually what I think I did 😆 If my changes are indeed successful, I will come back and post them here (and perhaps propose a PR).

Meanwhile, thanks again and have a nice day! ☺️

[francofusco]

Ok, I just tested the code more in depth and I think my changes works properly. I will prepare a PR this afternoon, but I will also post the code here for quick reference. The change involves the for loop at lines 702-710.

Original code:

foreach (var obj_state in settings.objects)
{
  // GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
  Debug.Log("obj_state id : " + obj_state.ID);
  GameObject obj = internal_state.getGameobject(obj_state.ID, gate_template);
  obj.transform.localScale = ListToVector3(obj_state.size);
    // obj.layer = 9;
}

My solution:

foreach (var obj_state in settings.objects)
{
  Debug.Log("obj_state id : " + obj_state.ID);
  // try to load the target object
  GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
  if(prefab == null)
  {
    // if loading failed, fallback to the gate
    Debug.Log("Failed loading object, using default");
    prefab = gate_template;
  }
  // Add the object
  GameObject obj = internal_state.getGameobject(obj_state.ID, prefab);
  obj.transform.localScale = ListToVector3(obj_state.size);
}

The rest of CameraController.cs was left untouched. After building the project in Unity and copying the "output files" into flightmare/flightrender/, I am able to spawn other objects by calling, e.g.,

// This spawns a red semi-transparent cube
std::shared_ptr<flightlib::StaticGate> red_cube =
    std::make_shared<flightlib::StaticGate>("floating_cube", "Transparen_Cube");
unity_bridge_ptr->addStaticObject(red_cube);
// The prefab-id is not recognized: a gate will be spawned instead
std::shared_ptr<flightlib::StaticGate> actually_a_gate =
    std::make_shared<flightlib::StaticGate>("bad_object", "Object-404");
unity_bridge_ptr->addStaticObject(actually_a_gate);

Furthermore, objects can be moved around using the StaticGate::setPosition method 👍

[test-bai-cpu]

Thanks for the solution! It helped me a lot!

There is only one issue, it only changes the instantiate function(as you proposed, in lines 702 - 710). So the problem is I need to instantiate all the obstacles before connect to unity. Cannot add obstacles afterwards.
For example,
I can only do this:

std::shared_ptr<StaticObstacle> obstacle1 = std::make_shared<StaticObstacle>("ob1", "Sphere");
obstacle1->setPosition(Eigen::Vector3f(0, 0, 0));
// to instantiate the obstacle1 
unity_bridge_ptr->addStaticObject(obstacle1);
bool unity_ready = unity_bridge_ptr->connectUnity(UnityScene::INDUSTRIAL);

// to update the position of obstacle1 
if (frame_id == 300) {
    obstacle1->setPosition(Eigen::Vector3f(0, 10, 2.5));
    unity_bridge_ptr->addStaticObject(obstacle1);
}

If I do like this:

std::shared_ptr<StaticObstacle> obstacle1 = std::make_shared<StaticObstacle>("ob1", "Sphere");
bool unity_ready = unity_bridge_ptr->connectUnity(UnityScene::INDUSTRIAL);

// to instantiate the obstacle1 
if (frame_id == 100) {
    obstacle1->setPosition(Eigen::Vector3f(0, 10, 2.5));
    unity_bridge_ptr->addStaticObject(obstacle1);
}

It will render the rpg_gate, not the sphere, because in the update function, it still use the gate_template.

So from line 561 - 568, the update_position function, the original code is

foreach (Object_t obj_state in sub_message.objects)
{
    // Apply translation, rotation, and scaling
    // GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
    GameObject other_obj = internal_state.getGameobject(obj_state.ID, gate_template);
    other_obj.transform.SetPositionAndRotation(ListToVector3(obj_state.position), ListToQuaternion(obj_state.rotation));
    other_obj.transform.localScale = ListToVector3(obj_state.size);
}

Might need to have the same change as you proposed in instantiate function, so it should be

foreach (Object_t obj_state in sub_message.objects)
{
    GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
    GameObject other_obj = internal_state.getGameobject(obj_state.ID, prefab);
    other_obj.transform.SetPositionAndRotation(ListToVector3(obj_state.position), ListToQuaternion(obj_state.rotation));
    other_obj.transform.localScale = ListToVector3(obj_state.size);
}

[francofusco]

I hadn't thought about it 🤔 It totally makes sense, thanks for the feedback! I'll work on a PR tomorrow with these modifications so that everyone can have the code with no additional efforts 😃

[francofusco]

Short update: I played a little with the code, and one can now insert other objects in the scene, both before and after the connection with Unity is established.

I will soon open the PR, however at this point I believe there is one little detail still missing: while I can add objects at any time, I think that they cannot be removed afterwards. @yun-long can you confirm? And if indeed the feature is currently missing, did you already have any plan to add such a feature? I have started giving it a try, see this video. If you find it useful, I can add the feature in the PR that I want to propose!