Cannot get it running with PX4 SITL and AirSim Libraries

[NoelStephens]

So here is what I have so far
1.) PX4 SITL Firmware running under Windows 10 BASH on Virtual Machine, seems to be running fine with the following output:
..........................................................................................................
commands file: ./posix-configs/SITL/init/lpe/iris

---

| ___ \ \ \ / / / |
| |/ / \ V / / /| |
| __/ / \ / /| |
| | / /^\ \ ___ |
_| / / |_/

px4 starting.

INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [simulator] Waiting for initial data on UDP port 14560. Please start the flight simulator to proceed..
..........................................................................................................

2.) Eigen and Boost Libraries built and AirSim Libraries built which generates an Unreal directory under the AirSim master root. (no compile errors)

3.) Unreal 4.15 (installed from the Unreal Launcher), tested this install and it (UE4) works fine.

4.) Unreal 4.15.x master branch, built this version and tested...works fine (UE4).

5.) An Unreal project (AirSimNow) that I:
created as a clean C++ unreal project
populated with simple terrain that runs fine when not setting the game to SimGameMode.
copied Plugins folder from Air Sim Master folder to the root of my newly created AirSimNow project
right clicked the uproject of the AirSimNow and re-generated the project files
compiled successfully using both the public release version (4.15) and master branch version (4.15.x)

The problem I run into under both versions of Unreal:

Unreal Master Branch 4.15.x locally built Binaries:
This gives me the error that the AirSimNow libraries were built with a different version of Unreal and exits out when I try to run it (i.e. F5 from within VS 2015).

Unreal Public Release 4.15 Binaries:
This actually loads the map with the AirSimNow libraries just fine, but when I go to actually "play" within the UE4-AirSimNow Editor instance, it will do one of two things:

1.) If I make sure there is no settings.json file within the \Documents\AirSim folder, then when I hit play it just draws the 3 boxes and REC button, but then just hangs... and does nothing. No generating of a settings.json file (which I thought was automatic?) or anything. So, this I have tried in all sorts of combinations and for sure -doesn't work-.

2.) If I manually create a settings.json file based on the example provided in the docs, and adjust the IP addresses to match the virtual machine running the PX4 Firmware, then it actually seems to get a bit further... and then crashes promptly at this location:
void MavLinkConnection::join(std::shared_ptr remote, bool subscribeToLeft, bool subscribeToRight)
{
pImpl->join(remote); <---- pImpl is NULL and it crashes here
}

I even tried to change the above function to look like this:
void MavLinkConnection::join(std::shared_ptr remote, bool subscribeToLeft, bool subscribeToRight)
{
if (pImpl != nullptr)
{
pImpl->join(remote);
}
}
Once changed, I completely cleaned and rebuilt the AirSim library (i.e. clean.cmd and then build.cmd), deleted the AirSimNow project's Plugins directory, copied the newly generated Plugins directory back to the AirSimNow project root folder, re-opened the AirSimNow project under visual studio, and completely rebuilt. Upon build complete, loaded the editor, hit play, and it crashes in the same spot as if I never rebuilt the AirSim libraries... as in the files it is loading don't seem to synch with the files I believe I am completely recompiling because the crash pointer under visual studio debugger points to the bracket with the access violation (i.e. null ptr) error:
void MavLinkConnection::join(std::shared_ptr remote, bool subscribeToLeft, bool subscribeToRight)
{ <---- this is where the debugger points to
if (pImpl != nullptr)
{
pImpl->join(remote);
}
}

So, I noticed all of your screen shots are with Unreal 4.14.3... has anyone got this working under 4.15 GA Release or 4.15.x tree?

My questions:

1.) Why does it not generate the settings.json file?
2.) Has PX4 Firmware SITL on Windows BASH been tested with the PX4 Firmware running on a separate system over the network?

Any ideas on how why this is not working?

As a side note, I noticed that you had a link to precompiled binaries but I also noticed that one of the issues you were having was the size of UE4 and maps (i.e. GB of data). Have you guys thought about just compiling everything else BUT UE4 and making just the unique binaries to this project available? (I know the PX4 Firmware can't be included, but still Eigen, Boost, and Airlib could all be provided)

This could help people troubleshoot any issues...they could be tested with the generally available UE4 4.15 binaries to help narrow down the issues.

It seems odd that when you build UE4 (4.15.x) if you take a project that actually loads to the editor phase using the GA 4.15 binaries and switch the UE4 version to your locally built version, rebuilt the same project, and then try and run it... it just never lets you...because it thinks it is a different version.

The most glaring issue that popped out to me was that the instructions for compiling the PX4 Firmware seem to imply that the settings.json file should be created upon the plugin loading (what I took away from them at least) but it seems to never get a chance to even create the settings.json file (when running under the GA 4.15 version) because it seems to be waiting for something. However, if you copy and paste from the settings example it promptly crashes at a place that seems (based on the pImpl definition in the header) should already exist within the class instance...

This the settings file I modified form the example:
{
"AirControlCompID": 1,
"AirControlSysID": 134,
"ExtRendererCompID": 1,
"ExtRendererSysID": 167,
"ExternalSimPort": 14588,
"LocalHostIp": "192.192.1.10",
"LogViewerHostIp": "192.192.1.10",
"LogViewerPort": 14388,
"Pixhawk": {
"SerialBaudRate": 115200,
"SerialPort": "*",
"UdpIp": "192.192.1.12",
"UdpPort": 14560,
"UseSerial": false
},
"QgcHostIp": "192.192.1.10",
"QgcPort": 14550,
"SimCompID": 42,
"SimSysID": 142
}
(I replaced the real IP values with dummy ones)
System Running Simulation and UE4: 192.192.1.10
System Running PX4 Firmware under W10 BASH: 192.192.12

Disabled firewall rules completely for both systems during testing.

I am out of ideas on my end, so I am throwing this out there to see if there could be some other thing I am missing in the process...

Cheers,

-N

[NoelStephens]

Did a bit more digging and came up with some possible answers to some of my questions... but need someone who has been on this project and is a bit more intimate with the code to take a gander please:

Couldn't test PX4 Initially becausde MavLinkTest-->UnitTests has several tests that don't actually use the program arguments:
UnitTest.cpp uses primarily hardcoded values (just a heads up on this)

MavLinkHelper-->Constructor:
Creates a new MavLinkHelper::impl structure and assigns it to the std::unique_ptr pimpl_ as a templated type of the MavLinkHelper::impl structure.

The pointer is relative to the class instance which is relative to the scope of the heap it was created and the pointer is a std:unique pointer which means that if the scope of the current heap is lost then the pointer will be lost as well.

Could the instance of the MavLinkHelper be losing its scope? I tried to reverse my way back out of MavLinkHelper by looking at who/what constructs this bugger and realized that there is some fancy loading happening. :)

It looks like it is instantiated sometime just before the call to World::insert(UpdateableObject* member), but at that point I had spent my alloted time for the week on this hunt and stopped tracing through the code to figure out who or what creates/uses World.

Either case the instance is placed within a templated vector of type TUpdatableObjectPtr which uses the UpdatableContainer as the templated type for the vector members...and at that point I would need to look much closer at how the heap is being managed since the template type of the vector members are (I beleive...more a C# guy now days) defined as the UpdatableContainer being the base object type...but since it derives from the UpdatableObject which is what ControllerBase derives from and in turn where the MavLinkHelper derives from (where we started) it could...depending upon memory alignments etc...do funny things to the heap and how memory is accesses since some UpdatableObject derived classes that are placed within the templated vector collection don't follow the same class memory pattern as UpdatableContainer...

Could this be one of the causes of the issues taking place since the MavLinkHelper is a distant child of UpdatableObject and MavLinkHelper creates the unique pointer to the impl structure?

Maybe...again a bit rusty on my C++... C# may have made my C++ brain lazy.. :)

I also noticed that I couldn't find anything referencing the Settings::saveJSonFile method, which could be why the settings.json file is never generated.

Within the Settings.cpp file in the area between the isLoadSuccess method and
the ensureAppDataFolder method might be where there could be some issues with the heap as well as it continues to try to create the AirSim directory even after it has been created, which then will always throw the error that the creation of the directory failed (because it already exists)..which that in turn makes a call through stdout...and not sure if that could be anything there.

At a minimum, it might be worth adding something like:

bool dirExists(const std::string& dirName_in)
{
DWORD ftyp = GetFileAttributesA(dirName_in.c_str());
if (ftyp == INVALID_FILE_ATTRIBUTES)
return false; //something is wrong with your path!

if (ftyp & FILE_ATTRIBUTE_DIRECTORY)
return true; // this is a directory!

return false; // this is not a directory!
}

and if it does exist then don't call the CreateDirectoryEx method.

And this is as far as I got...I would spend a bit more time on this if I didn't already have a tight schedule, but I think at least I provided enough bread crumbs to provide possible insights to the mystery behind the never appearing settings.json file and the issue with the pimpl being null (at leas that it could possibly be a heap scope type of thing where the unique ptr is lost).

Might look into this again over the weekend... but either case this is what I found out so far...

Cheers,

-N

[lovettchris]

You need to use Unreal 4.14. This is probably the root issue for everything above. When working properly, it will create the settings.json for you.

[sytelus]

Hi @NoelStephens,

I'll try to address some of your questions:

1. settings.json not getting created: This is my fault. At the last minute I found the bug and had to remove automatic saving of json. I didn't updated the doc and I'm sorry it caused you so much confusion. I'm not sure we would bring back automatic saving because of this bug. For now, please just create json file as you had already done. I and @lovettchris will figure out better strategy for json next week.
2. SITL on BoW: First, please see this if this applicable in your case. There were few problems with older version of BashOnWindows that prevented sockets from working properly. I'm wondering what version of Windows you are using. You might try opening a port in BoW (for example, run time service or something like that in Linux) and then try connect it from Windows. See if this basic thing works. I'll put an item in backlog to time out connection sooner so we don't keep waiting for ever creating an unresponsive app situation (please feel free to contribute this change if you like!).
3. Unreal version mismatch: When you build AirSim using build.cmd, it doesn't build any Unreal specific code. It's all just pure C++ with zero linking with Unreal. So there is no specific Unreal version stamped on AirSim binaries you compile and in theory, you can copy AirSim folder in to any Unreal project build with any version. The only reason it might not work is because Unreal engine binaries were built with different version of Visual Studio. We know that 4.12 though 4.14 is compatible with Visual Studio 2015 Update 3. I would expect 4.15 is also compatible with VS2015 Update 3 but I'm not 100% sure and haven't tested it yet. In any case, could you please sure VS version was same across the board (especially if you have multiple VS versions). I'm assuming you don't get this error with project without plugin but after you copy the plugin in the project do you get error.
4. nullptr error: This is surprising because I don't think we are calling MavLinkConnection::join from anywhere. Could you please post the stack trace?

[NoelStephens]

@styleus:
Regarding item 1: No worries, the PX4 SITL instructions (towards the end) mention the settings.json file and the perceived implication was that it auto generated the first time it loaded. Will keep the json file within the folder.

Regarding item 2, the virtual machine is Windows 10 version 10.0.14393.

Regarding item 3, the error only happens when loading the plugin with a locally built version of 4.15.x Unreal. This seemed to happen only after I installed the Epic provided runtime libraries of 4.15 Unreal, prior to that I believe didn't get this error with the local build. I will look into running all of this using 4.14.3.

Regarding item 4, I don't know why it is calling that code either... :)
However, running the MavLinkTest (not sure if this should be used for testing or not) when I use the -server::14560 as the arguments it crashes in the UdpClientPort::connect method when it is trying to bind the socket (socket.bind) with the following error:
Exception thrown at 0x00007FF888FC7788 in MavLinkTest.exe: Microsoft C++ exception: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::system::system_error > at memory location 0x0000002BFAD8C450.

Will look around a bit more to see what can be seen...

[NoelStephens]

Update on Item 4:
Within the MavLinkTest program, the method:
std::shared_ptr MavLinkConnectionImpl::connectLocalUdp(const std::string& nodeName, std::string localAddr, int localPort)

I believe was not complete, as the local address and port were being assigned the server address (i.e. the PX4 Firmware on the remote virtual machine's IP address).

When I hard coded the development system running MavLinkTest with the development system's IP Address as the "local IP" and duplicated the port the code looked a like:
//socket->connect(localAddr, localPort, "", 0);
socket->connect("", localPort,localAddr, localPort);

This adjustment seemed to work for me testing that the PX4 Firware was responding. Once I did this, the MavLinkTest actually ran though its tests:
INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [simulator] Waiting for initial data on UDP port 14560. Please start the flight simulator to proceed..
INFO [simulator] Got initial simuation data, running sim..
INFO [pwm_out_sim] MODE_8PWM
Sleeping for 1 s; (1000000 us).
INFO [platforms__posix__px4_layer] Computer load temporarily too high for real-time simulation. (slowdown delay: 7890278 us)
INFO [platforms__posix__px4_layer] Computer load temporarily too high for real-time simulation. (slowdown delay: 4993718 us)
INFO [tone_alarm] startup
INFO [platforms__posix__px4_layer] Computer load temporarily too high for real-time simulation. (slowdown delay: 4994729 us)
[lpe] fuse gps: 1, flow: 1, vis_pos: 1, vis_yaw: 0, land: 1, pub_agl_z: 1, flow_gyro: 1
INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14556 remote port 14550
INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14557 remote port 14540
WARN [attitude_estimator_q] Q POLL TIMEOUT
INFO [platforms__posix__px4_layer] Computer load temporarily too high for real-time simulation. (slowdown delay: 4995372 us)
WARN [attitude_estimator_q] Q POLL TIMEOUT

But.... it looks like running PX4 under Win-10 BASH makes the PX4 Firmware complain a bit about the computer load... but then again not sure if the MavLink test is all working properly at this time.

However, it at least tells me that the virtual machine running the PX4 Firmware is not having issues with opening the port and listening on it.

PX4 Firmware now is confirmed to be opening and listening... so now we are down to getting the Unreal Plugin to load and communicate...

Will run the 4.14.3 sometime later today or tomorrow and will let you know if that does the trick.

Cheers,

-N

[sytelus]

Looks like we should first eliminate 4.15 from equation. Let us know if you face same issues with 4.14. We will try to support 4.15 soon!

[lovettchris]

You will also need this fix.

[sytelus]

Closing this issue. If you are getting any other problems with 4.14, please re-open or create new issue.