[WiP] Fix upstream downstream package logic

[mathias-luedtke]

addresses #106

some unrelated minor fixes are bundled as well:

• build to install space (Consider building to install space directly #54) (build directly to install space #150)
• based on improved Fix wstool commands #136
• fold naming
• test result handling

The dependency helper covers most of the designed logic, but uses a consolidated rosdep install step for now.

It detects the target and downsteam packages automatically, if these are not provided.
The detection can be filtered by BUILD_PKGS_WHITELIST and BUILD_PKGS_BLACKLIST (new).
The upstream packages cannot be black- or whitelisted.

Based on all upstream, target and downstream packages the list of rosdep src paths gets compiled.
In addtion the list of tests gets generated and can be filtered with TEST_BLACKLIST (new).

The resulting package list gets printed outside of the folds, including the ignores packages.
As unrelated packages get irgnored now, the travis jobs had to be extended to enforce the downstream packages.

All steps are run in separated folds, so failures can be attributed more easily.
I took the chance to fix #118

(The list of commits is a little bit mixed up - sorted by date instead of relation - in the PR display because of the rebases)

TODO:

• documentation
• setting DOWNSTREAM _PKGS should prevent the auto-detection
• run rostest with env loader (removed source setup.bash by accident)
• make deps.py a proper module
• add unit tests for deps.py

[mathias-luedtke]

For some reasons the actionlib tests fail.
This raised the question: Should downstream tests be exercised? Do we want an opt-out flag?

[fmessmer]

Tested this together with @ipa-mdl on branch ipa320/cob_robots#626
Showed that transitive dependencies are not handled correctly as they eventually end up in the Ignored packages set

[mathias-luedtke]

I have rebased to current master and updated the logic.

catkin's topological_order does not respect exec dependencies in the ordering.So some dependencies were skpipped. This okay for the sake of the builds and tests, but rosdep installs all dependencies and got into trouble.

[mathias-luedtke]

The latest update improved the handling of export and test depends.
exec_depends are now treated as transitive test_depends.
In addtion uses exec_depends get skipped in rosdep install if applicable.

@ipa-fxm: please test again. On my laptop the full run of the kinetic build/test takes 20 minutes.

[fmessmer]

See my ipa320/cob_robots#626 (comment)...

[mathias-luedtke]

The same technique could be used for prerelease tests (#145)

[mathias-luedtke]

rebased on top of #150

[mathias-luedtke]

rebased to current master, works with ROSDEP_SKIP_KEYS

[mathias-luedtke]

relaxed mode is default, strict mode can be enabled with STRICT_TEST_DEPENDS=true.

[mathias-luedtke]

as discussed with @ipa-fxm downstream packages should not get detected automatically

[fmessmer]

having fixed our issue with the travis log length in #267, I would also like to see our travis build time reduced

what is the status of this feature/pr?
I'm willing to help getting this ready to merge

@ipa-mdl
can you summarize what still needs to be done or what is currently preventing this from being merged (despite needing to rebase)?

[130s]

I'm also interested in this feature.
Do you have any plan to come back to this @ipa-mdl?

[mathias-luedtke]

Do you have any plan to come back to this

Not in this form. I think we can migrate some of it to our current codebase. For the actual upstream/downstream detection I would rather use some interfaces of catkin-tools directly instead of my custom code.

[mathias-luedtke]

Proper handling of upstream and downstream workspaces is implemented in #361