Support external repos

[dirk-thomas]

Support generating scripts to run jobs locally for repositories which are not in the rosdistro database.

This goal is based on a conversation at a ROS-Industrial meeting about ROS-Industrial CI. An example of their configuration: https://github.com/shaun-edwards/packml/blob/indigo-devel/.travis.yml

@shaun-edwards FYI. While this is just an issue (sorry no pull request yet) I thought adding you will make sure you get updates on this topic if there is any progress.

[dirk-thomas]

Currently this is only possible for prerelease tests and the command line argument is kind of "verbose":

ros_buildfarm/scripts/prerelease/generate_prerelease_script.py

Lines 77 to 84 in b7ae43a

	group.add_argument(
	'--custom-repo',
	nargs='*',
	type=_repository_name_and_type_and_url_and_branch,
	default=[],
	metavar='REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME',
	help='The name, type, url and branch / tag name of a repository, '
	'e.g. "common_tutorials:git:https://github.com/ros/common_tutorials:pullrequest-1"')

[dirk-thomas]

I have added some docs about how to use the prerelease job for repositories which are not listed in the ROS distro: #401.

I don't think that support will be extended to the other job types. For example release jobs rely on the fact that all dependencies are available from Debian packages which is most of the time not the case for an external repo. So I will likely close this ticket soon assuming that supporting the use case with prerelease jobs is sufficient.

[dirk-thomas]

@shaun-edwards @130s @davetcoleman I would be curious what you guys think since you are currently using either ROS-Industrial CI or a custom script to run tests on Travis. Any input what features are missing in ros_buildfarm, what additional features your solutions offer and how the setup / invocation could be easier would be appreciated.

[130s]

My 2 cents:

• It'd be ideal if commonly usable features like testing unlisted repos is supported by the buildfarm. I started industrial_ci when I didn't find a similar feature available from buildfarm back then. We're willing to delegate as many features as possible Utilize ros.org as much as possible ros-industrial/industrial_ci#63, Improvement ideas from MoveIt! maintainers ros-industrial/industrial_ci#75 (comment)

• I don't think that support will be extended to the other job types. For example release jobs rely on the fact that all dependencies are available from Debian packages which is most of the time not the case for an external repo.

  Frankly speaking, I'm just not sure how useful it would be if there's only prerelease test option...From my experience of integrating prerelease test into industrial_ci, I see not many users use that prerelease option, although I recommended several times (I need to do better job here). Are there many access to http://prerelease.ros.org or if there's a way to count the PR test execution (e.g. run of generate_prerelease_script.py)?

  Integrating .rosinstall is obviously one way to enable build from source (industrial_ci allows that). Caveat at the downstream repo is handling the potential errors coming from the upstream upon source build...industrial_ci just lets a failure happen and stops but I'm not sure what's the right way to take care of the case.

CC @ipa-mdl

[dirk-thomas]

It'd be ideal if commonly usable features like testing unlisted repos is supported by the buildfarm.

You can do exactly that with the prerelease job. So I am not sure what you are expecting here / what is missing for your use case.

Frankly speaking, I'm just not sure how useful it would be if there's only prerelease test option.

Can you please describe which job type you would like to see supported?

• A devel job is essentially the same as a prerelease job with a single repo and nothing in the overlay. So I don't see why that needs to be supported for custom repos if the user can already use the prerelease job.
• A release job looks like a lot of effort to me since you would need to process each package in the repo separately (in topological order) and you need to be able to get all dependencies as Debian packages from somewhere. How does your use case look like for this?
• A doc job would be doable. But I would like to know if anybody actually has a use case for this and how they plan to host the cross referencing information as well as the generated docs.

Are there many access to http://prerelease.ros.org or if there's a way to count the PR test execution (e.g. run of generate_prerelease_script.py)?

I don't have any statistics about the website but since it is not necessary to run prereleases I don't think we can quantify how many users use this option.

Integrating .rosinstall is obviously one way to enable build from source

Passing a .rosinstall file is certainly nicer than the current --custom-repo argument. But they essentially contain the exact same information. So that would be only syntactic sugar to make it easier to call - but not a new feature.

[mathias-luedtke]

Currently we just skip prerelease_clone_underlay.sh and fill WORKSPACE with the merged target repository.

It would be great if the prerelease tests could work on local sources.
We could add more repo types to the clone script:

• rosinstall: uses wstool
• local: copy (cp -a) from path
• external: skip in clone script, user has to provide the packages in $WORKSPACE/src

Examples:
--custom-repo my_repo:rosinstall:file://my/path/to/.rosinstall:
--custom-repo my_repo:rosinstall:https://my_server.example/path/to/rosinstall:
--custom-repo my_repo:local:/home/travis/build/orga/repo:
--custom-repo my_repo:external::

[dirk-thomas]

We could add more repo types to the clone script:

The clone script only operates on scm's. So I don't think you have a different scm which isn't in that list yet?

I guess if the prerelease script could be invoked without any specific custom repos but then uses the workspace found in a known location all of your three use cases would work. You could use wstool and .rosinstall files to clone any set of repos, you could copy them from somewhere, or provide them directly in that location. (I don't see a benefit of adding explicit support for those in ros_buildfarm if it is doable with a separate invocation of an existing tool - less dependencies, less maintenance.)

If I am not mistaken the prerelease script should already support that by prepopulating the folder where the clone script would usually clone the repos to. I will give it a try and report back.

[mathias-luedtke]

if I am not mistaken the prerelease script should already support that by prepopulating the folder where the clone script would usually clone the repos to

That' s how it is working right now.
Supporting .rosinstall is on my list..

I guess if the prerelease script could be invoked without any specific custom repos

Another possibility is to use catkin as repo ;)
A --generic flag would be great.
Perhaps with an optional repository for downstream look-up?

The clone script only operates on scm's. So I don't think you have a different scm which isn't in that list yet?

But it could be extended to support other types as well:

@[elif repo_spec.type == 'rosinstall']@
    (set -x; wstool init @path @repo_spec.url)
    (set -x; wstool update -t @path)
    (set -x; wstool info -t @path)

This would enable the use of rosinstall for all users.

[shaun-edwards]

@dirk-thomas, the main benefit of industrial_ci is that it can be utilized by enabling TravisCI and then committing a yaml file (it's that simple). The build farm requires multiple steps, see Releasing a package for the first time

While digging through the wiki, I found this, which I believe addresses my use case (I'm sure there are other differences I don't understand).

[dirk-thomas]

@ipa-mdl

A --generic flag would be great.

Can you please elaborate what you imagine that flag to do.

Perhaps with an optional repository for downstream look-up?

Are you referring to apt repositories or repositories contain the sources of additional packages?

• In the first case: passing additional apt repos should be easy to add. But do you expect the dependencies listed in the package manifests to be mapped to the Debian packages provided by the apt repo? The ROS packages in the processed repo contain rosdep keys. Additionally package names released in the rosdistro are also known to rosdep. But arbitrary other Debian package names not covered by rosdep / rosdistro wouldn't be mapable even if the apt repo is known.
• In the second case: that is exactly the difference between prerelease jobs and devel jobs. For prerelease jobs you can specify multiple source repositories. You essentially have to specify all dependencies which are not available in binary form in order to build the workspace.

(In my opinion the term "downstream" is problematic in this context. In Jenkins "downstream" refers to jobs which depend on a specific job which is so pretty much the opposite direction than what you mean.)

This would enable the use of rosinstall for all users.

From my perspective that feature is already available for all users. They just call the two command in sequence (rather than having a single command doing both). It is more like the pipe concept in Linux - instead of writing a single program which does multiple things you can call separate programs and pipe the data between them, e.g. grep foo | wc -l.

@shaun-edwards The ros_buildfarm package is designed so that all jobs can be run by users locally. This allows them to reproduce exactly what happens on the buildfarm which is valuable to debug certain problems. So on Travis it can therefore also be run in the same way. You certainly don't need to go through the "Releasing a package" steps when all you want to run is a CI / devel / prerelease job.

[mathias-luedtke]

@shaun-edwards:

You can run the prerelease test without running bloom first, but you have to add a source entry to distribution.yaml beforehand.

@dirk-thomas

A --generic flag would be great.

Can you please elaborate what you imagine that flag to do.

"Support generating scripts to run jobs locally for repositories which are not in the rosdistro database."

It should switch the generate script into a mode that does not require a repository name.
This is more explicit than just leaving it out.

[...] In Jenkins "downstream" refers to jobs which depend on a specific job which is so pretty much the opposite direction than what you mean.

This is exactly what I meant, but it is called --level in the generate script.
For some reason the option is called PRERELEASE_DOWNSTREAM_DEPTH in industrial_ci.

From my perspective that feature is already available for all users. They just call the two command in sequence (rather than having a single command doing both).

If I recall it correctly it involves more steps like creating the src directory first.
I had to read the generated scripts to figure it out.
However, this is not really user friendly.

[dirk-thomas]

@ipa-mdl

You can run the prerelease test without running bloom first, but you have to add a source entry to distribution.yaml beforehand.

I was pretty sure that this was not necessary if you use the prerelease job with the custom repo argument. Turns out that what not the case (probably a regression at some point?). Please see #405 which resolves that (I will update the PR to also add some documentation about this use case and probably cover it in a Travis test). With that the following example works without the repo being listed in the rosdistro at all:

mkdir /tmp/prerelease && cd /tmp/prerelease
generate_prerelease_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml kinetic default ubuntu xenial amd64 --output-dir . --custom-repo  repo_name:git:https://github.com/ros-infrastructure/ros_buildfarm:dummy_package
./prerelease.sh

It should switch the generate script into a mode that does not require a repository name.
This is more explicit than just leaving it out.

The repo name determines under which path the repository is being cloned. As an alternative you can clone the repo beforehand and let the prerelease script take care of only the "processing" (not the cloning). See next paragraph for an example.

If I recall it correctly it involves more steps like creating the src directory first. I had to read the generated scripts to figure it out. However, this is not really user friendly.

mkdir /tmp/prerelease && cd /tmp/prerelease
git clone -b dummy_package https://github.com/ros-infrastructure/ros_buildfarm /tmp/test/catkin_workspace/src/ros_buildfarm
generate_prerelease_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml kinetic default ubuntu xenial amd64 --output-dir .
./prerelease.sh [-y]

The git clone is just an example. You can replace it with whatever tool invocation you like (wstool, vcstool, etc.). If any of these tools require the destination path to exist it might be good to fill a feature request for them to not require that.

[mathias-luedtke]

Thanks @dirk-thomas, I will test this together with the #399 options.

[mathias-luedtke]

First version: ros-industrial/industrial_ci#145

[dirk-thomas]

With #405 being merged and released #394 I will close this ticket for now since all uses cases I am aware of at the moment seem to be supported. Thank you for all your feedback.

Please feel free to comment on the closed ticket with additional feedback or consider opening new tickets (or even better pull requests) if you have additional use cases you think should be supported.

[130s]

Being late on board...Thanks for clarification and discussion!
The wiki page for prerelease test needs to reflected all the great advancement. I just edited the wiki, which apparently isn't accurate any more now that prerelease test nicely allows testing repos that aren't listed on rosdistro (this is also noted at ros-infrastructure/prerelease_website#38 (comment)). I'm willing to do that but won't have a gear this week.