initial support for FreeBSD

[wfurt]

This needs more testing but it somewhat builds and runs.

cc: @jonfortescue @MattGal

[TingluoHuang]

cc: @bryanmacfarlane @vtbassmatt @chrisrpatterson

@wfurt does dotnet support build self-contained app on FreeBSD?

[wfurt]

I'm investigating that @TingluoHuang The SDK support is in early stages. I did some tests yesterday and failed to produce self-contained app for HelloWorld. But that can be also packaging problem as there are no packages published on MyGet/NuGet.

[wfurt]

I got self-contained app working on FreeBSD. It is tricky at this moment but that should be easier once we get all packages published. Also note that the RID is still somewhat in flux and this change uses least specific like other OSes.

Few more notes:
FreeBSD does not have bash in /bin. That can be documented and one can use "bash dev.sh XXX"
We seems to download pre-built copy of Node. There is not one for FreeBSD. For now, I use link to system copy. Maybe agent could be changed that if externals/node/bin/node does not exist we simply search PATH. (or we let OS to do it)

[wfurt]

This should be ready @TingluoHuang . We have 2.142.1 running for several weeks and now we also have 2.144 building official builds.

let me know if you have any questions.

cc: @safern

[TingluoHuang]

@wfurt i am not worry about the code change, but we do have work on the service (UI/acquisition) & doc side before we can add this new FreeBSD agent.
i think we need to figure out the schedule with @chrisrpatterson @bryanmacfarlane @vtbassmatt

[wfurt]

that make sense. I can progress on Documentation and notes. Even if not part of official builds this change helps others to make builds as needed. (and possibly takes that from me) It is also harder to keep up with other changes and keep this PR fresh.

[wfurt]

CI tests all green. sorry for the turn.

[vkrasnov]

It would be really awesome to have a FreeBSD agent

[wfurt]

Any more thoughts on this @bryanmacfarlane @vtbassmatt @chrisrpatterson @TingluoHuang
Even if we don't have official builds it would be so much easier to simply checkout to commit and build instead of merging and fiddling every time when CI pool upgrades.

[vtbassmatt]

Unfortunately, we can't pursue a FreeBSD agent right now. We don't have official .NET Core support for the platform, and my team can't take on the responsibility for maintaining shims or compat hacks.

[wfurt]

Maybe you misunderstood the ask @vtbassmatt. This is not about your team taking responsibility or producing builds. Look at it as community support at this moment.

This is essential for .NET Core building up FreeBSD support. It is dependency loop problem. Without build agent we will not be able ever have official support.
We have FreeBSD build queues with agent agent since fall last year and we working on better support and fixes. Taking this (or follow-up fixes) would greatly simplify our task.

[vkrasnov]

@wfurt is there some documentation on how to build this? I tried but ran into issues. Thanks.

[wfurt]

I'll update PR to resolve conflicts and I'll post notes later @vkrasnov.

[holzingk]

This is the single missing feature that keeps us from choosing Azure Pipelines.

[wfurt]

I updated freebsd branch. One can build with following commands:

sed -i .old 's/2.1.403/3.0.100-preview-010021/' dev.sh
find . -name *csproj | xargs  sed -i .old s/netcoreapp2.1/netcoreapp3.0/
bash dev.sh layout Release
bash dev.sh package Release

There is no FreeBSD in 2.x and there was no easy way how to make it configurable.
There is no pre-build version of node so FreeBSD package depends on system copy installed via pkg install node

Note, that .Net on FreeBSD is not complete and production quality. As early adopters you may run to issues. If so, please report on corefx or coreclr repo with os-freebsd label.

Most testing was done on 11.2. 12.0 had reported issue with thread locking and it should be fixed in FreeBSD updates as well as there is workaround.

If you running on Azure instance and if you use /mnt/resource volume, add following line to /etc/fstab
/dev/da1s1 /mnt/resource ufs rw 0 1
That volume is mounted too late for agent to start as service.

good luck.

[vkrasnov]

Thank you Tomas.

When trying to build on an azure FreeBSD 11.2 instance I get this:

 Failed to initialize CoreCLR, HRESULT: 0x8007001F
/usr/home/vkrasnov/azure-pipelines-agent/src/dir.proj(39,5): error MSB3073: The command "dotnet publish Agent.Sdk/Agent.Sdk.csproj --output "/usr/home/vkrasnov/azure-pipelines-agent/src/../_layout/bin" --runtime freebsd-x64 -c Release /p:PackageRuntime=freebsd-x64 /p:Version=2.149.0" exited with code 137.

Should I report to coreclr?

[vkrasnov]

Never mind, the problem was too little RAM, resizing to an 8GB instance solved the problem!

[vkrasnov]

I can confirm that this works on FreeBSD 11.2, on clean install also requires:
pkg install bash
pkg install libunwind

And 8GB of RAM (to build)

[josteink]

FreeBSD does not have bash in /bin. That can be documented and one can use "bash dev.sh XXX"

I thought the canonical way of doing this was “/usr/bin/env bash”?

That’s what we’ve done in lots of the dotnet build-scripts to make them fully portable.

[wfurt]

This is not complete change @josteink. For one we need to target 3.0 as well as use different SDK version. But at least this would get us to more manageable state.
We will need similar changes for ARM64 as there is no working 2.x SDK.

[davidchisnall]

Apparently I can't comment on the code directly, so I'll leave some notes here:

• On FreeBSD, the ps -e command may give you truncated output. If you're happy to invoke an external utility then procstat -e {pid} will give you better information. Ideally, you'd use libprocstat (in the base system, so on any FreeBSD system). I have attached a simple program (compile with -lprocstat to link libprocstat) that demonstrates how to use this interface.
• Please use /usr/bin/env bash in the shell scripts, not /usr/local/bin/bash - this will find bash wherever in the path it is installed. If FreeBSD is configured with a LOCALBASE somewhere other than /usr/local when building ports, bash may be elsewhere. I didn't see anything obviously bash-specific in that script though, so /bin/sh would be preferable if it's possible to avoid bash dependencies. In particular, including /etc/rc.subr may be dangerous in a bash script because these scripts are all intended to be used with a POSIX shell and may hit some subtle incompatibilities when used with bash.

#include <sys/param.h>
#include <sys/queue.h>
#include <sys/socket.h>
#include <sys/sysctl.h>
#include <sys/user.h>
#include <libprocstat.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
        if (argc != 2)
        {
                fprintf(stderr, "PID not specified\n");
                return -1;
        }
        pid_t pid = strtol(argv[1], NULL, 0);
        struct procstat *ps = procstat_open_sysctl();
        unsigned count;
        struct kinfo_proc *proc = procstat_getprocs(ps, KERN_PROC_PID, pid,
                        &count);
        if (count != 1)
        {
                fprintf(stderr, "Failed to get process\n");
                return -1;
        }
        // env is owned by proc.
        char **env = procstat_getenvv(ps, proc, 0);
        for (char **e=env ; *e != NULL ; e++)
        {
                puts(*e);
        }
        procstat_freeenvv(ps);
        procstat_freeprocs(ps, proc);
        procstat_close(ps);
        return 0;
}

[vtbassmatt]

Until .NET has FreeBSD on their roadmap for official support, we can't commit to this. (There's a chicken-and-egg problem in that they need a Pipelines agent before they can ship. I could break that cycle if we had directional alignment.) The most recent comment I found indicates that this is still not on the roadmap, so I'm going to close this PR. Again, we're very open to doing this once it appears on .NET's horizons for real.

[davidchisnall]

Is it possible to build the agent with another .NET implementation? Mono is supported on FreeBSD, for example.

[vtbassmatt]

@davidchisnall it's probably possible. And as an open source codebase, we wouldn't dare try to stop someone from doing so on their own effort. But without either .NET (with whom we have a special relationship, since we are their CI platform and they are our implementation platform) or a large enough community desire, we can't commit to the maintenance & support overhead. And FWIW, we recently added two features which would make it easier to maintain a fork:

• We stopped automatically sliding the requested agent version forward just because a YAML pipeline is in play. Now we will only request a new version if the system truly needs a feature not present in down-level agents.
• We also built a setting on the pool side which says, in effect, "instead of requesting agent updates in this pool, fail the build instead". While failing builds doesn't sound great, it's better than silently hanging every build waiting for an agent update that's never going to happen.

There are probably a lot of Linux-isms in the adjunct code (config.sh, run.sh, installdependencies.sh, and so on). Also, I spoke with someone (possibly you?) and learned that there are a lot of differences in container details, which rules out a pretty big swath of our features.

[davidchisnall]

Thanks. We are working on a collaboration with ARM, the University of Cambridge, and others under the Digital Security By Design Industrial Strategy Challenge Fund (£70m UK government funding, over £100m industry matched funding) and the demonstration platform there is all FreeBSD-based. Lack of support here is the blocker for our partners to use Azure for CI, so a way of unblocking this would be very helpful.

We are also considering adding Cirrus CI to Microsoft/snmalloc, so that we can avoid regressions in our FreeBSD implementation, but it's not a great look for a Microsoft GitHub project to use non-Microsoft CI, so we'd much prefer to see a fully supported offering in Azure Pipelines.

[wfurt]

FreeBSD support is coming back in .NET5. Current strategy is to cross-compile native code on Linux (and use Linux agents)
It would still be nice if others can build agent even if core project does not want to commit to it. That is also true for arm64.

[joperator]

Current strategy is to cross-compile native code on Linux (and use Linux agents)

I can provide an implementation of the azure-pipelines-agent (version 2.165.2) for FreeBSD. It was built using the .NET Core SDK and essential NuGet packages mentioned on this wiki page.
Maybe this makes it easier to compile native code for FreeBSD rather than cross-compiling it on Linux?

[jasonpugsley]

@joperator I'd be happy to help.
I'm also wondering where discussion around this sort of thing happens. Is it just in github Issues/PRs or are you guys chatting somewhere else?