Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use libjulia for Libtask #2087

Merged
merged 4 commits into from
Nov 19, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 0 additions & 62 deletions L/Libtask/build_dylib.sh

This file was deleted.

77 changes: 47 additions & 30 deletions L/Libtask/build_tarballs.jl
Original file line number Diff line number Diff line change
@@ -1,52 +1,69 @@
# Note that this script can accept some limited command-line arguments, run
# `julia build_tarballs.jl --help` to see a usage message.
using Pkg
using BinaryBuilder
using BinaryBuilder, Pkg

name = "Libtask"
version = v"0.3.2"
commit_id = "fbe338053f402d76524d0a01f3796dd7da90b781"
julia_version = v"1.3.1"

# see https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/336
# ENV["CI_COMMIT_TAG"] = ENV["TRAVIS_TAG"] = "v" * string(version)
name = "Libtask"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW I don't see a good reason for renaming this to Libtask_julia or so -- why should it matter that this links against libjulia_jll? We don't generally mention other dependencies in the name, do we?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the idea from the name of the library libcxxwrap_julia but the main argument would be that, contrary to what the name suggests, this is not a build file for a binary package of the C library Libtask but rather an implementation of task copying for Julia (which is mainly/only used by particle filter implementations in Turing).

version = v"0.4.0"

# Collection of sources required to build Libtask
sources = [
"https://github.com/TuringLang/Libtask.jl.git" => commit_id,
DirectorySource("./bundled"),
]

# Bash recipe for building across all platforms
script_file = joinpath(@__DIR__, "build_dylib.sh")
if !isfile(script_file) # when run generate_buildjl.jl
script_file = joinpath(@__DIR__, "L/Libtask/build_dylib.sh")
end
script = read(script_file, String)
script = raw"""
# create output directory
mkdir -p "${libdir}"

# compile library
cd "${WORKSPACE}/srcdir/"

CFLAGS="-I${includedir} -I${includedir}/julia -O2 -shared -std=gnu99 -fPIC"
if [[ "${target}" == i686-* ]]; then
CFLAGS="${CFLAGS} -march=pentium4"
fi
if [[ "${target}" == *-mingw* ]]; then
CFLAGS="${CFLAGS} -Wl,--export-all-symbols"
elif [[ "${target}" == *-linux-* ]]; then
CFLAGS="${CFLAGS} -Wl,--export-dynamic"
fi

LDFLAGS="-L${libdir} -ljulia"
if [[ "${target}" == *-mingw* ]]; then
LDFLAGS="${LDFLAGS} -lopenlibm"
fi

$CC $CFLAGS $LDFLAGS task.c -o "${libdir}/libtask_julia.${dlext}"

# install license
install_license LICENSE.md
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
Platform("i686", "linux"; libc="glibc"),
Platform("x86_64", "linux"; libc="glibc"),
Platform("powerpc64le", "linux"; libc="glibc"),
# Platform("armv7l", "linux"; libc="glibc"),
Platform("aarch64", "linux"),
Platform("x86_64", "macos"),
Platform("i686", "windows"),
Platform("x86_64", "windows")
]
platforms = supported_platforms()

# skip i686 musl builds (not supported by libjulia_jll)
filter!(p -> !(Sys.islinux(p) && libc(p) == "musl" && arch(p) == "i686"), platforms)

# skip PowerPC builds in Julia 1.3 (not supported by libjulia_jll)
if julia_version < v"1.4"
filter!(p -> !(Sys.islinux(p) && arch(p) == "powerpc64le"), platforms)
end

# The products that we will ensure are always built
products = [
LibraryProduct("libtask_v1_0", :libtask_v1_0)
LibraryProduct("libtask_v1_1", :libtask_v1_1)
LibraryProduct("libtask_v1_2", :libtask_v1_2)
LibraryProduct("libtask_v1_3", :libtask_v1_3)
LibraryProduct("libtask_julia", :libtask_julia),
]

# Dependencies that must be installed before this package can be built
dependencies = [
BuildDependency(PackageSpec(name="libjulia_jll", version=julia_version)),
]

# Build the tarballs, and possibly a `build.jl` as well.
# build_file = "products/build_$(name).v$(version).jl"
build_tarballs(ARGS, name, version, sources,
script, platforms, products, dependencies)
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies;
julia_compat = "~$(julia_version.major).$(julia_version.minor)",
lock_microarchitecture = false)
21 changes: 21 additions & 0 deletions L/Libtask/bundled/LICENSE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2018 The Turing Language

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
74 changes: 74 additions & 0 deletions L/Libtask/bundled/task.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
/*
task.c
task copying (aka continuation) for lightweight processes (symmetric coroutines)
*/

#include "julia.h"

// Workaround for JuliaLang/julia#32812
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so I read this as saying: "JuliaLang/julia#32812 is fixing something we need but it is not available in all Julia versions so we work around this"; but you are really saying "JuliaLang/julia#32812 introduced a regression that we are working around". But where is the Julia issue which reports the regression and discusses solutions?

jl_task_t *vanilla_get_current_task(void)
{
jl_ptls_t ptls = jl_get_ptls_states();
return (jl_task_t*)ptls->current_task;
Comment on lines +11 to +12
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I don't really understand JuliaLang/julia#32812; so the following may be stupid, please forgive me: I am guessing that for some reason there is a problem with directly using ccall on jl_get_current_task, which is why you re-implement it here. Not sure what the problem is, though: Is it an inefficiency? Or wrong behaviour? Or what? (It might be cool to document this here in a quick comment?)

At the same time, this specific function is a source of ABI incompatibility. If you instead did the following, this function would work fine with all Julia versions from 1.3-1.6, regardless of which Julia version was used to produce the binary:

Suggested change
jl_ptls_t ptls = jl_get_ptls_states();
return (jl_task_t*)ptls->current_task;
extern JL_DLLEXPORT jl_value_t *jl_get_current_task(void);
return jl_get_current_task();

Of course if an inefficiency is the reason for this code, then my suggestion makes no sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking further, you dig far more into innards of the TLS and tasks anyway, so I guess this question is moot :-). Except that I still feel a comment explaining its existence beyond just pointing at an issue might be helpful, but I am just watching from the sidelines, so don't hesitate to ignore me on this :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the file from Libtask.jl without looking at the concrete implementation 😄

It seems that the linked PR caused hangs on Julia 1.5: TuringLang/Libtask.jl#59

I agree that the comment could/should be improved. I guess @KDr2 might be able to explain the details here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I didn't dive deeply enough into Julia's internal when I wrote this. At that time I found that calling to jl_get_current_task using ccall didn't work anymore after I updated Julia, then I found the commit(JuliaLang/julia#32812) which caused the issue by using git bisect, the newly added optimization (specialized codegen) for jl_get_current_task made it problematic (crash the process while calling it using call). So I add this new function to keep the old implementation and fix the problem.

}


jl_task_t *jl_enable_stack_copying(jl_task_t *t)
{
if (!t->copy_stack) {
jl_ptls_t ptls = jl_get_ptls_states();
t->copy_stack = 1;
t->bufsz = 0;
memcpy(&t->ctx, &ptls->base_ctx, sizeof(t->ctx));
}
return t;
}

jl_task_t *jl_clone_task(jl_task_t *t)
{
jl_ptls_t ptls = jl_get_ptls_states();
jl_task_t *newt = (jl_task_t*)jl_gc_allocobj(sizeof(jl_task_t)); // More efficient
//jl_task_t *newt = (jl_task_t*)jl_new_task(t->start, t->ssize); // Less efficient
memset(newt, 0, sizeof(jl_task_t));
jl_set_typeof(newt, jl_task_type);
newt->stkbuf = NULL;
newt->gcstack = NULL;
JL_GC_PUSH1(&newt);

newt->state = t->state;
newt->start = t->start;
newt->tls = jl_nothing;
newt->logstate = ptls->current_task->logstate;
newt->result = jl_nothing;
newt->donenotify = jl_nothing;
newt->exception = jl_nothing;
newt->backtrace = jl_nothing;
newt->eh = t->eh;
newt->gcstack = t->gcstack;
newt->tid = t->tid; // TODO: need testing
newt->started = t->started; // TODO: need testing


newt->copy_stack = t->copy_stack;
memcpy((void*)newt->ctx.uc_mcontext, (void*)t->ctx.uc_mcontext, sizeof(jl_jmp_buf));
newt->queue = t->queue;
newt->sticky = t->sticky;

if (t->stkbuf) {
// newt->stkbuf = allocb(t->bufsz);
// newt->bufsz = t->bufsz;
// memcpy(newt->stkbuf, t->stkbuf, t->bufsz);
// workaround, newt and t will get new stkbuf when savestack is called.
t->bufsz = 0;
newt->bufsz = 0;
newt->stkbuf = t->stkbuf;
} else {
newt->bufsz = 0;
newt->stkbuf = NULL;
}

JL_GC_POP();
jl_gc_wb_back(newt);

return newt;
}