Skip to content

Conversation

@kinke
Copy link
Member

@kinke kinke commented Jul 30, 2017

Just the quick first version which can be successfully built (on Win64 at least) after ~2.5 hours of work. :)

@kinke kinke force-pushed the merge-2.075 branch 2 times, most recently from 09d55ee to e425d0c Compare July 30, 2017 00:58
@joakim-noah
Copy link
Contributor

joakim-noah commented Aug 1, 2017

Built this branch and ran the stdlib tests natively on Android/ARM, had two issues:

I'm currently running the dmd-testsuite, will update with the results.

@joakim-noah
Copy link
Contributor

The following tests now fail in the dmd-testsuite on Android/ARM:

  • runnable/sdtor
  • runnable/ldc_github_420
  • compilable/test17502
  • fail_compilation/diag6373
  • fail_compilation/ice13259
  • fail_compilation/fail13902

@kinke
Copy link
Member Author

kinke commented Aug 2, 2017

The tests for std.uri don't compile because it can't implicitly convert char[] to string, reverting the part of dlang/phobos#5409 that amended std.uri.decode got it to build again. Strangely, the build error is reproducible on linux/x64 if I try to build the 2.075.0 version of std.uri's tests using ldc 1.3, but not with dmd 2.073-5.

DMD 2.074.0 errors out for this minimized testcase derived from failing std.uri:324...

string foo()
{
    char[] r;
    return r;
}

@kinke
Copy link
Member Author

kinke commented Aug 2, 2017

But templatizing makes it work for DMD and LDC...

string foo()()
{
    char[] r;
    return r;
}

int main()
{
    auto r = foo();
    return cast(int) r.length;
}

@PetarKirov
Copy link
Contributor

@kinke that's by design - mutable result of pure functions can be converted implicitly to immutable. Templating the function allows the compiler to infer purity.

@kinke
Copy link
Member Author

kinke commented Aug 3, 2017

Thanks Petar. So I guess this is an additional check performed by LDC which needs to take this special case of legal implicit conversion into account.

I lost the colors in the console on Windows, so I probably screwed up somewhere.

@PetarKirov
Copy link
Contributor

@kinke I'm really confused as to why ldc differs from dmd here, as this feature has been in the language for quite a while and much more unittests in phobos should have been failing by now if ldc never supported this feature. For reference:

@kinke
Copy link
Member Author

kinke commented Aug 3, 2017

This is the stack trace for the (front-end) error emitted for -c -unittest ...\uri.d:

ddmd.errors.verrorPrint(const ddmd.globals.Loc * p2, int p1, const char * ap, const char * format, char * header, const char * headerColor, const char * headerColor) Line 148	D
ddmd.expression.Expression.error(const char * format) Line 2643	D
ddmd.dcast.implicitCastTo.ImplicitCastTo.visit(ddmd.expression.Expression *) Line 104	D
ddmd.visitor.Visitor.visit(ddmd.expression.SymbolExp * e) Line 875	D
ddmd.visitor.Visitor.visit(ddmd.expression.VarExp * e) Line 885	D
ddmd.expression.VarExp.accept(ddmd.visitor.Visitor * v) Line 6711	D
ddmd.dcast.implicitCastTo(ddmd.expression.Expression * e, ddmd.dscope.Scope * sc, ddmd.mtype.Type * t) Line 167	D
ddmd.func.FuncDeclaration.semantic3(ddmd.dscope.Scope *) Line 3785	D
ddmd.dtemplate.TemplateInstance.semantic3(ddmd.dscope.Scope *) Line 6823	D
ddmd.dtemplate.TemplateInstance.trySemantic3(ddmd.dscope.Scope * sc2) Line 8438	D
ddmd.dtemplate.TemplateInstance.semantic(ddmd.dscope.Scope * sc, ddmd.root.array.Array!(ddmd.expression.Expression).Array * fargs) Line 9055	D
ddmd.dtemplate.functionResolve(ddmd.declaration.Match * fargs, ddmd.dsymbol.Dsymbol * tthis, ddmd.globals.Loc tiargs, ddmd.dscope.Scope * sc, ddmd.root.array.Array!(ddmd.root.rootobject.RootObject).Array * loc, ddmd.mtype.Type * dstart, ddmd.root.array.Array!(ddmd.expression.Expression).Array * dstart) Line 2915	D
ddmd.func.resolveFuncCall(ddmd.globals.Loc loc, ddmd.dscope.Scope * sc, ddmd.dsymbol.Dsymbol * s, ddmd.root.array.Array!(ddmd.root.rootobject.RootObject).Array * s, ddmd.mtype.Type * tiargs, ddmd.root.array.Array!(ddmd.expression.Expression).Array * tiargs, int tthis) Line 4305	D
ddmd.expression.CallExp.semantic(ddmd.dscope.Scope * sc) Line 10338	D
ddmd.expression.AssignExp.semantic(ddmd.dscope.Scope * sc) Line 12970	D
ddmd.statementsem.StatementSemanticVisitor.visit(ddmd.statement.ExpStatement *) Line 99	D
ddmd.statement.ExpStatement.accept(ddmd.visitor.Visitor * v) Line 709	D
ddmd.statementsem.StatementSemanticVisitor.visit(ddmd.statement.CompoundStatement *) Line 3581	D
ddmd.statement.CompoundStatement.accept(ddmd.visitor.Visitor * v) Line 896	D
ddmd.func.FuncDeclaration.semantic3(ddmd.dscope.Scope *) Line 1604	D
ddmd.dmodule.Module.semantic3(ddmd.dscope.Scope *) Line 1182	D
ddmd.mars.mars_mainBody(ddmd.root.array.Array!(const(char)*).Array * files, ddmd.root.array.Array!(const(char)*).Array * files) Line 1620	D
cppmain(int argc, char * * argv) Line 1069	C++
D main(...) Line 36	D

No obvious LDC specifics in there, so no idea why we diverge here.

@kinke
Copy link
Member Author

kinke commented Aug 3, 2017

Okay, it's just that we have our own LDC-specific core.stdc.stdlib.alloca, which isn't pure yet as the upstream version... ;)

@kinke
Copy link
Member Author

kinke commented Aug 3, 2017

Looking pretty good already, interpreting the Travis LLVM 4.0.1 Linux x64 job:

  • All LDC and stdlib unittests are green.
  • Single failing lit-test codegen/vector_init.d: vector related.
  • dmd-testsuite failures:
    • vector related: runnable/ldc_github_420.d, runnable/testargtypes.d
    • compilable/test17502.d: possibly vector related
    • runnable/sdtor.d
    • runnable/traits_getPointerBitmap.d
    • fail_compilation/{diag6373,ice13259}.d: missing backticks in error msg (edit: fixed)

Note that I simply allowed all vector ops in new Target::isVectorOpSupported() for now.

For DMD's new syntax highlighting. I grepped in all .{h,cpp} files only.
kinke added 3 commits August 4, 2017 02:03
I don't know how much sense that makes, as the front-end expects a result
expression of a single bool.
LLVM returns a vector of i1 values, the pair-wise results.
From my experience with SIMD on x86_64, what's mostly needed is a vector
bit mask, as that's what the CPU returns and which is later used to mask
accesses/writes.

Anyway, due to new `Target.isVectorOpSupported()` simply allowing all ops,
(not-)equality comparisons of vectors now land here, and I reduce the
pair-wise results via integer bitcast and an additional integer
comparison.
Conflicts:
	runtime/druntime
@kinke
Copy link
Member Author

kinke commented Aug 4, 2017

Update, remaining failures based on Semaphore (with enabled assertions; Travis only shows 2 remaining dmd-testsuite failures) and my local tests on Win64:

  • runnable/sdtor.d(2842): Assertion failure
  • runnable/test12.d: LLVM assertion /build/llvm/include/llvm/CodeGen/SelectionDAGNodes.h:948: llvm::SDValue::SDValue(llvm::SDNode*, unsigned int): Assertion '(!Node || !ResNo || ResNo < Node->getNumValues()) && "Invalid result number for the given node!"' failed.
  • compilable/test17502.d: LDC assertion ../gen/llvmhelpers.cpp:1845: DValue* makeVarDValue(Type*, VarDeclaration*, llvm::Value*): Assertion 'isIrVarCreated(vd) && "Variable not resolved."' failed.
  • compilable/test17352.d: LLVM assertion /build/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp:1213: virtual void llvm::DwarfDebug::endFunction(const llvm::MachineFunction*): Assertion '!FnScope || SP == FnScope->getScopeNode()' failed.
  • Phobos unittests:
    • std.datetime.stopwatch features brittle unittests for a benchmarking helper function. With enabled optimizations, that code may run too fast (duration = 0), probably because it's all optimized away. Happens for Semaphore (Travis is too slow of course => no problems :D) and on my Win64 box.
    • Windows-specific: std.datetime.timezone fails with enabled optimizations on my Win64 box, due to an apparently invalid registry key handle when enumerating the time zones.

Oh and there's also druntime's core.thread-debug failing most of the time on my box due to an infinite recursion in fiber_switchContext() IIRC, but always working when I attach a debugger. Not specific to 2.075 and not bugging AppVeyor apparently.

@kinke
Copy link
Member Author

kinke commented Aug 6, 2017

runnable/sdtor.d:2842 apparently fails due to a missed opportunity for in-place construction.

compilable/test17502.d fails due to an implicit __result variable in an out contract (no out (result) { ... }, but out { assert(__result); }) with the method return type being auto. That special result variable seems to be wrongly detected as nested variable, while it is/should actually be passed as by-ref argument when calling the contract.

gen/toir.cpp Outdated
return true;
}
// DMD issue 17457: detect structliteral.ctor(args)
else if (ce->e1->op == TOKdotvar) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else is meaningless as the previous if ends in return anyway. Ditching it also makes the comment indentation less awkward.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad these commits get actually looked at. ;) - Will address that tiny nit.

Any chance you could help out with the contract thing?

Fixes runnable/sdtor:2842 which tests this for return expressions (in a
function using sret).
Conflicts:
	runtime/druntime
	runtime/phobos
@PetarKirov
Copy link
Contributor

PetarKirov commented Oct 1, 2017

DIP1000 is not consistent with the current meaning of scope

On the contrary, I think DIP1000 is necessary to make using scope classes safe. I don't see any contradiction.

For example, https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md#scope-function-returns contradicts the spec https://dlang.org/spec/attribute.html#scope regarding scope on function return values.

How so? That part of the spec talks only about scope classes. With or without DIP1000 you can't return the address of a local variable (at least you shouldn't be allowed in @safe code). scope return is about allowing to return pointer function parameters, which is safe as long as the caller knows that an argument passed to scope return parameter may be returned (but not otherwise escaped).

@rainers
Copy link
Contributor

rainers commented Oct 1, 2017

DIP1000 is not the spec yet. Regarding classes the spec is pretty explicit that "scope is used to guarantee cleanup."

@PetarKirov
Copy link
Contributor

Walter also added a workaround to allow scope delegates to keep their existing semantic with -dip1000.

I'm not sure if scope delegate parameters are now working safely, but the idea is that the closure environment (context) is captured and allocated on the stack of the caller and the callee receives essentially a pointer of local variable, which is safe as long as the pointer is not escaped.

The stack-allocation is a transparent optimization which should not affect the behavior of the caller or callee. Whether it happens or not can be thought of as implementation-defined behavior. DIP1000 just now enforces the safety.

DIP1000 is not the spec yet

De jure it is not, de facto it is. The reason Walter did not put it there yet is that he wanted to right down all the changes to @safe as a coherent whole. I disagree that this is right approach, as it leads to confusion about the current meaning of @safe and scope in the meantime (several releases!) but that's where we're at today.

Regarding classes the spec is pretty explicit that "scope is used to guarantee cleanup."

That's about initialization - scope Class c = new Class() - this does not affect assignment, otherwise you'll end up with use after free, as horrid behavior that this test case used to exhibit: https://github.com/dlang/dmd/pull/6826/files#diff-1147486c08c85586a3015dd70e5b3c9aR14

@rainers
Copy link
Contributor

rainers commented Oct 1, 2017

DIP1000 just now enforces the safety.

DIP1000 changes the scope from the delegate to the variable holding the delegate, that's why a workaround was necessary: https://github.com/dlang/dmd/blob/master/src/ddmd/mtype.d#L6222

De jure it is not, de facto it is.

DIP1000 might be the future, but it is hidden behind -dip1000 which probably only few people use. It's still too broken, it can even causes code generation errors, e.g. https://issues.dlang.org/show_bug.cgi?id=17449, so I wouldn't call it the de facto spec yet.

That's about initialization - scope Class c = new Class() - this does not affect assignment,

There is nothing in the current spec that says so.

otherwise you'll end up with use after free, as horrid behavior that this test case used to exhibit: https://github.com/dlang/dmd/pull/6826/files#diff-1147486c08c85586a3015dd70e5b3c9aR14

Unsafe, but not necessarily a bug. You only add scope to more variables because of DIP1000, though.

I think changing semantic without a changelog entry is a regression, so I'll file a bug report with the example above. Let's hear what the scope experts say...

@rainers
Copy link
Contributor

rainers commented Oct 1, 2017

I'll file a bug report

https://issues.dlang.org/show_bug.cgi?id=17869

BTW: the code in #2252 (comment) does fail to compile with -dip1000.

@kinke
Copy link
Member Author

kinke commented Oct 1, 2017

After dlang/phobos#5749 for Windows, we'll only have to take care of an apparent LLVM debuginfo inlining bug to be fully green. I compiled the earlier code snippet with current LDC master and this PR using a common LLVM 5.0. LDC master works with -g -enable-inlining, this PR asserts (log output with LLVM 4.0.1 which also fails):

/build/llvm/include/llvm/CodeGen/SelectionDAGNodes.h:948: llvm::SDValue::SDValue(llvm::SDNode*, unsigned int): Assertion `(!Node || !ResNo || ResNo < Node->getNumValues()) && "Invalid result number for the given node!"' failed.

The base IR (without inlining) only differs in one function which has (nicely) changed due to befd05e (in-place construction):

 define weak_odr void @_D7xtest4617__T3mapVAyaa1_61Z3mapFNaNbNiNfAdZS7xtest4617__T3mapVAyaa1_61Z3mapFAdZ6Result(%"xtest46.map!\22a\22.map.Result"* noalias sret align 8 %.sret_arg, { i64, double* } %r_arg) #0 comdat !dbg !46 {
   %r = alloca { i64, double* }, align 8           ; [#uses = 2, size/byte = 16]
-  %.structliteral = alloca %"xtest46.map!\22a\22.map.Result", align 8 ; [#uses = 3, size/byte = 8]
   store { i64, double* } %r_arg, { i64, double* }* %r, !dbg !51 ; [debug line = 3:10]
   call void @llvm.dbg.declare(metadata { i64, double* }* %r, metadata !50, metadata !36), !dbg !51 ; [debug line = 3:10] [debug variable = r]
-  %1 = getelementptr inbounds %"xtest46.map!\22a\22.map.Result", %"xtest46.map!\22a\22.map.Result"* %.structliteral, i32 0, i32 0 ; [#uses = 1, type = i8**]
+  %1 = getelementptr inbounds %"xtest46.map!\22a\22.map.Result", %"xtest46.map!\22a\22.map.Result"* %.sret_arg, i32 0, i32 0 ; [#uses = 1, type = i8**]
   store i8* null, i8** %1, !dbg !52               ; [debug line = 12:9]
   %2 = load { i64, double* }, { i64, double* }* %r, !dbg !52 ; [#uses = 1] [debug line = 12:9]
-  %3 = call %"xtest46.map!\22a\22.map.Result"* @_D7xtest4617__T3mapVAyaa1_61Z3mapFAdZ6Result6__ctorMFNaNbNcNiNfAdZS7xtest4617__T3mapVAyaa1_61Z3mapFAdZ6Result(%"xtest46.map!\22a\22.map.Result"* nonnull returned %.structliteral, { i64, double* } %2) #0, !dbg !52 ; [#uses = 0] [debug line = 12:9] [display name = xtest46.map!"a".map.Result.this]
-  %4 = bitcast %"xtest46.map!\22a\22.map.Result"* %.sret_arg to i8*, !dbg !52 ; [#uses = 1] [debug line = 12:9]
-  %5 = bitcast %"xtest46.map!\22a\22.map.Result"* %.structliteral to i8*, !dbg !52 ; [#uses = 1] [debug line = 12:9]
-  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %4, i8* %5, i64 8, i32 1, i1 false), !dbg !52 ; [debug line = 12:9]
+  %3 = call %"xtest46.map!\22a\22.map.Result"* @_D7xtest4617__T3mapVAyaa1_61Z3mapFAdZ6Result6__ctorMFNaNbNcNiNfAdZS7xtest4617__T3mapVAyaa1_61Z3mapFAdZ6Result(%"xtest46.map!\22a\22.map.Result"* nonnull returned %.sret_arg, { i64, double* } %2) #0, !dbg !52 ; [#uses = 0] [debug line = 12:9] [display name = xtest46.map!"a".map.Result.this]
   ret void, !dbg !52                              ; [debug line = 12:9]
 }

@kinke
Copy link
Member Author

kinke commented Oct 4, 2017

@dnadlinger
Copy link
Member

Does the line info attached to both the call and the function body look sane? Previously, we had issues where no line information was set on some call instruction, which led to assertions after inlining since the inliner code didn't expect that. Potentially something similar is going on here?

@kinke
Copy link
Member Author

kinke commented Oct 5, 2017

The debuginfos look okayish; all calls have line infos. What looks wrong/suboptimal is the line info for the implicit LL ret of the empty struct ctor (it's set to where the declaration begins; the closing brace is 2 lines down). So implicit this, the ctor param and ret have an identical line+column info.

@kinke
Copy link
Member Author

kinke commented Oct 5, 2017

I'm leaning towards disabling runnable/xtest46 for now for the CI systems using an LLVM with enabled assertions (i.e., AppVeyor and Semaphore) in order not to delay this any longer. Objections/better ideas?

Edit: It's Semaphore only, as the assertion isn't triggered for MSVC targets with LLVM < 5.

@dnadlinger
Copy link
Member

@kinke: I'd say just comment out the respective test, and create a ticket for it to be resolved before the final release.

@kinke
Copy link
Member Author

kinke commented Oct 6, 2017

to be resolved before the final release

I'm not that optimistic, but we'll see. ;) - I upgraded SemaphoreCI to LDC-LLVM 5.0.0 as well.

It was previously untested by AppVeyor (as it requires LLVM 5.0+).
@kinke
Copy link
Member Author

kinke commented Oct 6, 2017

Dammit, running out of (RAM?) disk space for Semaphore with LLVM 5, even when building with 2 jobs only... edit: down to 4.0.1 again.

This inlining test now works as expected on Win32 with LLVM 5. XFAIL was
the wrong choice here, as the test fails if it does NOT actually fail.
@joakim-noah
Copy link
Contributor

Looks like all CIs pass, with the only failure on Travis with llvm 3.7.1 that times out for some reason. Maybe it's time to merge this into master and let more people bang on it with a beta release?

@kinke kinke merged commit 7372432 into master Oct 7, 2017
@kinke kinke deleted the merge-2.075 branch October 7, 2017 11:31
@JohanEngelen
Copy link
Member

@kinke Great work!

@joakim-noah
Copy link
Contributor

Ran the master stdlib and dmd tests on Android/ARM, only issue is the std.stdio failure when running all the tests. I need to look into that.

@joakim-noah
Copy link
Contributor

joakim-noah commented Oct 9, 2017

Rather than step through the std.stdio issue with a debugger, I kicked off some test builds on a linux/x64 VPS to see where it may be coming from. Turns out it comes from upstream, as both dmd 2.074.1 and 2.075.1 show the same failure, though I cannot reproduce with ldc 1.4. However, I think Oleg saw it on a RPi 2 with ldc 1.4, so maybe it depends on the toolchain used.

This doesn't surface in the CI for dmd or ldc because they don't just run the phobos test runner with no arguments so it runs all the tests at once, but pass each test to the runner separately and this problem doesn't show up that way.

I also looked at the issue where phobos2-test-runner hangs after running all the tests, which I've seen on linux/x64 for some time and Oleg saw on linux/armhf too. Turns out it goes away if the druntime tests are not run, which I got by changing this line to link against libdruntime-ldc.a instead. There's no reason to include the druntime tests in phobos2-test-runner, but we should probably figure out why it hangs before removing them.

Running all the tests from one run of the test runner is a good stress test of running a lot of the stdlib code: it looks like we have a couple issues that are not exposed by our CI or upstream, because they don't run the test runner that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants