Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to compile a simple Swift file (maybe not possible?) #2427

Closed
geelen opened this issue Jun 14, 2014 · 106 comments
Closed

Unable to compile a simple Swift file (maybe not possible?) #2427

geelen opened this issue Jun 14, 2014 · 106 comments
Labels

Comments

@geelen
Copy link

geelen commented Jun 14, 2014

Hi team, playing around with Emscripten in earnest for the first time, not my usual area of work so apologies if this is clearly never going to work. What I'd like to do is compile a simple Swift program to JS:

println("Hello, world!")

So far what I've tried is:

xcrun swift hello_world.swift -emit-bc -o hello_world.bc
emcc hello_world.bc

but that fails because it can't find certain symbols (looks like the standard library isn't present):

Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Drilling a bit further, and looking at how xcrun swift -v hello_world.swift actually works, I'm able to use this linking command to compile the bitcode to an executable using normal ld.

cp /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libswift_stdlib_core.dylib .
/usr/bin/ld hello_world.bc \
              -lSystem -arch x86_64 \
              -L . -rpath . \
              -o hello_world
./hello_world
=> Hello, world!

So, it seems like libswift_stdlib_core.dylib is the only dependency for hello_world.bc to be properly linked to an executable.

But I'm stuck - is there some equivalent between the -L and -rpath flags on ld that I should be passing to emcc? Or is a dylib like that not possible to be used in emscripten? The final command I tried was:

> emcc libswift_stdlib_core.dylib hello_world.bc                                 ~/src/experiments/swift.js • 2.1.0p0
WARNING  root: emcc: cannot find library "swift_stdlib_core"
Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Hopefully something's possible from here - Swift is a neat high-level language with a nice type system and no garbage collector, which makes me think it's a good fit for compiling to JS.

@juj
Copy link
Collaborator

juj commented Jun 14, 2014

Unfortunately it is not possible to link .dylib files to Emscripten. The reason for that is that the .dylibs already contain native machine code for x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form again. What one would have to do is implement the standard library for Swift and compile that in. The unrecognized struct value errors sounds like something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions with your own implementation?

@geelen
Copy link
Author

geelen commented Jun 14, 2014

Interesting, I suspected as much, but had hoped dylibs might have llvm
bitcode in there.

I'll take a look to see how much of the Swift standard library source is
available to compile from, but I fear it won't be enough.

Appreciate the response, thanks!

On Saturday, June 14, 2014, juj notifications@github.com wrote:

Unfortunately it is not possible to link .dylib files to Emscripten. The
reason for that is that the .dylibs already contain native machine code for
x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form
again. What one would have to do is implement the standard library for
Swift and compile that in. The unrecognized struct value errors sounds like
something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions
with your own implementation?


Reply to this email directly or view it on GitHub
#2427 (comment).

@iongion
Copy link

iongion commented Oct 5, 2014

What if we implement our own standard lib, according to http://practicalswift.com/2014/06/14/the-swift-standard-library-list-of-built-in-functions there are just 74 functions

@JoshCheek
Copy link

If the println command was omitted, would this work? (it didn't for me, I still got the same result, but I'm very ignorant) Then one could create a library and just call it from js without it depending on anything.

Also, an update, the commands in the first post seem deprecated, here is what I tried:

Versions

$ xcrun swiftc --version
Swift version 1.0 (swift-600.0.45.3.2)
Target: x86_64-apple-darwin13.2.0

$ emcc --version
emcc (Emscripten GCC-like replacement) 1.25.0 ()
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The file to compile:

$ cat f.swift
var message = 1

The IR

$ xcrun swiftc -emit-ir f.swift
; ModuleID = '-'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

%Si = type <{ i64 }>

@_Tv1f7messageSi = global %Si zeroinitializer, align 8

define internal void @top_level_code() {
entry:
  store i64 1, i64* getelementptr inbounds (%Si* @_Tv1f7messageSi, i32 0, i32 0), align 8
  ret void
}

define i32 @main(i32 %argc, i8** %argv) {
entry:
  %0 = call i8* @_TFSsa6C_ARGCVSs5Int32()
  %1 = bitcast i8* %0 to i32*
  store i32 %argc, i32* %1
  %2 = call i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()
  %3 = bitcast i8* %2 to i8***
  store i8** %argv, i8*** %3
  call void @top_level_code()
  ret i32 0
}

declare i8* @_TFSsa6C_ARGCVSs5Int32()

declare i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()

!llvm.module.flags = !{!0, !1, !2, !5, !6, !7, !8}

!0 = metadata !{i32 2, metadata !"Dwarf Version", i32 3}
!1 = metadata !{i32 1, metadata !"Debug Info Version", i32 1}
!2 = metadata !{i32 6, metadata !"Linker Options", metadata !3}
!3 = metadata !{metadata !4}
!4 = metadata !{metadata !"-lswiftCore"}
!5 = metadata !{i32 1, metadata !"Objective-C Version", i32 2}
!6 = metadata !{i32 1, metadata !"Objective-C Image Info Version", i32 0}
!7 = metadata !{i32 1, metadata !"Objective-C Image Info Section", metadata !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!8 = metadata !{i32 4, metadata !"Objective-C Garbage Collection", i32 0}

The SIL (IDK if it's useful, but can't hurt):

$ xcrun swiftc -emit-sil f.swift
sil_stage canonical

import Builtin
import Swift
import SwiftShims

var message: Int

// top_level_code
sil private @top_level_code : $@thin () -> () {
bb0:
  %0 = global_addr #message : $*Int               // user: %3
  %1 = integer_literal $Builtin.Word, 1           // user: %2
  %2 = struct $Int (%1 : $Builtin.Word)           // user: %3
  store %2 to %0 : $*Int                          // id: %3
  %4 = tuple ()                                   // user: %5
  return %4 : $()                                 // id: %5
}

// Swift.Int._convertFromBuiltinIntegerLiteral (Swift.Int.Type)(Builtin.Int2048) -> Swift.Int
sil public_external [transparent] @_TFSi33_convertFromBuiltinIntegerLiteralfMSiFBi2048_Si : $@thin (Builtin.Int2048, @thin Int.Type) -> Int {
bb0(%0 : $Builtin.Int2048, %1 : $@thin Int.Type):
  %2 = builtin_function_ref "s_to_s_checked_trunc_Int2048_Word" : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %3
  %3 = apply %2(%0) : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %4
  %4 = tuple_extract %3 : $(Builtin.Word, Builtin.Int1), 0 // user: %5
  %5 = struct $Int (%4 : $Builtin.Word)           // user: %6
  return %5 : $Int                                // id: %6
}

Trying to compile with emcc:

$ xcrun swiftc -emit-bc f.swift -o f.bc
$ emcc f.bc
WARNING: Linking two modules of different data layouts: '/Users/josh/.emscripten_cache/libc.bc' is 'e-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-p:32:32:32-v128:32:128-n32-S128' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'
WARNING: Linking two modules of different target triples: /Users/josh/.emscripten_cache/libc.bc' is 'asmjs-unknown-emscripten' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'x86_64-apple-darwin13.2.0'
Unknown specifier in datalayout string
UNREACHABLE executed at /Users/clb/emscripten-fastcomp/lib/IR/DataLayout.cpp:300!
0  opt                      0x00000001086d04ae llvm::sys::PrintStackTrace(__sFILE*) + 46
1  opt                      0x00000001086d07bb PrintStackTraceSignalHandler(void*) + 27
2  opt                      0x00000001086d0b4c SignalHandler(int) + 412
3  libsystem_platform.dylib 0x00007fff8b0e35aa _sigtramp + 26
4  libsystem_platform.dylib 0x00007fff6492d380 _sigtramp + 3649347056
5  opt                      0x00000001086d07eb raise + 27
6  opt                      0x00000001086d08a2 abort + 18
7  opt                      0x000000010865a7a6 llvm::llvm_unreachable_internal(char const*, char const*, unsigned int) + 198
8  opt                      0x0000000108416b74 llvm::DataLayout::parseSpecifier(llvm::StringRef) + 2804
9  opt                      0x0000000108415c57 llvm::DataLayout::init(llvm::StringRef) + 471
10 opt                      0x000000010749b47e llvm::DataLayout::DataLayout(llvm::StringRef) + 158
11 opt                      0x0000000107482ba5 llvm::DataLayout::DataLayout(llvm::StringRef) + 37
12 opt                      0x000000010747943c main + 3756
13 libdyld.dylib            0x00007fff8a2865fd start + 1
Stack dump:
0.  Program arguments: /Users/josh/code/emsdk_portable/clang/e1.25.0_64bit/opt /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc -strip-debug -internalize -internalize-public-api-list=main,malloc,free -globaldce -pnacl-abi-simplify-preopt -pnacl-abi-simplify-postopt -enable-emscripten-cxx-exceptions -o /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc.opt.bc
Traceback (most recent call last):
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/emcc", line 1224, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/tools/shared.py", line 1357, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

@JoshCheek
Copy link

If anyone can look at the above and give me an idea of what specifically it's missing, ie "it's trying to link stdlib.swift" and a link to somewhere that someone documents how they solved a similar problem, then I'll put a few hours into trying to get it. If there's any reasonable progress in that time, I'll report it here and probably keep at it.

Right now I'm just so ignorant that I don't even know what what the problem is or what to look for in terms of resolving it. But I have like 50 options I'm considering for my project, including things like Elm which I've already put a few days into playing with, so I need some guidance to not wander in circles for a months.

@kripken
Copy link
Member

kripken commented Oct 20, 2014

The targets need to match. As the IR shows, the swift frontend emits

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

Emscripten uses its own triple (asmjs-unknown-emscripten). You might be able to force the frontend to use it? If not, you might be able to just edit the LLVM IR file in text format to fix it (create one using emcc, to see the right values for those two lines). But, this is risky, as the triple affects stuff that is baked into the output in various places...

@JoshCheek
Copy link

So, I'm ignorant enough that I don't know how to interpret "the targets need to match" (I don't know what "target" means, or what a "triple" is -- kinda sounds like a fixed-length set of memory, like a "tuple" in a functional language). So it's unclear to me if it's something that I fucked up (e.g. did I compile it wrong, or with my own version of some binary, since emscripten uses its own set of binaries -- e.g. python2 was not in the PATH, and I couldn't find it in the emsdk_portable, so I just made a bash script named python2 that then invoked python and forwarded the args. Seemed reasonable since I had Python 2.7.5 installed, which the readme alleged was compatible https://github.com/kripken/emscripten/blob/06961a0ef6e3d8d92d5e36ff904262fefec62bec/tests/python/readme.md, and xcrun is some binary that I don't understand, which is just sort of generally available, but didn't come from emscripten) or is it some issue caused by Swift being overly zealous about an OSX environment, or is it some flag to find to tell it to chill the fuck out and forget about being execuatble and just compile as if some other program will load it and deal with all the integration stuff? I mean, you'll notice all I do in this script is declare a variable and assign it the value 1, so what is it trying to dynamically load? And does that have anything to do with the "targets" not matching?

I'll create the file using emcc and diff them and spend an hour poring over the differences, confused and trying and failing repeatedly, to figure out what I need to do, but I don't know how. The bc format si the only one I've found so far that emcc will even consider trying to compile.

The reality is that I have no particularly good model for figuring out how this stuff works. I once spent 8 hours getting a C program to compile and load a library I was interested in, that's about the extent of my experience. Code that operates at this level operates under a model that I am simply ignorant of, so while we're in this domain, it's reasonable to me like I've never programmed before.

I'm actually pretty good at guessing and cursing repeatedly trying and failing and guessing and cursing again, but I need some sort of model or context to iterate upon. And there's a sufficiently large amount of information available out there, that I need someone who's familiar with this domain to clue me in, and imply to me that this isn't a giant waste of my time (b/c, lets be honest, I could just suck it up and write my shit in JavaScript).

@kripken
Copy link
Member

kripken commented Oct 21, 2014

A target is what clang/LLVM is emitting code for. When you build normally, on OSX your target triple is something like darwin-intel-apple or some other combo that contains the information that you are building for intel hardware, to run on a darwin kernel on an apple userspace. This affects codegen in various ways, and the LLVM IR is not portable because of it.

Emscripten has its own target, and emcc tells clang and LLVM to use it, -triple=asmjs-unknown-emscripten or something like that.

@JoshCheek
Copy link

Hmm. Guess I thought LLVM was the target.

Anyway, asmjs-unknown-emscripten looks correct:

$ ag asmjs-unknown-emscripten | wc -l
     115

-triple isn't the correct flag, apparently, and it tells me the wrong help file:

$ xcrun swiftc -Xllvm "-triple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '-triple=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-spiller=asmjs-unknown-emscripten'?

$ swift '(LLVM option parsing)' -help
fish: Unknown command 'swift'

$ xcrun swift '(LLVM option parsing)' -help
<unknown>:0: error: no such file or directory: '(LLVM option parsing)'

$ xcrun swift 'LLVM option parsing' -help
<unknown>:0: error: no such file or directory: 'LLVM option parsing'

$ xcrun swift 'option parsing' -help
<unknown>:0: error: no such file or directory: 'option parsing'

llvm isn't a binary, but tab-complete suggestions include llvm-gcc and llvm-g++, their help screens don't mention a triple, but do talk about a --target, but that doesn't work either. Which is strange, because it's listed in the help output, so it seems like the error should be something about using it wrong, rather than "Unknown command line argument", so maybe this isn't the binary that it invokes for llvm:

$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '--target=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-stats=asmjs-unknown-emscripten'?

# same output for each of these potential variations:
$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target" -Xllvm "asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# for that last one, it says "swift (LLVM option parsing): Did you mean '-stats'?"
# so lets verify that llvm-gcc is the right binary by seeing if `-stats` is one of its options:
$ llvm-gcc --help | grep stats

# ...nope, what the fuck is llvm?
# searching implies its binary is named "clang" for some reason,
# but no dice here either:
$ clang --help | grep stats

# man pages have any ideas?
$ man llvm-gcc
No manual entry for llvm-gcc

$ man llvm-g++
No manual entry for llvm-g++

$ man clang # this one works!

# After looking through this man page, I try -arch, which doesn't do shit
# was going to try setting MACOSX_DEPLOYMENT_TARGET env var
# but then I realized there's some way to search all man pages for "-stats"
# some stupid searching (man man) eventually I figure out:
$ man -K stats

# go through this list, and `gcc-4.8` shows up, so maybe llvm is just gcc? *shrug* lets try it
# it turns out to be fruitless
# at this point, I'm considering trying to find in the emscripten code where they invoke it
# figure I'll try googling a bit, find some docs for `llc`, which isn't a binary on my system, but does have a `-mtriple` flag, so lets try that:
$ xcrun swiftc -Xllvm "-mtriple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# run the gamut of possible ways that I'm supposed to pass this thing
# (why can't I find a fucking example of how to do this?)

# eventually, one of the options suggests "-spiller" as a correction possibility
$ man -K spiller  # ...just some thing about crypto

At this point, documenting in code would add another hour, but many searches, fancy -k and -K uses of the man flag. A google search for llvm -spiller and -stats takes me back to that llc page, look through it a second time, this time notice that they have two dashes before them, even though the suggestion when I got it wrong only has 1 dash. So, try the permutations of possible ways to pass an argument to -Xllvm with two dashes. None of them work.

Get pissed, internet search like the fourth time for an example of how to use -Xllvm flag, no results, cd to xcode and search for the string 'Xllvm', nothing. Try emsdk_portable, nothing.

Look one more time at llvm-gcc, llvm-g++, lldb, apropos llvm, clang, nothing.

Decide to go see if I can find where this happens in emsdk_portable, looks like the makefiles set a build_triple but never use it. Probably some makefile magic, and I don't feel like reading a book about makefiles just to find out how this gets persisted. Notice some of the files use LLVM_TARGET, I don't know Python, maybe that's an env var, so try compiling with that set env LLVM_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc, but it doesn't do anything (triple is still set to "x86_64-apple-darwin13.2.0"), remember the clang man pages had an env var that seemed like it had potential, so env MACOSX_DEPLOYMENT_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc. Nope.

So, I'm like 3 hours in, and literally no further than when I started. Looks like someone got Rust compiling to asm.js, maybe that's a better choice. Someone got Go compiling directly to JavaScript. Might be sufficient. Could go suffer through the Elm type system for another week or two (it was basically like this: hours of slamming myself into the wall, but I kept at it for like 3 times longer), or maybe write the thing in C, since that at least compiles, but some part of me thinks that's masochism. If I did it in Opal, I wouldn't have to learn a new language... but that's part of the appeal, and I'm not sure it'll be able to scale in directions I might want to go with my project.

Going to go for a drink and weigh the shittiness of C as a language against the fact that I'll almost certainly succeed.

@endash
Copy link

endash commented Feb 20, 2015

I was able to get very, very simple programs to build with the 6.3 beta at least. Unfortunately even something as simple as a string literal will not work, and regardless the resulting file has a bunch of missing symbols. Some of the symbols (swift_once) are defined in libswiftCore.dylib, so it's likely using Swift will be impossible until the standard library is open sourced.

@guidobouman
Copy link

So, luckily, this will help later on: https://developer.apple.com/swift/blog/?id=29

@nmn
Copy link

nmn commented Jul 16, 2015

So I was trying to do the same thing, and I just stumbled upon this Issue. So glad that I won't waste my time again.

By the way did you try again with XCode 7 and the swiftc tool??

@stepanhruda
Copy link

Since Apple now supports LLVM Bitcode, would that enable including dylibs that contain it?

@kripken
Copy link
Member

kripken commented Oct 6, 2015

LLVM bitcode is not portable. Bitcode for the App Store would be specific to Apple's hardware and OS.

@stepanhruda
Copy link

Thanks, that makes sense. In case anyone is interested in more details as I was, LLVM FAQ: Can I compile C or C++ code to platform-independent LLVM bitcode?

@geelen
Copy link
Author

geelen commented Dec 3, 2015

It's been 18 months since I started this issue and I still really want this to happen. With Swift going open-source today let's hope that means it's now possible!

https://github.com/apple/swift 🎉

@kripken
Copy link
Member

kripken commented Dec 4, 2015

This will definitely make it a lot easier! :)

Ok, to move forward here, we need to do the following:

  1. Figure out how to use emscripten's llvm with swift's llvm. Does swift have a fork of llvm, or can it use stock? If the latter, then this should be easy, just make swift use emscripten's llvm. We just need to see which version of llvm it expects (the incoming branch is on trunk llvm from last week, hopefully that is good and swift doesn't need an older tag).
  2. Get swift to emit code with the asmjs-unknown-emscripten triple.
  3. Get swift to stop at the bitcode stage.
  4. Run emcc on that bitcode and see what happens.
  5. We would need similar bitcode of whatever runtime libraries swift requires, except for libc etc. which we already have.

@Gaelan
Copy link

Gaelan commented Dec 4, 2015

Swift does have a fork of LLVM at apple/swift-llvm. Not sure what changes they have made.

@nmn
Copy link

nmn commented Dec 4, 2015

the swift-cli has a bunch of handy options to output just the llvm byte code etc.

@jcampbell05
Copy link

Let me know what I can help with would love to have something working!

@kripken
Copy link
Member

kripken commented Dec 6, 2015

Thanks for the info @Gaelan. Ok, if they have their own fork, then the best workflow would probably be:

  1. Upstream our target triple to LLVM
  2. When they merge in new LLVM, they'll get that
  3. Get Swift to emit bytecode to our target triple
  4. Run that bytecode through emcc

The first step is to upstream our triple, which is already partially there (due to nacl upstreamings), so it's a small patch and probably not controversial. What we need to send is basically this:

https://gist.github.com/kripken/0b7ba068faf21d5449a3

Anyone interested to help upstream that?

@ianyh
Copy link

ianyh commented Dec 7, 2015

It looks like Apple's llvm clone is pretty strict about changes being made upstream unless they are specifically swift-related. See: https://swift.org/contributing/#llvm-and-swift

@jcampbell05
Copy link

Could open an issue in the Swift LLVM github to ask them for changes

@ianyh
Copy link

ianyh commented Dec 7, 2015

I think that reduces to asking them to merge the changes in themselves, which is against their policy. The patch would have to be submitted directly to llvm by someone here.

I have no familiarity with the llvm community, so I don't know how they would feel about, for example, introducing asmjs as an architecture. Anyone else have thoughts?

@sberan
Copy link

sberan commented Dec 7, 2015

This is very exciting @kripken! In theory, should we be able to make these modifications locally and try things out using the swift LLVM clone? I modified the relevant files in the swift LLVM clone, according to your patch - here is the diff of the changes I made: https://gist.github.com/sberan/43bc5fbff78ea47658f0

Unfortunately I don't see a new target for asmjs in the swift recompiled LLVM:

./build/Ninja-DebugAssert/llvm-macosx-x86_64/bin/llc --version
LLVM (http://llvm.org/):
  LLVM version 3.8.0svn
  DEBUG build with assertions.
  Built Dec  6 2015 (19:49:40).
  Default target: x86_64-apple-macosx10.9
  Host CPU: haswell

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_be - AArch64 (big endian)
    arm        - ARM
    arm64      - ARM64 (little endian)
    armeb      - ARM (big endian)
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64

Maybe more work needs to be done here in order to register asmjs as a target? Forgive me if this is a noob question, I'm very new to llvm and emscripten, just very excited to try swift in the browser 😄

I'd be happy to try and push these changes upstream, but I'd like to verify that they'll work first! 😆

@ianyh
Copy link

ianyh commented Dec 7, 2015

I'll try to give it a go when I get a chance!

@kripken
Copy link
Member

kripken commented Dec 8, 2015

I think you might not see a target because you can't build a full executable for js. But you should be able to build with -target asmjs-unknown-emscripten -emit-llvm and it should emit llvm for that specific target triple.

@kripken
Copy link
Member

kripken commented Jan 6, 2016

There is another possible path here. Upstream LLVM has a WebAssembly triple now. It is almost identical to our asm.js triple. It should be possible for swift to build to the wasm-wasm-wasm triple, then import that bitcode into emscripten which can compile it to asm.js.

(We would still need to make sure it's the same LLVM version, or close enough, to avoid e.g. debug info changes. But emscripten merges upstream every week or two now, so that should be easy, if someone tells me what is a good time to merge for Swift, i.e., when Swift merges.)

Note that the wasm triple is still experimental, so you need to build LLVM with it enabled, from the list of experimental targets.

@jcampbell05
Copy link

@kripken - This is an overview of the release process for Swift https://swift.org/blog/swift-2-2-release-process/

@naan
Copy link

naan commented Feb 1, 2016

Swift uses llvm 3.8.0. They have their branch and not exactly sure their policy but once they finish swift 2.2, they'll probably merge their branch to upstream trunk.

3.8.0 already has wasm as kripken said, so I was able to build swift-llvm with wasm64 enabled and pass basic tests, but struggling to build swift with llvm-wasm and build swift stdlibs with with llvm-wasm + emscripten, main reason is that their build system is huge but not so flexible. (There're lots places that have hard-coded config like `if system==darwin then build for Mac, else build linux kind of thing) But I'm new for their build system so I may miss something.

I just wanted to make sure the process I'm doing is in the right course.

  • Build llvm with wasm enabled (already done)
  • Setup emscripten to use llvm-wasm enabled (not the one bundled with emscripten)
  • Build swift with llvm-wasm cross compile enabled, build swift stdlibs with llvm-wasm + emscripten

@pannous
Copy link

pannous commented Oct 30, 2018

error: no such file or directory: 'wasm32-unknown-unknown-wasm'

my ignorant suggestion:
-target=wasm32-unknown-unknown-wasm
or
-target:wasm32-unknown-unknown-wasm
instead of
-target wasm32-unknown-unknown-wasm
?

@MaxDesiatov
Copy link

MaxDesiatov commented Oct 30, 2018

It doesn't look like the argument is passed incorrectly, otherwise I'd expect line-directive or its proxied swiftc invocation to explicitly complain about the incorrect argument.

My current guess is that something's missing in build-script-impl or one of the numerous CMake files and their cross-compilation settings. Those seem to fail to create this directory within the main build directory tree. I plan to have a closer look at this line-directive script and possibly to debug it and swiftc to see where exactly the attempt to address that directory occurs and what's the absolute path of it. Then we'd need to retrace the build scripts to understand what's the best setting to tweak.

@patcheng
Copy link

patcheng commented Nov 1, 2018

nice progress!
according to swiftc -h:

  -sdk <sdk>              Compile against <sdk>

so, looks like <sdk> is missing.

looking at SwiftConfigureSDK.cmake, configure_sdk_windows defines:

    set(SWIFT_SDK_${prefix}_ARCH_${arch}_PATH "/")

but we are not setting the variable in configure_sdk_webassembly.

@MaxDesiatov
Copy link

MaxDesiatov commented Nov 1, 2018

Thank you @patcheng, that's a great shout!

I've fixed that missing directory problem and a few more header import errors following after that in my fork. Here's the error I'm currently stuck at:

[1049/1331] Compiling /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o
FAILED: stdlib/public/core/webassembly/wasm32/Swift.o 
cd /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt -- /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk / -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt
<unknown>:0: error: file '/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm' is not a valid precompiled module file
/Users/maxd/Documents/swift-source/swift/stdlib/public/core/ArrayBody.swift:18:8: error: no such module 'SwiftShims'
import SwiftShims

Looking at this precompiled module file we can see that it's compiled to WebAssembly:

% file module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm
module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm: WebAssembly (wasm) binary module version 0x1 (MVP)

Not sure if that's any good, a bit more googling leads to this clang doc on precompiled modules. There's also a mention of an unmerged patch for WebAssembly precompiled modules here and an unresolved clang bugreport probably related to this problem, both posted by @patcheng by the way 👍

Probably need to try and apply that patch or figure another way to make clang consume these precompiled WebAssembly modules. Or maybe these modules shouldn't be compiled to WebAssembly altogether? 🤔

@aemino
Copy link

aemino commented Dec 22, 2018

Hey @MaxDesiatov, a big thanks for all the work you've been doing on this!

I was wondering what the current status of compiling Swift to WASM is right now. I lack much experience in this area, but I'm very excited to potentially see support for this, so I would be happy to attempt contributing toward these efforts, given some guidance on what needs to be worked on.

Thanks!

@ephemer
Copy link

ephemer commented Dec 27, 2018

I’m pretty sure the precompiled modules should be readable to the cross-compile host (in your case Mac OS?) so shouldn’t be in web assembly format.

This is a common pain when trying to cross-compile generally (host vs target), and the last I looked the swift toolchain was doing weird things in this regard because it doesn’t follow CMake’s standard- Apple was having toolchain build performance issues internally using it in that way.

So to fix your issue it’s possible it’s an LLVM bug but it may also be an issue with the CMake scripts (or what is being passed to them by build-script-impl). I haven’t looked at the build system in a while.

I’m still tied up with some other projects but I do look here periodically and will try to share what limited experience I have with the build system.

@patcheng
Copy link

patcheng commented Jan 2, 2019

I just discovered that @ddunbar has been creating PRs to add WASM support to Swift:

apple/swift-clang#235
swiftlang/swift#20684
swiftlang/swift#20687

@ddunbar
Copy link

ddunbar commented Jan 3, 2019

Just a few, very minimal initial ones... this bug has the status: https://bugs.swift.org/browse/SR-9307

@MaxDesiatov
Copy link

MaxDesiatov commented Feb 12, 2019

Hi all, sorry for the delayed reply. There are few more blocking things that I've discovered while trying to make stdlib and a few basic examples compile:

  1. The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do #include <atomic> or maybe find some libc++ implementation that provides those stubs for us?

  2. The LLVM code that Swift's compiler generates in IRGen also contains instructions for atomics. Those need to be omitted during code generation as they can't be lowered to WASM binary code due to the same reasons mentioned in 1.

  3. Some of the instructions generated by Swift's IRGen also specify memory offsets that can't be lowered to WASM. I didn't have enough time to figure out if that's a temporary limitation of LLVM WASM backend or has to be fixed in Swift's IRGen similar to 2.

Hope this helps.

@MaxDesiatov
Copy link

MaxDesiatov commented Feb 13, 2019

It looks like short-term it would be great if we could push swiftlang/swift#20684 PR towards being merged. Reviewing its code, commenting on the review, upvoting the PR, pinging the authors/reviewers once a week or at least once a month and sharing to your friends so that they do the same would definitely help. And yes, the Apple's contribution guide for the Swift compiler says we can ping people once a week 😃

  • Ping the change. If it is urgent, provide reasons why it is important to get this change landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you’re asking for valuable time from other professional developers.

Here's also quick guide if anyone's interested in investigating the rest of the issues I mentioned above. The gist is that you need to compile a Swift toolchain that supports -target wasm32-unknown-unknown-wasm flag passed to the Swift compiler (which is what swiftlang/swift#20684 is about). It's available in my wasm-next branch here https://github.com/MaxDesiatov/swift/tree/wasm-next, you can clone the dependencies after checking out that main forked repository this way: ./swift/utils/update-checkout --clone --scheme wasm-next.

Or you can try @ddunbar's branch from the PR, but you need to check if it pulls fresh LLVM/Clang when doing update-checkout from that branch.

I haven't compiled any of these branches for quite some time so that's why some rebasing on top of the latest code and resolving conflicts might be needed.

After you've got the toolchain compiling successfully you can try compiling MicroStdlib test:

% ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/swift-macosx-x86_64/bin/swiftc \
-target wasm32-unknown-unknown-wasm -c -force-single-frontend-invocation \
-parse-as-library -parse-stdlib -module-name Swift -emit-module \
-emit-module-path Swift.swiftmodule -o Swift.ll \
swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift

This will generate a Swift.ll file with LLVM instructions that we need to "lower" to WebAssembly binaries. You can even start with commenting out all of the code in swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift and compiling an empty file. If that works, you can uncomment the most simple code line by line to proceed towards next issue.

If all goes well at the previous stage you can then compile Swift.ll file to WebAssembly with ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/llvm-macosx-x86_64/bin/llc Swift.ll. That's where you might stumble upon problems with instructions not supported by the WebAssembly LLVM backend. It looks like most of the time it's Swift compiler generating instructions that don't make sense for WASM (atomics and probably address offsets, but the latter needs to be confirmed). You can also refer to LLVM guide for more details on these instructions.

Instructions that don't make sense should be fixed in IRGen code, otherwise we need to report problems with WebAssembly backend to upstream LLVM, but fixing those locally in forked LLVM repositories in the meantime is fine too. The earlier those problems are reported, fixed and upstreamed to the main LLVM repository (not Apple's), the higher is the chance they end up in the imminent LLVM 8.0 release and we won't need to maintain fixes in our forks anymore.

@jcampbell05
Copy link

  1. The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do #include <atomic> or maybe find some libc++ implementation that provides those stubs for us?

Maybe could be simulated using green threads ?

@MaxDesiatov
Copy link

MaxDesiatov commented Feb 13, 2019

Maybe could be simulated using green threads ?

It's not that the standard library needs green threads or multi-threading to make it work, it's just that reference counting and the rest of the runtime were never written for platforms like WASM (or AVR for example) in mind, so it guards everything with atomics and synchronization assuming you always have to be safe when passing ref-counted objects around. On WASM that's not a problem as long as you don't have threads, we could even throw out all of the atomics and synchronization, the problem would be to keep that forked version maintained. That's why I'd prefer to have the shims compiled conditionally with upstream stdlib source code if/when we have shims available for atomics. And this is what people from the Swift core team recommended to do anyway:

It would be interesting to add a configuration flag to disable threading support in the "normal" runtime and standard library. Nobody's done that yet to my knowledge. For that specific issue, though, it is incorrect to avoid using in a single-threaded system; I'm not sure why that #if is there. is still meaningful for a single-threaded system for atomicity relative to signals and other preemption mechanisms.

It’s unfortunate libc++ doesn’t have a single-threaded atomic implementation, but having a stubbed-out version of atomic, mutex, and other synchronization primitives would be nice to minimize the amount of conditional code we need in the runtime itself.

@therealbnut
Copy link

therealbnut commented Feb 13, 2019

I left a comment on the forum, gist being that wasm has plans for atomics, but not any time soon, so stubbing <atomic> sounds like the most pragmatic choice.

I’d avoiding trying to fork the things to not use atomics in the stdlib/runtime because it’s much more likely to change than atomics, and you can write/maintain NOP functions much more easily than maintaining a fork of the standard library.

@alflennik
Copy link

Thanks for all the hard work trying to get this to work - Swift is a fantastic language that I would love to see running in the browser (although the 5 megabyte bundle size will make me think twice).

@zhuowei
Copy link

zhuowei commented Apr 13, 2019

I spent the past week looking into this: it looks like LLVM/WebAssembly's support for custom data sections isn't mature yet, so Swift's runtime type reflection won't work.

I got to the same stage as @patcheng's and @MaxDesiatov's ports: specifically, I can compile a simple Swift function that doesn't use the stdlib and run it in Firefox.

swiftwasm_first

but I'm stuck here now as well.

I had to comment out a lot of stuff in LLVM to get asserts to stop firing when writing the type information sections to the wasm file.This results in invalid wasm files when I compile any Swift file with a class or struct declaration in it... including the stdlib.

While it's possible to disable runtime type information in Swift (with the -disable-reflection-metadata flag), this would break pretty much everything in the stdlib. (I remember from the Swift 2 era that even println("hello world") would crash without runtime type info)

I know @patcheng sent some patches to LLVM to try to resolve this, but I'm not sure what they do or whether more changes are needed.

I wonder how other language with runtime types support, such as Go or Rust's Any trait, encode their type information in WebAssembly.

@MaxDesiatov
Copy link

MaxDesiatov commented Apr 13, 2019

I wonder if using something like Binaryen instead of LLVM would allow us to write the metadata properly 🤔

I know that Binaryen IR doesn't even use SSA, this would probably require writing whole codegen pass for it, but maybe still worth it? Does anyone here have any experience with Binaryen and can provide their opinion on this?

@sbc100
Copy link
Collaborator

sbc100 commented Apr 15, 2019 via email

@zhuowei
Copy link

zhuowei commented Apr 15, 2019

@sbc100 I don't have a Bugzilla account on LLVM yet; I'll apply for one soon.

In the meantime, here are the issues I ran into, with links to patches/workarounds. I'm planning to send some of these upstream when possible

  • MCBinaryExpr in relocations won't work

Swift has a bunch of data relocations that are (symbol - constant value); this becomes a MCBinaryExpr, which causes asserts when writing object.

patcheng has a patch to check for these at https://reviews.llvm.org/D42564; I have a terrible kludge for fixing them at swiftwasm/llvm-project@d0a183f and swiftwasm/llvm-project@3dc1a43

This one can be worked around by not passing the equivalent of that flag when compiling Swift

  • Comdat with custom sections don't work

Comdat only works when custom sections aren't specified. Comdat in LLVM's Wasm object writer expects unique section name for each comdat symbol; The default section names are ".rodata.(symbol name)", which does work. If I specify a custom name, it doesn't: https://gist.github.com/zhuowei/597b77698634aac9b302d43b5c464838

Worked around by not using Comdat (since Swift disables comdats on Linux/ELF and macOS anyways)

  • No PC-Relative relocations on Wasm

This isn't an LLVM issue; it's just that Swift writes relative addresses in the metadata table to avoid runtime relocations (ie (target address - current address)); this requires a relocation type referencing two symbols, which doesn't exist in Wasm.

I asked a Swift developer on Twitter and it seems the Windows port has some workarounds for this that I could look into.

  • No -gmodules support for precompiled header/modules on Wasm

Patcheng has a patch for this: https://reviews.llvm.org/D42233 but it predated custom sections support, so it was decided to wait for custom sections to be included and switch the patch to use that.

I've addressed the comment and switched that patch to using custom sections: my version of that patch is swiftwasm/swift-llvm@de2966b and swiftwasm/swift-llvm@502e753

  • WebAssembly doesn't have swift calling convention enabled yet

ddunbar added Swift calling convention support to llvm, but it isn't turned on yet (maybe because there are no tests?)

I enabled it at swiftwasm/llvm-project@ea5605a, and I'm currently trying to port the ARM32 swift-return.ll/swifterror.ll/swiftself.ll test cases to WebAssembly so I can submit that patch.

Thank you so much for your work on the WebAssembly backend!

@zhuowei
Copy link

zhuowei commented Apr 20, 2019

@MaxDesiatov A quick update on my progress:

The Stdlib links. ...For certain values of "links": I basically completely broke the metadata support to get it to link. Attempting to run the resulting WebAssembly file gives me:

zhuowei@zhuowei-laptop:~/Documents/wasmtime$ target/release/wasmtime ~/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm
failed to demangle superclass of _ContiguousArrayStorage from mangled name 's28__ContiguousArrayStorageBaseC'
error while processing main module /home/zhuowei/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm: Instantiation error: Trap occurred while invoking start function: wasm trap at 0x7fbe9489677d

Yeah, this needs a lot more work before swift_getTypeByMangledName would run.

All my changes are pushed to the SwiftWasm repo, and I'll setup CI and document how to build it soon.

@kripken
Copy link
Member

kripken commented May 10, 2019

@zhuowei - I'm curious to hear more about your goals for the port. You're using wasi, so is this mostly for non-browser use cases? Is running on the web not a goal? (And, is using an API like OpenGL not a goal?)

@zhuowei
Copy link

zhuowei commented May 11, 2019

@kripken Thank you so much for your comments!

I'm looking to target the browser, with serverside being a nice to have.

I mostly used WASI because it's new and defaults to using LLVM's wasi-ld for linking, instead of Emscripten's bitcode-based emcc linker. (I know Emscripten can use wasm-ld as well, but I think? the standard libraries are all compiled for emcc?) I actually started with a Emscripten sysroot, and it didn't take too many changes to switch to WASI. Thus, I'm sure most of the work done for this port would also be applicable to an Emscripten-based port.

I was planning to modify WASI's browser polyfill for better interop with web APIs and JavaScript (since Emscripten's EM_ASM macros probably won't work in Swift anyways). I understand that Emscripten's the more mature choice right now for interoperating with the Web. I would really appreciate your expertise in what approach I should take to turn Swift into a language for the web.

@kripken
Copy link
Member

kripken commented May 12, 2019

Interesting, thanks @zhuowei!

Yes, emscripten does support wasm-ld and the LLVM wasm backend - basically if you just point emscripten to a plain upstream LLVM build, it will use all that (on linux, you can already try it out using emsdk install latest-upstream). Then it uses wasm object files, links them with wasm-ld, etc. We will switch to that by default soon, actually!

Anyhow, the reason I asked is I'm curious how emscripten can help here. Sounds like using the LLVM wasm backend is a good step we're close to completing. Otherwise, I guess it depends on the applications you want to port. My general thoughts on the topic, for languages using LLVM:

  • If the goal is to do mostly just minimal pure computation on the web, then you don't need wasi or emscripten - you can just use LLVM (and manual wasm imports/exports for JS integration). I assume Swift needs non-trivial standard library support, though, so probably this one isn't relevant.
  • If the goal is server-side code in pure wasm, then wasi is probably best. Or, if you want wasm+JS and/or to use node.js, then emscripten would be better.
  • If the goal is code that uses some low-level libc APIs, then wasi may be good - since that's pretty close to the server-side case anyhow. For code size on the web, though, it will be smaller to use emscripten instead of wasi + the wasi web polyfill. I wonder if emscripten can do something like "port" wasi applications and optimize them? But I'm not sure that can be as good as just recompiling them from source with emscripten. So supporting both wasi and emscripten may be optimal.
  • If the goal is to run on the web and to have natural integration with web APIs, then you probably want something language-specific like emscripten's WebIDL binder or embind for C++, or Rust's wasm-bindgen. (Btw, I hope Swift can do a lot better than any of those, because Swift's memory management is in principle more compatible with the web, being much more automatic.)
  • If the goal is to run on the web and use native C APIs like OpenGL, pthreads, SDL, etc., then emscripten is the best option, as we have support for those in very web-specific ways. For example, emscripten's pthreads use workers, which is necessary on the web (I'd expect wasi's eventual pthreads support to focus more on "normal" native pthreads support, the way it works on servers, which is very different); also emscripten has an SDL2 port, optimized WebGL bindings for OpenGL, etc. etc.

Since you say your main goal is the browser, then maybe emscripten has a role to play here - let me know what you think. We'd definitely like to help here if it's relevant!

Btw, a question about your Swift PR: I see it has ifdefs on both __wasi__ and __wasm__. The wasm ones would work with both wasi and emscripten (and any other target that uses the LLVM wasm backend), so using the wasi one only where absolutely necessary could make it easier to use emscripten or other things later - maybe that's already how it's written?

@zhuowei
Copy link

zhuowei commented May 12, 2019

@kripken I did see the progress on the Wasm-ld stuff: I honestly was just trying to be hipster by using the latest shiny SDK...

Minimal pure computation was possible even before we started this port (see my comment above) - most of the work is mainly about getting the full stdlib working.

I originally started by targetting Emscripten, and switching to WASI was pretty much a find-and-replace (since both used Musl), so adding support for Emscripten is definitely something that we'll look into once we get the basic support upstreamed.

Yes, that's what I tried to do for defines: I tried to use the __wasm__ define for changes common to the wasm architecture or the wasm file format, and use __wasi__ for changes specific to the WASI sysroot/musl port.

I named the platform as just "Wasm" in the Swift stdlib, so the conditional compiles there are os(Wasm); I'll probably change it to os(Wasi) there as well to match.

I definitely agree that support for Emscripten would be great for Swift - we're definitely interested in working together to make that possible in the future.

Thank you so much for the comments and for all your work on Emscripten.

@therealbnut
Copy link

@kripken @zhuowei Hi I’m really excited for these developments, but I’ve got a lot of questions. 🎉

I don’t know too much about Emscripten, but I read that Rust chose not to use it because of its weighty runtime (rust-lang/rust#45905).

Do you mind giving a high level overview (or linking) what functionality the runtime provides?

Also, how can we minimize the weight of that runtime? Can we use dead code stripping, LTO, import just what we need from Swift?

Thanks for your insights ♥️

@kripken
Copy link
Member

kripken commented May 12, 2019

@zhuowei Great, let me know when and how we can help :)

@therealbnut Sure, let me try to explain what the extra runtime stuff does here. Maybe a step back first though - note that Rust didn't remove Emscripten support, they added a second option. But yes, that new option is definitely where most of the activity is, as you said. This makes sense because of the different use cases:

  • Emscripten's main use case is existing code in C/C++ that uses native APIs like POSIX, OpenGL, SDL, etc.
  • Rust's main use case is new code written for the web.

This is simply because Rust is a new language - there aren't many large important native applications written in Rust yet. Most relevant applications are currently written in C++ - for example, the Unity and Unreal game engines.

The runtime that emscripten provides is exactly to support POSIX, OpenGL, SDL, etc. It consists of ports of those libraries, of new C APIs to make porting them easy, and of basic building blocks to enable all of that. All of this is very web-specific and includes a lot of complexity because of that - because it's not easy to implement pthreads using Workers or normal OpenGL using WebGL, etc. The web platform is weird!

Note that we already do LTO and other things to reduce code size. We even do dead code elimination across both the wasm and the JS ("metadce"). We also make sure to keep our JS code compatible with closure advanced optimizations for the best JS minification on that side, and use binaryen to minify the wasm. But we can do more, and we do have some unnecessary code in the runtime we'd like to remove (the ongoing MINIMAL_RUNTIME project - help accelerating that is welcome!). But in the end, if an application uses POSIX files for example, then there is just a bunch of code that is unavoidable. But it's definitely worth it if you have a large codebase you want to just work on the web - since the runtime is tiny compared to your own code anyhow, and it saves you porting your code to new APIs. Using the same APIs for native and web also means your code has fewer web-specific differences, one useful consequences of which is you can often debug issues you see on the web version in a native build.

That's how I see the extra value that emscripten brings here. It's incredibly important for C/C++ because existing apps need that, but maybe not for Rust since it's new code anyhow. But with that said, if Rust becomes important in native application development - which seems plausible - then maybe the next 1 million line game engine will be written in Rust, and then emscripten would be the right tool to port it. I don't know enough about the Swift ecosystem to know how it fits in all this, but as discussed above, it should be pretty easy to support both wasi and emscripten, leaving options open for both new code as well as porting native applications.

Btw, emscripten brings another, separate type of value as well - it's mostly the same team of developers that work on emscripten, the LLVM wasm backend, wasm-ld, and binaryen. Since it's the same people, I think it's natural that we do a good job of making emscripten use those other tools in an optimal way. In particular, emscripten makes sure to call wasm-ld and binaryen with carefully-chosen flags for optimizing for code size and speed, uses features like metadce that was mentioned before, etc. So the out-of-the-box code quality - just grab the emsdk and do emcc -Os - is very good with emscripten, while otherwise you'd need to do more toolchain tinkering yourself.

@therealbnut
Copy link

@kripken Awesome, thank you very much for taking the time to respond, that answers a lot of my questions, and some I hadn’t thought of yet :)

@stale
Copy link

stale bot commented May 11, 2020

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

@stale stale bot added the wontfix label May 11, 2020
@stale stale bot closed this as completed May 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests