Enable carthage #282

JonasVautherin · 2018-08-09T13:53:35Z

Opening the work for #13.

Note that there are very few changes in the build system: most of this PR is about archiving the project SwiftGRPC-Carthage.xcodeproj that is generated by make project-carthage.

The issue here is that it won't work unless make is run (and versioned).

Trying the PR:

Pull this branch
Run carthage build --no-skip-current and observe that it fails because files are missing
Run make
Run carthage build --no-skip-current again, and see that it works this time.

JonasVautherin · 2018-08-09T15:13:15Z

@MrMage: I would like to have your opinion about that (just because I have no clue about what files generated by make are necessary for the build), but my feeling is that it will not be possible to version them, and Carthage will probably never support running prebuild scripts (again, my intuition).

This said, I want to emphasize that the Carthage build works. So a last resort would be to provide the required frameworks (BoringSSL, CgRPC, SwiftGRPC, SwiftProtobuf) as "binary-only" in the github releases.

In my understanding (according to the Carthage documentation), it works like this:

Carthage can build the archives
Each framework will need to be in one archive (so we are talking about 4 archives here)

This way, without versioning the .xcodeproj at all (i.e. with minimal changes to the repo), the CI would be able to create the archives and add them to the release, effectively adding support for Carthage. People wanting to build the frameworks themselves (instead of downloading the binaries from github) would need to do it manually.

How does that sound to you?

MrMage · 2018-08-09T15:18:08Z

Releasing binary Swift frameworks doesn’t make sense until module stability is reached by Seift, which comes way after ABI stability. So that’s not an option anytime soon.

In terms of stuff generated by make that would be needed by Carthage, shouldn’t that mostly be just the project file? What about having a test that generates the current Carthage project and diffs it with the currently committed one, failing when differences are detected? We already have a similar test for echo.grpc.swift.

Haven’t looked into the PR code yet, will do that later.

MrMage

To be honest, this looks very good to me already. Nice work!

Only suggestion would be to add a test similar to the one that diffs the generated code for the echo service (can be found in the makefile) and add that to the Travis config.

MrMage · 2018-08-09T15:18:56Z

Makefile

+
+project-carthage:
+	swift package generate-xcodeproj --output SwiftGRPC-Carthage.xcodeproj
+	ruby fix-indentation-settings.rb SwiftGRPC-Carthage.xcodeproj || echo "You may need to install xcodeproj ('sudo gem install xcodeproj')!"


Use @- for those as well?

+1, except for remove-unwanted-targets-for-carthage.rb that should not be ignored in case of failure, I believe.

MrMage · 2018-08-09T15:21:50Z

(All of this would still require sign-off from @timburks, of course.)

JonasVautherin · 2018-08-09T22:16:55Z

You were right, I was missing some source files, that are coming from Package.swift:21, I believe.

So I had to version swift-protobuf.git--7219529775138357838/Sources/SwiftProtobuf. Which is an issue, because I am not sure how the CI could manage that. So whenever you change the tag in Package.swift, I guess you have to remove the old swift-protobuf.git--<number> and version the new one. And I don't know what "7219529775138357838" stands for =/.

What do you think?

Only suggestion would be to add a test similar to the one that diffs the generated code for the echo service

I'm not sure what exactly you want to test. Would this test check that running make project-carthage doesn't change the files? Because it seems to me that no two runs of make project-carthage end up in the exact same SwiftGRPC-Carthage.xcodeproj.

What I can test, though, is that carthage update succeeds.

MrMage · 2018-08-10T08:03:50Z

Sorry, I had forgotten that the build is not self-contained because it also requires Swift-Protobuf to be downloaded from GitHub. Versioning SwiftProtobuf in the repo is definitely a no-go.

It appears that the SwiftProtobuf folks have managed to create an .xcodeproj file that Carthage can use (https://github.com/apple/swift-protobuf#using-carthage), but their build process doesn't depend on other libraries. Is there a way for a Carthage package to specify transitive dependencies? Otherwise I don't see a way to make Carthage work with SwiftGRPC, either.

I'm not sure what exactly you want to test. Would this test check that running make project-carthage doesn't change the files? Because it seems to me that no two runs of make project-carthage end up in the exact same SwiftGRPC-Carthage.xcodeproj.

Sorry, I had been hoping that generating the project file was deterministic.

JonasVautherin · 2018-08-10T09:04:36Z

Versioning SwiftProtobuf in the repo is definitely a no-go.

That's what I was thinking.

Is there a way for a Carthage package to specify transitive dependencies?

Yes, we can add a Cartfile for that. But here it doesn't work like this. I am very new to Apple development, and I am not familiar with the way the dependencies are linked. Let me try to describe my current understanding:

Package.swift describe the packageDependencies here.
I assume that targets then can depend on those packageDependencies, like here: SwiftGRPC depends on SwiftProtobuf that comes from the packageDependencies declaration above.
At build time, $ swift build will resolve the dependencies into .build/.

Somehow, targets depending on those dependencies know where to find their sources. For instance, I get the following kind of command when building:

<path>/swiftc <...> -module-name SwiftProtobuf <...> /path/to/swift-protobuf.git--<num>/Sources/SwiftProtobuf/AnyUnpackError.swift <other similar swift files>

Looking into SwiftGRPC-Carthage.xcodeproj/project.pbxproj, I see all those files ("AnyUnpackError.swift" and all) defined as objects there. And there is one object Package.swift defined like this (note that it is in .build/):

OBJ_1159 /* Package.swift */ = {isa = PBXFileReference; explicitFileType = sourcecode.swift; name = Package.swift; path = “/path/to/grpc-swift/.build/checkouts/swift-protobuf.git--7219529775138357838/Package.swift”; sourceTree = “<group>“; };

Still in the project.pbxproj above, there is an object SwiftProtobuf that looks like so:

OBJ_1082 /* SwiftProtobuf */ = {
                 isa = PBXGroup;
                 children = (
                         OBJ_1083 /* AnyMessageStorage.swift */,
                         OBJ_1084 /* AnyUnpackError.swift */,
                         OBJ_1085 /* BinaryDecoder.swift */,
                         OBJ_1086 /* BinaryDecodingError.swift */,
                         OBJ_1087 /* BinaryDecodingOptions.swift */,
                         OBJ_1088 /* BinaryDelimited.swift */,
                         ...
                         OBJ_1157 /* type.pb.swift */,
                         OBJ_1158 /* wrappers.pb.swift */,
                 );
                 name = SwiftProtobuf;
                 path = “.build/checkouts/swift-protobuf.git--7219529775138357838/Sources/SwiftProtobuf”;
                 sourceTree = SOURCE_ROOT;
         };

So I see that Package.swift depends somehow on SwiftProtobuf, and I see that this is translated into the xcodeproj. But that xcodeproj specifically depends on the way the SwiftPM fetches dependencies, apparently.

Again, I don't know much about build in Apple systems, but my guess would be that SwiftProtobuf is linked statically (which explains why SwiftGRPC.framework doesn't depend on SwiftProtobuf.framework) to an external dependency (i.e. those files are not versioned but come from SPM). But Carthage apparently doesn't support that, as the project needs to be self contained in the xcodeproj.

Maybe a solution would be to link SwiftProtobuf dynamically. That way the xcodeproj would be self-contained regarding the build, and SwiftGRPC.framework would link SwiftProtobuf.framework at runtime, which would be fetched by Carthage.

I am not sure if I am right, but if I am: would it be possible to link SwiftProtobuf dynamically?

JonasVautherin · 2018-08-10T14:21:43Z

Well, there may actually be another solution. Running $ swift package resolve downloads the dependencies. Putting that as a Run Script in the Build Phases of SwiftProtobuf solves the problem.

Question now: is there a way to add a Run Script programmatically from the xcodeproj tool? I haven't found documentation about that =/.

JonasVautherin · 2018-08-10T16:53:30Z

I believe I may have reached an acceptable solution!

I added a script (add-swift-resolve-prebuild-phase.rb) that alters the SwiftGRPC-Carthage.xcodeproj file to add a prebuild phase that simply runs swift package resolve. When Carthage runs xcodebuild, it effectively fetches the dependencies first into .build/, and then the build succeeds.

I am actually quite happy with this solution, because it only adds SwiftGRPC-Carthage.xcodeproj (but Carthage requires an xcodeproj to be versioned anyway) and a few lines of code in the build system.

What do you think @MrMage and @timburks?

JonasVautherin · 2018-08-10T16:54:38Z

Makefile

-		xcodebuild -configuration "Debug" -parallelizeTargets -target SwiftGRPC -target Echo -target Simple -target protoc-gen-swiftgrpc build
+		xcodebuild -project SwiftGRPC.xcodeproj -configuration "Debug" -parallelizeTargets -target SwiftGRPC -target Echo -target Simple -target protoc-gen-swiftgrpc build
+
+build-carthage:


Note that this is not necessary. But by having the CI run Carthage, that checks that the Carthage build at least succeeds.

JonasVautherin · 2018-08-12T11:55:10Z

NOTE: I need travis to make clean before it can run carthage build (otherwise there is a conflicting xcodeproj that fails to build). I realized that the clean task has issues, so I fixed it (see here).

An xcodeproj needs to be present in the repo for Carthage to work. Only 4 frameworks are enabled for Carthage builds: - BoringSSL - CgRPC - SwiftGRPC - SwiftProtobuf The other targets are removed from SwiftGRPC-Carthage.xcodeproj. Moreover, a prebuild script is added to the build phases of the cartage xcodeproj file in order to download the dependencies required by Package.swift. The Carthage-compatible xcodeproj can be generated with the following: $ make project-carthage

timburks · 2018-08-12T17:52:01Z

LGTM - thanks @JonasVautherin !

MrMage

LGTM from my side too, very creative how you found that workaround! 👍

Sorry for the late reply, I had been OOO for most of Friday and planned to review this on Monday.

JonasVautherin · 2018-08-12T20:18:51Z

Yay, thanks a lot! I wasn't expecting it to get merged so quickly! :-)

Two more things:

The carthage build now works, but people will still encounter issue missing required module 'CgRPC' #272. I fix it with a sed on my end, so we would just need to understand why CgRPC is not generated correctly by xcode (I will follow-up on missing required module 'CgRPC' #272).
For iOS build, right now the IPHONEOS_DEPLOYMENT_TARGET is not set, defaulting to 11.4. I am quite sure this is too high, and I will suggest 9.0 in another PR.

SebastianThiebaud · 2018-10-12T04:57:40Z

First, this is amazing that some work has been done to enable Carthage. This was the main blocker for us to use this repo instead of our fork.

I've been trying to use this repository with Carthage but I'm still running into problem with SwiftProtobuf. It looks like we still need to manually run make carthage-project before to be able to build Carthage dependencies. Without this manual step, the SwiftProtobuf dependency isn't downloaded in .build, therefore the project cannot be compiled.

Am I doing something wrong or that's expected? Thank you

MrMage · 2018-10-12T05:45:28Z

Hmm, my understanding is that the swift resolve step in the SwiftProtobuf target should automatically download these dependencies. However, that step might download them into a different checkout path, which would mean that the project contains the correct files, but with the wrong paths. Could this be the case here as well? (You can see examples of this in the Travis logs for my SwiftGRPC-NIO PR.)

Also, would having to run make carthage-project once before calling carthage build be a big impediment? If not, I would suggest removing the Carthage project from version control again and asking Carthage users to run make carthage-project themselves.

SebastianThiebaud · 2018-10-12T05:47:17Z

It's not a huge issue per se, but definitely not ideal.

MrMage · 2018-10-12T05:52:33Z

What are the downsides to having to run make carthage-project once yourself?

To be honest, at this point I really don't see how we can avoid that, as the paths into which SwiftPM downloads dependencies seem to be non-deterministic and may change without notice (causing the paths to mismatch inside the pre-built project).

JonasVautherin · 2018-10-12T09:45:36Z

What are the downsides to having to run make carthage-project once yourself?

Not sure I understand that. So you mean that people should git clone the repository, run make carthage-project, and point their Cartfile to that local project?

That's not how Carthage works. Either you can add grpc-swift in your Cartfile, or you can't. So as a user, in your Cartfile, you want to do something like:

github "grpc/grpc-swift" == 0.4.2

If we don't manage to do that, I believe that we cannot say that we support Carthage here. People will probably have to fork, and maintain their fork with the correct SwiftGRPC-Carthage.xcodeproj, because that's what Carthage uses. Then they may run into dependency issues:

Say project A depends on the fork grpcA
Say project B depends on the fork grpcB, and on project A.
Carthage will see grpcA and grpcB as two different dependencies, and the build will probably fail.

This said, when does SwiftPM change the paths? I believe it is non-deterministic, but it is fixed for a given commit. Right now my Cartfile works fine with:

github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417"

Which I am doing because there hasn't been a release since Carthage is supported. But ideally I would do something like:

github "grpc/grpc-swift" == 0.4.2

So maybe a tradeoff would be this: whenever you make a release, you have to run make carthage-project to update SwiftGRPC-Carthage.xcodeproj for that release tag. Following commits may not work with Carthage, but I believe we could live with having Carthage support only for the releases (and not for the intermediate commits).

Does that make sense to you? Would need to be tested, but that's my current understanding :-).

JonasVautherin · 2018-10-12T10:07:03Z

@SebastianThiebaud, would you mind trying with this in your Cartfile?

github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417"

MrMage · 2018-10-12T10:43:27Z

No, my idea was for people to add the regular SwiftGRPC repo to their Cartfile and then, when they try to build, prompt them to run `make carthage-project` once in the repo that Carthage downloaded. That shouldn't cause any dependency issues. In terms of non-deterministicity (?), it seems like every single commit in #281 fails on Travis CI in the `carthage build` step due to a path mismatch (I'm on the go right now, but you should be able to click any of the red Xes to view the CI logs yourself). So it seems like `swift resolve` *always* changes the paths in that branch, and I don't know why. It's fairly frustrating how Carthage seems so fundamentally incompatible with SwiftPM. If you have any other idea what could be going on there (or how to improve the current situation), however, I'd love to hear it :-)

…

On 12. Oct 2018, at 11:45, Jonas Vautherin ***@***.***> wrote: What are the downsides to having to run make carthage-project once yourself? Not sure I understand that. So you mean that people should git clone the repository, run make carthage-project, and point their Cartfile to that local project? That's not how Carthage works. Either you can add grpc-swift in your Cartfile, or you can't. So as a user, in your Cartfile, you want to do something like: github "grpc/grpc-swift" == 0.4.2 If we don't manage to do that, I believe that we cannot say that we support Carthage here. People will probably have to fork, and maintain their fork with the correct SwiftGRPC-Carthage.xcodeproj, because that's what Carthage uses. Then they may run into dependency issues: Say project A depends on the fork grpcA Say project B depends on the fork grpcB, and on project A. Carthage will see grpcA and grpcB as two different dependencies, and the build will probably fail. This said, when does SwiftPM change the paths? I believe it is non-deterministic, but it is fixed for a given commit. Right now my Cartfile works fine with: github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417" Which I am doing because there hasn't been a release since Carthage is supported. But ideally I would do something like: github "grpc/grpc-swift" == 0.4.2 So maybe a tradeoff would be this: whenever you make a release, you have to run make carthage-project to update SwiftGRPC-Carthage.xcodeproj for that release tag. Following commits may not work with Carthage, but I believe we could live with having Carthage support only for the releases (and not for the intermediate commits). Does that make sense to you? Would need to be tested, but that's my current understanding :-). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

JonasVautherin · 2018-10-12T14:36:20Z

No, my idea was for people to add the regular SwiftGRPC repo to their Cartfile and then, when they try to build, prompt them to run make carthage-project once in the repo that Carthage downloaded. That shouldn't cause any dependency issues.

Won't that break the carthage build? You mean you carthage bootstrap, wait for it to fail, go in the the grpc-swift directory that failed, run make carthage-project and then run the carthage build again?

On my end, I think I would rather fork grpc-swift than providing my users with a Cartfile that always fails the first time.

So it seems like swift resolve always changes the paths in that branch

Right. Whenever you run swift resolve, you need to run make carthage-project. I believe the CI should do it (because the CI tries to carthage build, which is a local test, when users run carthage bootstrap or carthage update, which is different).

But Carthage doesn't run swift resolve, right? So if you do swift resolve and make carthage-project and push that to a tag, it should work with Carthage. Juste as it work on master right now, e.g. by having the following Cartfile:

github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417"

I am trying to confirm that on my computer, but somehow I cannot run swift resolve 🤔. But if you can build with the Cartfile above, it means that there is a way to have release tags compatible with Carthage, I believe.

MrMage · 2018-10-12T14:54:15Z

Why should this be any different if pushed to a tag? AFAICT the only time when we don't need to run `resolve+make project` is *when we ship the repo with all dependencies (and the corresponding project)* included, and we don't want to do that. The current approach (added by you, IIRC) specifically calls `swift resolve` as the first step of building Swift Protobuf, so I am a bit surprised that it ever works. Also, we specifically didn't want to call`make carthage-project` on CI to ensure that the build fails if the Carthage project currently committed to the repo is out of date (so we notice that during CI runs). As far as I see it, we somehow need to have the user download this project's SPM dependencies before building (which we currently do in the generated Xcode project as explained above), but that sometimes non-deterministically messes up the resolved paths inside the project. I still don't see a way to avoid that, correct me if I'm wrong.

…

On 12. Oct 2018, at 16:36, Jonas Vautherin ***@***.***> wrote: No, my idea was for people to add the regular SwiftGRPC repo to their Cartfile and then, when they try to build, prompt them to run make carthage-project once in the repo that Carthage downloaded. That shouldn't cause any dependency issues. Won't that break the carthage build? You mean you carthage bootstrap, wait for it to fail, go in the the grpc-swift directory that failed, run make carthage-project and then run the carthage build again? On my end, I think I would rather fork grpc-swift than providing my users with a Cartfile that always fails the first time. So it seems like swift resolve always changes the paths in that branch Right. Whenever you run swift resolve, you need to run make carthage-project. I believe the CI should do it (because the CI tries to carthage build, which is a local test, when users run carthage bootstrap or carthage update, which is different). But Carthage doesn't run swift resolve, right? So if you do swift resolve and make carthage-project and push that to a tag, it should work with Carthage. Juste as it works on master right now: github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417" I am trying to confirm that on my computer, but somehow I cannot run swift resolve 🤔. But if you can build with the Cartfile above, it means that there is a way to have release tags compatible with Carthage, I believe. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

JonasVautherin · 2018-10-12T16:30:42Z

I guess what I'm saying is that for me, using github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417" in a Cartfile always works. So I'm trying to understand why it fails in the CI and in @SebastianThiebaud's build.

I'll spend some time on it over the week-end. I spent quite some time trying to get this to work, so I'm willing to push a bit further now instead of dropping support for Carthage :-).

MrMage · 2018-10-12T17:01:46Z

Yeah, I haven't figured out why it sometimes fails and sometimes works, either. But it looks like my current open PR *always* fails *on CI*, so you could experiment with that one. (For faster iteration times, you can temporarily skip other CI steps.)

…

On 12. Oct 2018, at 18:30, Jonas Vautherin ***@***.***> wrote: I guess what I'm saying is that for me, using github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417" in a Cartfile always works. So I'm trying to understand why it fails in the CI and in @SebastianThiebaud's build. I'll spend some time on it over the week-end. I spent quite some time trying to get this to work, so I'm willing to push a bit further now instead of dropping support for Carthage :-). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

JonasVautherin · 2018-10-15T22:53:35Z

So, I took your NIO branch and ran make project-carthage, and Travis passed.

So I believe this is the situation:

Whenever there is a need to run swift resolve (I don't know when that is), then the Carthage project gets outdated.
It is most likely not practical to always update the Carthage project (because it adds noise in the commits).
A release of grpc-swift is made once in a while (see releases here).
The way Carthage works is that you set the dependency in the Cartfile, usually to a release tag (e.g. github "grpc/grpc-swift" == 0.4.2).

It seems like we won't be able to always have the Carthage project up-to-date (because it would imply constantly running and commiting make project-carthage), a tradeoff would be to say that whenever you guys make a release, you run and commit make project-carthage. So that people can use Carthage normally (i.e. using github "grpc/grpc-swift" == 0.4.2, github "grpc/grpc-swift" == 0.4.3, github "grpc/grpc-swift" == 0.5.3 etc). However, using a custom commit with Carthage will not be supported (e.g. something like github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417"), meaning that it may or may not work.

But I believe it is fine, because it is not really how we would expect people to depend on Carthage anyway, and at least depending on the releases should work flawlessly.

Now back to the CI: it should run make project-carthage when it runs swift resolve for feature branches, and ideally it should not run make project-carthage when building a release (to make sure that the Carthage xcodeproj is up-to-date). One way to do that (that I can think of) would be to have a develop branch for development, and a master branch for releases. When opening a PR to master, the CI does not run make project-carthage.

@MrMage does that make sense to you? So it seems like swift resolve requires make project-carthage, and both are undeterministic. But it doesn't mean we need to drop support for Carthage, as we can decide to support Carthage for release tags. Not ideal, but most users will probably never see that.

MrMage · 2018-10-16T16:32:34Z

@JonasVautherin thank you for the elaboration! This definitely makes sense for me. Personally, I would simply always run make project-carthage on CI. Actually, maybe we should skip testing Carthage builds altogether, given that they add significantly to our CI runtime while not being relevant except for release builds anyway. (The expectation should be that a fresh Carthage project built with make project-carthage should build only if our regular build tests pass as well.)

Instead, before a release is cut, it would be the responsibility of whoever is cutting the release to run make project-carthage, run the Carthage tests with that project, commit that project, and only then tag the release. Would that make sense?

JonasVautherin · 2018-10-16T21:53:09Z

Instead, before a release is cut, it would be the responsibility of whoever is cutting the release to run make project-carthage, run the Carthage tests with that project, commit that project, and only then tag the release. Would that make sense?

Makes perfect sense to me! :-)

Note (for the record) that until the next release of grpc-swift, I'll keep using this commit that I know works in my Cartfile:

github "grpc/grpc-swift" "23a0ebdee9613f615f2f2469ed3e700df5856417"

MrMage reviewed Aug 9, 2018

View reviewed changes

JonasVautherin force-pushed the enable-carthage branch from d32cbc4 to d53636e Compare August 9, 2018 20:34

JonasVautherin force-pushed the enable-carthage branch from 3fc11f9 to d7f37cc Compare August 9, 2018 22:57

JonasVautherin commented Aug 10, 2018

View reviewed changes

JonasVautherin force-pushed the enable-carthage branch 4 times, most recently from cc908b2 to b21d895 Compare August 12, 2018 11:51

fix make clean target

d8b2b51

JonasVautherin force-pushed the enable-carthage branch from b21d895 to 2c5a80e Compare August 12, 2018 11:53

JonasVautherin changed the title ~~[WIP] Enable carthage~~ Enable carthage Aug 12, 2018

JonasVautherin added 2 commits August 12, 2018 18:54

version SwiftGRPC-Carthage.xcodeproj for Carthage

e07895f

JonasVautherin force-pushed the enable-carthage branch from 2c5a80e to e07895f Compare August 12, 2018 16:54

timburks merged commit 051651e into grpc:master Aug 12, 2018

MrMage reviewed Aug 12, 2018

View reviewed changes

This was referenced Aug 12, 2018

Carthage: support from iOS 9.0 #284

Merged

missing required module 'CgRPC' #272

Closed

MrMage mentioned this pull request Oct 16, 2018

Minor tweaks in preparation for SwiftGRPCNIO. #318

Merged

MrMage mentioned this pull request Nov 8, 2018

Carthage not working for 0.5.1 & 0.6.0 #329

Closed

Enable carthage #282

Enable carthage #282

Uh oh!

Conversation

JonasVautherin commented Aug 9, 2018

Uh oh!

JonasVautherin commented Aug 9, 2018

Uh oh!

MrMage commented Aug 9, 2018

Uh oh!

MrMage left a comment

Choose a reason for hiding this comment

Uh oh!

MrMage Aug 9, 2018

Choose a reason for hiding this comment

Uh oh!

JonasVautherin Aug 9, 2018

Choose a reason for hiding this comment

Uh oh!

MrMage commented Aug 9, 2018

Uh oh!

JonasVautherin commented Aug 9, 2018

Uh oh!

MrMage commented Aug 10, 2018

Uh oh!

JonasVautherin commented Aug 10, 2018

Uh oh!

JonasVautherin commented Aug 10, 2018

Uh oh!

JonasVautherin commented Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonasVautherin Aug 10, 2018

Choose a reason for hiding this comment

Uh oh!

JonasVautherin commented Aug 12, 2018

Uh oh!

timburks commented Aug 12, 2018

Uh oh!

MrMage left a comment

Choose a reason for hiding this comment

Uh oh!

JonasVautherin commented Aug 12, 2018

Uh oh!

SebastianThiebaud commented Oct 12, 2018

Uh oh!

MrMage commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SebastianThiebaud commented Oct 12, 2018

Uh oh!

MrMage commented Oct 12, 2018

Uh oh!

JonasVautherin commented Oct 12, 2018

Uh oh!

JonasVautherin commented Oct 12, 2018

Uh oh!

MrMage commented Oct 12, 2018 via email

Uh oh!

JonasVautherin commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrMage commented Oct 12, 2018 via email

Uh oh!

JonasVautherin commented Oct 12, 2018

Uh oh!

MrMage commented Oct 12, 2018 via email

Uh oh!

JonasVautherin commented Oct 15, 2018

Uh oh!

MrMage commented Oct 16, 2018

Uh oh!

JonasVautherin commented Oct 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

JonasVautherin commented Aug 10, 2018 •

edited

Loading

MrMage commented Oct 12, 2018 •

edited

Loading

JonasVautherin commented Oct 12, 2018 •

edited

Loading

JonasVautherin commented Oct 16, 2018 •

edited

Loading