Add large runtime resource directory to lfs, and expose it to the Docker build. #4530

cmnbroad · 2018-03-14T19:13:18Z

This adds a new folder for large runtime resources to git-lfs. Unlike the large test resources, which are only accessed via a volume mount when tests are run on the Docker image, the runtime resources need to be accessible to the GATK build during the Docker build process, since they're included in the jar. AFAICT there is no docker build equivalent to docker run -v. So for now the runtime resources are git pulled into the Docker staging area, and thus onto the Docker image. We need this for @lucidtronix 's CNN branch (and possibly for @TedBrookings) if we're going to load models for resources, but longer term, we need a better solution.

cmnbroad · 2018-03-14T19:16:15Z

@jamesemery @lbergelson Any alternate suggestions ?

codecov-io · 2018-03-14T20:09:10Z

Codecov Report

Merging #4530 into master will decrease coverage by 0.224%.
The diff coverage is 81.818%.

@@              Coverage Diff               @@
##             master     #4530       +/-   ##
==============================================
- Coverage     79.84%   79.616%   -0.224%     
- Complexity    16958     17758      +800     
==============================================
  Files          1062      1068        +6     
  Lines         61677     65188     +3511     
  Branches       9983     10738      +755     
==============================================
+ Hits          49243     51900     +2657     
- Misses         8539      9232      +693     
- Partials       3895      4056      +161

Impacted Files	Coverage Δ	Complexity Δ
...g/broadinstitute/hellbender/utils/io/Resource.java	`52.381% <ø> (+9.524%)`	`5 <0> (+1)`	⬆️
...g/broadinstitute/hellbender/utils/NativeUtils.java	`25% <0%> (ø)`	`3 <0> (ø)`	⬇️
...adinstitute/hellbender/utils/R/RScriptLibrary.java	`100% <100%> (ø)`	`6 <1> (ø)`	⬇️
...rg/broadinstitute/hellbender/utils/io/IOUtils.java	`68.358% <88.889%> (+6.549%)`	`100 <3> (+49)`	⬆️
...itute/hellbender/tools/funcotator/Funcotation.java	`45.455% <0%> (-4.545%)`	`7% <0%> (+3%)`
.../discovery/alignment/ContigAlignmentsModifier.java	`81.25% <0%> (-4.285%)`	`70% <0%> (+29%)`
...ignment/AssemblyContigWithFineTunedAlignments.java	`66.234% <0%> (-3.669%)`	`41% <0%> (+9%)`
.../tools/copynumber/PostprocessGermlineCNVCalls.java	`87.931% <0%> (-3.14%)`	`30% <0%> (+7%)`
...ools/funcotator/FuncotatorArgumentDefinitions.java	`85.075% <0%> (-1.592%)`	`1% <0%> (ø)`
...ools/copynumber/DetermineGermlineContigPloidy.java	`95.041% <0%> (-1.429%)`	`19% <0%> (+5%)`
... and 49 more

lbergelson · 2018-03-14T21:21:50Z

src/test/java/org/broadinstitute/hellbender/utils/io/IOUtilsUnitTest.java

@@ -270,5 +267,21 @@ public void testUrlEncodeDecode(final String string, final String encoded) {
        Assert.assertEquals(string, IOUtils.urlDecode(encoded));
    }

+    @Test
+    public void testLoadLargeRuntimeResource() throws IOException {
+        final String largeResourcePath = "large/largeResourceTest.txt";


this path should be a constant somewhere (the "large" I mean)

lbergelson

This seems sane to me. I think we're going to need to update the readme to explain that some large files are required to build a functional gatk now though. Maybe we can wait until a tool that requires them though.

We might want to add a check and warning to the gradle build that the large runtime resources are available as well so that it doesn't blindly build without them and then people get surprised. (also the tools that require them need to have good error checking around their existence...)

cmnbroad · 2018-03-19T14:47:41Z

I added a check to build.gradle to verify that the large (runtime) resources are not lfs pointer files. I also added a test to verify that there is a .git folder, and non-null ToolProvider classLoader (fixes #4532). To test the pointer file check, I created a branch of this branch with a temporary commit that reverts the addition of the large resources to the Docker staging area, and submitted it to Travis to verify that the build fails when they're not present.

droazen · 2018-03-19T19:48:15Z

README.md

@@ -55,6 +55,8 @@ releases of the toolkit.
 * Optional, but recommended:
    * Gradle 3.1 or greater, needed for building the GATK. We recommend using the `./gradlew` script which will
      download and use an appropriate gradle version automatically (see examples below).
+    * [git-lfs](https://git-lfs.github.com/) 1.1.0 or greater, needed for building the GATK. Used to download the large files required to run the build and to run the test suite.
+      Run `git lfs install` after downloading, followed by `git lfs pull` from the root of your git clone to download the large files. The download is several hundred megabytes.


Maybe we should split into "build" and "runtime" requirements?

Also, can you check whether git lfs pull still works?

I split this into separate sections for runtime and build. Updated to state that the lfs download is around 2GB. And git lfs pull does work.

droazen · 2018-03-19T19:49:12Z

README.md

@@ -55,6 +55,8 @@ releases of the toolkit.
 * Optional, but recommended:
    * Gradle 3.1 or greater, needed for building the GATK. We recommend using the `./gradlew` script which will
      download and use an appropriate gradle version automatically (see examples below).
+    * [git-lfs](https://git-lfs.github.com/) 1.1.0 or greater, needed for building the GATK. Used to download the large files required to run the build and to run the test suite.


"run the build" -> "build GATK"

droazen · 2018-03-19T19:50:18Z

README.md

@@ -102,7 +102,7 @@ You can download and run pre-built versions of GATK4 from the following places:

    * This creates a zip archive in the `build/` directory with a name like `gatk-VERSION.zip` containing a complete standalone GATK distribution, including our launcher `gatk`, both the local and spark jars, and this README.    
    * You can also run GATK commands directly from the root of your git clone after running this command.
-    * Note that you *must* have a full git clone in order to build GATK. The zipped source code alone is not buildable.
+    * Note that you *must* have a full git clone in order to build GATK, incuding git-lfs files. The zipped source code alone is not buildable.


incuding -> including

"including the git-lfs files in src/main/resources" ?

droazen · 2018-03-19T19:56:43Z

build.gradle

+task validateBuildPrerequisites {
+    if (ToolProvider.getSystemToolClassLoader() == null) {
+        throw new RuntimeException(
+                String.format(buildPrerequisitesErrorFormatString, "A (version 8) JDK must be installed."));


Extract required Java version as a named constant at the top

droazen · 2018-03-19T19:57:25Z

build.gradle

+    if (!file(".git").isDirectory()) {
+        throw new RuntimeException(
+                String.format(buildPrerequisitesErrorFormatString,
+                        "The GATK Github repository must be cloned to run the build."));


Add "using git clone"

Also provide a link to github tutorial

These are all switched to use string interpolation now. The embedded buildPrerequisites message directs the user to the gatk github repo.

droazen · 2018-03-19T19:59:13Z

build.gradle

+                        "The GATK Github repository must be cloned to run the build."));
+    }
+    // Next, ensure that the git-lfs files required to run the build are present
+    final FileTree largeResources = fileTree(dir: "src/main/resources/large")


Declare src/main/resources/large as a named constant at the top

droazen · 2018-03-19T19:59:43Z

build.gradle

+def validateLargeResourceFile(final File largeResourceFile, buildPrerequisitesErrorFormatString) {
+    largeResourceFile.withReader("UTF-8") { reader ->
+        final String firstLine = reader.readLine();
+        if (firstLine != null && firstLine.startsWith("version https://git-lfs.github.com/spec/v1")) {


Declare this string as a named constant somewhere

droazen · 2018-03-19T20:00:39Z

build.gradle

+// that the lfs files have been fetched and that the directory does not contain the git-lfs pointer files
+def validateLargeResourceFile(final File largeResourceFile, buildPrerequisitesErrorFormatString) {
+    largeResourceFile.withReader("UTF-8") { reader ->
+        final String firstLine = reader.readLine();


Read first X bytes instead of a full line in case the first line is very large.

droazen · 2018-03-19T20:02:12Z

build_docker.sh

@@ -98,6 +98,11 @@ if [ -n "$STAGING_DIR" ]; then
    GIT_CHECKOUT_COMMAND="git checkout ${GITHUB_DIR}${GITHUB_TAG}"
    echo "${GIT_CHECKOUT_COMMAND}"
    ${GIT_CHECKOUT_COMMAND}
+    # pull just the large runtime resources into the staging area so they are available to the build
+    # when its done as part of the Docker build so the resources can be compiled into the jar as resources


its -> it's

droazen · 2018-03-19T20:03:30Z

build_docker.sh

@@ -98,6 +98,11 @@ if [ -n "$STAGING_DIR" ]; then
    GIT_CHECKOUT_COMMAND="git checkout ${GITHUB_DIR}${GITHUB_TAG}"
    echo "${GIT_CHECKOUT_COMMAND}"
    ${GIT_CHECKOUT_COMMAND}
+    # pull just the large runtime resources into the staging area so they are available to the build
+    # when its done as part of the Docker build so the resources can be compiled into the jar as resources
+    GIT_PULL_LARGE_COMMAND="git lfs pull --include src/main/resources/large/"


Might be good to mention this command in the README for users who want to build GATK without downloading all of the large test data.

droazen · 2018-03-19T20:05:46Z

src/main/java/org/broadinstitute/hellbender/utils/io/Resource.java

@@ -9,6 +9,7 @@
 public final class Resource {
    private final String path;
    private final Class<?> relativeClass;
+    public final static String LARGE_RESOURCES_PATH = "large";


Add a comment here explaining what this is

droazen · 2018-03-19T20:07:10Z

src/test/java/org/broadinstitute/hellbender/utils/io/IOUtilsUnitTest.java

+    @Test
+    public void testLoadLargeRuntimeResource() throws IOException {
+        final Resource largeResource = new Resource(Resource.LARGE_RESOURCES_PATH + "/largeResourceTest.txt", null);
+        final File resourceFile = IOUtils.writeTempResource(largeResource);


Might want to extract a simple utility method for loading a resource and writing to a temp location (would return a File)

cmnbroad · 2018-03-20T18:53:22Z

@droazen please take a look at the updated README.md in this PR.

droazen

Second-pass review complete, back to @cmnbroad for more changes. A bunch of little comments, but also a request to have the build script run git lfs pull --include src/main/resources/large/ as needed during a build.

droazen · 2018-03-21T17:48:47Z

build.gradle

 final pythonPackageArchiveName = 'gatkPythonPackageArchive.zip'
+final largeResourcesFolder = "src/main/resources/large"
+final lfsFileSentinel = "version https://git-lfs.github.com/spec/v1"


Add a comment documenting what lfsFileSentinel is, since it's not immediately obvious.

Also, recommend renaming to lfsStubFileHeader for clarity

I changed all of these names to use "stub", but I'm not sure thats better, since AFACIT git-lfs refers to these as pointer files in all of the help and doc.

droazen · 2018-03-21T17:53:38Z

build.gradle

+    final byte[] actualBytes = readBytesFromFile(largeResourceFile, lfsSentinel.length());
+    if (new String(actualBytes, "UTF-8") == lfsSentinel) {
+        throw new GradleException(
+                "$largeResourceFile appears to be a git-lfs pointer file." +


pointer file -> stub file

droazen · 2018-03-21T17:53:59Z

build.gradle

+    if (new String(actualBytes, "UTF-8") == lfsSentinel) {
+        throw new GradleException(
+                "$largeResourceFile appears to be a git-lfs pointer file." +
+                        " The resource files in src/main/resources/large must be downloaded using git-lfs. " +


Use the largeResourcesFolder constant here

droazen · 2018-03-21T17:54:17Z

build.gradle

+final buildPrerequisitesMessage = "See https://github.com/broadinstitute/gatk#building for information on how to build GATK."
+
+// Large runtime resource files must be present at build time to be compiled into the jar, so validate
+// that the lfs files have been fetched and that the directory does not contain the git-lfs pointer files


pointer files -> stub files

droazen · 2018-03-21T17:56:00Z

build.gradle

+// Validate that we have a clone of the repository and that any required git-lfs files have been resolved
+def validateBuildPrerequisites(javaVersion, largeResourcesFolder, lfsSentinel, buildPrerequisitesMessage) {
+    if (ToolProvider.getSystemToolClassLoader() == null) {
+        throw new GradleException("The SystemToolClassLoader obtained from the ToolProvider is null. A Java $RequiredJavaVersion JDK must be installed. $buildPrerequisitesMessage")


When can this happen?

I think it happens when you have a JRE rather than a JDK (see #4532). Also, although I haven't tried it I suspect it will happen with JDK9.

droazen · 2018-03-21T18:26:05Z

src/test/java/org/broadinstitute/hellbender/utils/io/IOUtilsUnitTest.java

+        finally {
+            resourceFile.delete();
+        }
+        Assert.assertEquals(resourceContents, "line in large resource file");


Declare expected text as a constant, and use it in both test methods.

droazen · 2018-03-21T18:26:32Z

src/test/java/org/broadinstitute/hellbender/utils/io/IOUtilsUnitTest.java

+    @DataProvider(name="resourcePaths")
+    public Object[][] getResourcePaths() {
+        return new Object[][] {
+                { Resource.LARGE_RUNTIME_RESOURCES_PATH + "/largeResourceTest.txt", null }


Can you add a test case involving a non-null relative class as well?

droazen · 2018-03-21T18:30:48Z

README.md

+    * [git-lfs](https://git-lfs.github.com/) 1.1.0 or greater. Required to download the large files used to build GATK and
+      run the test suite. Run `git lfs install` after downloading, followed by `git lfs pull` from the root of your git clone
+      to download all of the large files. The full download is approximately 2 gigabytes. Alternatively, to pull the minimal
+      set of large `lfs` resource files required to only build ()without running tests), the test resources can be excluded


()without running tests) -> (without running tests)

droazen · 2018-03-21T18:32:29Z

README.md

+      run the test suite. Run `git lfs install` after downloading, followed by `git lfs pull` from the root of your git clone
+      to download all of the large files. The full download is approximately 2 gigabytes. Alternatively, to pull the minimal
+      set of large `lfs` resource files required to only build ()without running tests), the test resources can be excluded
+      from the download by running `git lfs pull --include src/main/resources/large/` instead of `git lfs pull`. This greatly


Can we modify build.gradle to run this command automatically at build time, if lfs stubs are detected? Let's do that (if possible) -- then this set of instructions can be simplified to just "install git lfs"

I changed the build to check for stubs, and to only run git lfs pull --include src/main/resources/large/ if it finds any, then validate again. Also, we still have to manually download the files into the staging area, since the Docker doesn't have git-lfs installed. However, that doesn't pull the test files. So the readme still needs to reflect that difference. Updated accordingly. See what you think.

Done see above note.

droazen · 2018-03-21T18:33:51Z

README.md

-    * [git-lfs](https://git-lfs.github.com/) 1.1.0 or greater (needed to download large files for the complete test suite).
-      Run `git lfs install` after downloading, followed by `git lfs pull` from the root of your git clone to download the large files. The download is several hundred megabytes.
-* Alternatively, pre-packaged images with all needed dependencies installed can be found on [our dockerhub repository](https://hub.docker.com/r/broadinstitute/gatk/). This requires a recent version of the docker client, which can be found on the [docker website](https://www.docker.com/get-docker).
+* To build GATK:


Is R a build-time dependency as well?

Only to run tests, AFAIK. I added it to this section as well saying its needed for running tests.

cmnbroad · 2018-03-23T13:50:17Z

Back to @droazen. Responded to all of the comments. I did an interim commit to test the lfs changes on docker/travis so there are two commits for this round.

droazen

👍 Final review complete, back to @cmnbroad. Merge after addressing remaining comments.

droazen · 2018-03-23T19:30:47Z

build.gradle

+        println "Fetching large runtime resources via: $execLFSPull"
+        def retCode = execLFSPull.execute().waitFor()
+        if (retCode.intValue() != 0) {
+            throw new GradleException("git-lfs pull execution failed: $retCode");


"failed" -> "failed with exit code: "

droazen · 2018-03-23T19:41:09Z

build.gradle

+        }
+        return retCode
+    } catch (IOException e) {
+        throw new GradleException("git-lfs must be installed to run the GATK build", e)


Is git-lfs not being installed the only possible cause of an IOException here?

I can't be sure. I changed the message to report the IOException and suggest checking git-lfs.

droazen · 2018-03-23T19:42:55Z

build.gradle

+        if (checkLargeResourcesForStubFiles(largeResourcesFolder, lfsStubFileHeader, buildPrerequisitesMessage)) {
+            throw new GradleException(
+                    "$largeResourcesFolder contains git-lfs stub files." +
+                            " The resource files in $largeResourcesFolder must be downloaded using git-lfs. " +


Provide the actual git-lfs command in this error message.

droazen · 2018-03-23T19:46:32Z

build.gradle

+}
+
+// Validate that we have a clone of the repository and that any required git-lfs files have been resolved
+def validateBuildPrerequisites(javaVersion, largeResourcesFolder, lfsStubFileHeader, buildPrerequisitesMessage) {


This now does more than validation (it also potentially actually downloads prerequisites). Update the comment (and possible the name as well) to reflect this.

droazen · 2018-03-23T19:49:48Z

build.gradle

@@ -262,6 +321,8 @@ processResources {
 processTestResources {
    //Don't waste time packaging unnecessary test data into the test resources:
    include "org/broadinstitute/hellbender/utils/config/*"
+    //Required for IOUtils resource tests
+    include "org/broadinstitute/hellbender/utils/io/*"


Will this block need to be modified every time a resource is added?

Only if you add a resource that needs to be packaged into the test jar as part of the test resources.

droazen · 2018-03-23T20:01:33Z

src/test/java/org/broadinstitute/hellbender/utils/io/IOUtilsUnitTest.java

+    {
+        final Resource largeResource = new Resource(resourcePath, relativeClass);
+        final File resourceFile = IOUtils.writeTempResource(largeResource);
+        final String resourceContentsFirstLine = getFirstLineAndDeleteTempFile(resourceFile);


Shouldn't the IOUtils.writeTempResource() methods always set the extracted resource to be deleted on JVM exit? It seems like we will end up leaving garbage around if we don't do that in the methods themselves.

We probably should. I'm going to do that in a separate PR though since all of the call sites should be cleaned up as well.

droazen · 2018-03-23T20:11:49Z

README.md

+    * [git-lfs](https://git-lfs.github.com/) 1.1.0 or greater. Required to download the large files used to build GATK, and
+      test files required to run the test suite. Run `git lfs install` after downloading, followed by `git lfs pull` from
+      the root of your git clone to download all of the large files, including those required to run the test suite. The
+      full download is approximately 2 gigabytes. Alternatively, you can skip this step since the build will itself use


"Alternatively, " -> "Alternatively, if you are just building GATK and not running the test suite, "

droazen

👍 One or two last comments, then this is good to merge.

Can you create a ticket for the task of moving to delete-on-exit for extracted resources, so we don't forget to do it?

droazen · 2018-03-26T19:18:20Z

README.md

+      the root of your git clone to download all of the large files, including those required to run the test suite. The
+      full download is approximately 2 gigabytes. Alternatively, if you are just building GATK and not running the test
+      suite, you can skip this step since the build itself will use git-lfs to download the minimal set of large `lfs`
+      resource files required to complete the build. The tests resources will not be downloaded, but this greatly reduces


"tests resources" -> "test resources"

droazen · 2018-03-26T19:20:14Z

build.gradle

+    final lfsStubFileHeader = "version https://git-lfs.github.com/spec/v1"  // first line of a git-lfs stub file
+    def readBytesFromFile = { largeFile, n ->
+        final byte[] bytes = new byte[n]
+        largeFile.withInputStream { stream -> stream.read(bytes, 0, bytes.length) }


Does the input stream here get closed?

Yes, its equivalent to try-wth-resources.

…ker build.

…ker build. (broadinstitute#4530)

lbergelson reviewed Mar 14, 2018

View reviewed changes

cmnbroad force-pushed the cn_large_runtime_resources branch 2 times, most recently from 109c9f1 to 2a09e01 Compare March 16, 2018 20:09

droazen reviewed Mar 19, 2018

View reviewed changes

cmnbroad force-pushed the cn_large_runtime_resources branch from d00d04f to 485ef3c Compare March 20, 2018 18:46

droazen self-assigned this Mar 21, 2018

droazen suggested changes Mar 21, 2018

View reviewed changes

droazen assigned cmnbroad and unassigned droazen Mar 21, 2018

cmnbroad force-pushed the cn_large_runtime_resources branch 2 times, most recently from 700667a to f5fed00 Compare March 23, 2018 00:22

magicDGS mentioned this pull request Mar 23, 2018

Generate different artifacts/modules for different project components #3900

Open

cmnbroad assigned droazen and unassigned cmnbroad Mar 23, 2018

droazen suggested changes Mar 23, 2018

View reviewed changes

droazen assigned cmnbroad and unassigned droazen Mar 23, 2018

cmnbroad force-pushed the cn_large_runtime_resources branch from 72e5adb to 1fb2207 Compare March 26, 2018 13:27

droazen approved these changes Mar 26, 2018

View reviewed changes

Add large runtime resource directory to lfs, and expose it to the Doc…

9c646ca

…ker build.

cmnbroad force-pushed the cn_large_runtime_resources branch from 1fb2207 to 9c646ca Compare March 26, 2018 19:56

cmnbroad merged commit b572047 into master Mar 27, 2018

cmnbroad mentioned this pull request Mar 27, 2018

improve error message during build when ToolProvider is unavailable #4532

Closed

cmnbroad deleted the cn_large_runtime_resources branch April 26, 2018 13:19

cwhelan pushed a commit to cwhelan/gatk-linked-reads that referenced this pull request May 25, 2018

Add large runtime resource directory to lfs, and expose it to the Doc…

42bdced

…ker build. (broadinstitute#4530)

Add large runtime resource directory to lfs, and expose it to the Docker build. #4530

Add large runtime resource directory to lfs, and expose it to the Docker build. #4530

Conversation

cmnbroad commented Mar 14, 2018 • edited Loading

cmnbroad commented Mar 14, 2018

codecov-io commented Mar 14, 2018 • edited Loading

Codecov Report

lbergelson Mar 14, 2018 • edited Loading

Choose a reason for hiding this comment

lbergelson left a comment

Choose a reason for hiding this comment

cmnbroad commented Mar 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmnbroad commented Mar 20, 2018

droazen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmnbroad Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmnbroad commented Mar 23, 2018

droazen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen left a comment

cmnbroad commented Mar 14, 2018 •

edited

Loading

codecov-io commented Mar 14, 2018 •

edited

Loading

lbergelson Mar 14, 2018 •

edited

Loading

cmnbroad Mar 23, 2018 •

edited

Loading

droazen Mar 23, 2018 •

edited

Loading