Federated Amazon S3 support for IGV desktop, migration towards Java11, Gradle and better release engineering

[brainstorm]

This pullrequest is a followup on one of my tweets and should fix issue #320. I am not a Java expert, so forgive my not-so-idiomatic code :-S

• Adds AWS Cognito OAuth2/OIDC federated identity (with Google in our case, but any social login is supported).
• Adds dynamic JTree S3 object loader, so that object listings are populated as needed, not all at once in a LoadDialog nor hardcoded statically as in the GA4GH prototype LoadDialog.
• Corrects some OAuth2 implementation defects, i.e: state parameter should be a random string to avoid CSRF.
• Gradle pulls AWS dependencies instead of keeping them as binaries under /lib.
• Avoids persisting OAuth refresh (long-lived) token on disk (security risk), keep it on memory as the app is running instead. In the case of AWS Cognito the default is 30 days, if an attacker gets access to it, that's how long they would have to exfiltrate data.

TODO:

• A bit more cleanup and refactoring (ActionListeners for UI are too complex/specific, factor out).
• Ask @igvteam about their Gradle dependency management policy, integrate dependency pulling in .travis.yml. In other words: does it make sense to store the dep .jars under /lib instead of just declaring them on build.gradle?
• Discuss with @igvteam and refactor oauth-config.json vs .properties logic in OAuthUtils.java fetchOauthProperties().
• Make sure the pre-existing Google flow still works.
• Write up a blog post about the whole authentication/cognito setup and integration with IGV.: https://umccr.org/blog/2019/02/16/igv-amazon/
• Maybe ask @igvteam to fix up the dialog/forms that need JFormDesigner? (we don't have a license) :/
• Fully migrate to Java11, deprecate Java8
• Migrate lib/*.jar to Gradle implementation definitions.
• Tighten up a bit the (transitive) dependencies
• Cleanup as stated in Federated Amazon S3 support for IGV desktop, migration towards Java11, Gradle and better release engineering #620 (comment)
• @igvteam pushing to this branch to retain credit/authorship as mentioned below, also include @reisingerf as co-author. Credit for the documentation goes to @andrewpatto.
• Merge into master! 🎉

[brainstorm]

@igvteam @jrobinso, first, thanks a ton for considering this PR!

Practical question to move towards master merging goal faster as itemized in the PR description: Would it make sense to remove some of the .jars you keep under lib/ and put the equivalent Maven references on build.gradle? It should easily remove binary fat from the repo, what are your thoughts and internal discussions on that point?

[jrobinso]

@brainstorm We don't have time to take that on right now. Perhaps in a future refactoring when I can devote some time. The amount of disk space taken is infinitesimal by modern standards, no motivation to devote time to this.

When you think you are done developing (I see regular pushes) and this is ready for review let me know. It looks exciting, I'd like to get it in. Try to make it as minimally disruptive as possible, perhaps opening other git issues for non-essential improvements. I'm juggling many projects now so review time is precious, and anything that requires a lot of re-testing of unrelated areas will make it difficult. Thanks!

[brainstorm]

@jrobinso Gotcha, I'll leave the Gradle stuff as-is then, without removing the jars, good points.

I think that in the interest of getting this merged fast, I'll fix the oauth-config.json vs .properties logic. That bit and the automatic .vcf/.bam indexes detection are the two main issues preventing merge to master right now, imho.

I will try to expedit those changes during this week and call it done over here once I've tested that I didn't break anything else.

Thanks for your patience and interest!

[jrobinso]

OK, sounds good. We can tackle the jars and other issues in future work.

[jrobinso]

@brainstorm It looks like there is a compile error (travis build). I don't have time currently to dig into it, could you check the travis output?

[brainstorm]

@jrobinso No worries, working on it right now! As I mentioned I will @-mention you when it's ready-ready for good, tackling the fetchOauthProperties() now ;)

[jrobinso]

Ahh, ok, I saw a comment and interpreted it as "ready". No rush, I couldn't look at it right away even if it was ready.

[jrobinso]

@brainstorm Thanks for the update. I'm progressing but underestimated the impact. Ill plan to do the merge Monday. Then we'll wait a few days for any issue reports, I don't expect any, and plan a release soon. One item for that will documentation, we will probably rely on your web pages for that. Also, please coordinate any tweets or announcements with myself and @helgathorv. Thats all for now.

[brainstorm]

Thanks for the heads up @jrobinso, eager to see this merged on master finally.

@helgathorv Please do let me know if IGV was not properly mentioned for your (community manager? lawyer?) internal criteria. Also I'll send you drafts of future blogposts if there's any (none planned so far)... any preferred channel for that? Email?

[helgathorv]

Thanks @brainstorm. For direct email: you can use igv-team@broadinstitute.org. That reaches both me and Jim. Just ignore the auto-reply that says please post to the forum.

The main issue I've noticed is that IGV has been misnamed both in tweets (Interactive Genome Viewer & Integrated Genomics Viewer) and in the description of the umccr/igv repo (Interactive Genome Viewer).

Also, while the IGV was initially developed only at the Broad, the PI of our lab and many of the lab members, including Jim Robinson, moved to UCSD a few years ago. The folks who moved still also have appointments at the Broad, and IGV is a joint project between Broad and UCSD. I totally understand that most people aren't aware of this - the Broad is still in many of our URLs etc, but we try not to say things like "The Broad Institute's IGV".

[jrobinso]

@brainstorm merged! whew. One thing I didn't review was the readme, oversight on my part. Could you update it with current build instructions and send another PR? What's there is clearly wrong.

[ohofmann]

@jrobinso @brainstorm @davideby Wow. Thanks so much for all your work on this. Been following from the sidelines; really happy to see this go in.

[brainstorm]

Please don't forget @reisingerf, without him this would have been impossible... also @andrewpatto for the docs ;)

Good catch @jrobinso, totally forgot about the README.md, I will field that soon in a separate PR. And also one more PR for a very minor backend cognito-related improvement that we just found out recently (does not affect existing IGV functionality at all, just a backend (CloudTrail) detail).

@helgathorv Thanks for the explanation, I had no idea of all this :-S

[ohofmann]

Ouch, fair - thanks to you, too ;)

[welcomege]

@brainstorm Thank the team for developing this IGV add-on. We followed the umccr blog page in setting up AWS backend. It is working fine in loading s3 files using Add File | Add file from URL. When I was trying to add a file from the dialog Amazon | Load file from S3 bucket, I got the following error:

INFO [2020-10-16T14:49:03,916] [MessageUtils.java:76] Unexpected error: null.
See igv.log for more details
ERROR [2020-10-16T14:49:03,917] [LongRunningTask.java:75] Exception running task
java.lang.NullPointerException: null
at org.broad.igv.util.AmazonUtils.isObjectAccessible(AmazonUtils.java:250) ~[igv.jar:?]
at org.broad.igv.aws.S3LoadDialog.lambda$loadButtonActionPerformed$0(S3LoadDialog.java:106) ~[igv.jar:?]
at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]

Any idea where it has an issue? The same BAM file is working if using "Add File | Add file from URL".

[brainstorm]

Indeed @welcomege, sorry about this bug. We detected this regression through another lab that was setting this up two days ago, we'll investigate and come up with a fix soon... it is related with AWS Storage tiers that apparently does not affect our lab, so that's why it took a while to detect this regression on others.

[jrobinso]

@brainstorm if a patch needs to be applied to the released IGV in the near term it should be on branch 2.8. I can merge any fix to master. I've been working on an igv.js and igv-web release for several months now, when that's out I'll return to focus on igv desktop, in the meantime though I can do a fairly fast release for patches on the 2.8 branch.

[welcomege]

@brainstorm , Thank you very much. I was testing and found IGV version >= 2.7.1 & <= 2.8.0 are working properly. We are now using 2.8.0 and looking forward to the fix :).

[brainstorm]

@welcomege @jrobinso I just fixed it in v2.8.10...umccr:2.8.10_fix_storage_tier_null based on tag v2.8.10... sorry, for the oversight, I was doing a .tostring() on a null field (🤦🏻 , as returned by AWS STANDARD objects).

Since our lab does not have many objects on plain/untouched STANDARD tier anymore we didn't notice that :/

Jim, will you fetch that commit and push it for release 2.8.11? Do you want me to pull request against some other specific branch? Whatever incurs the least amount of work for you Jim, open to suggestions.

[jrobinso]

A PR against branch 2.8 would be easiest for me. Thanks.

…

On Sun, Oct 18, 2020 at 8:17 PM Roman Valls Guimera < ***@***.***> wrote: @welcomege <https://github.com/welcomege> @jrobinso <https://github.com/jrobinso> I just fixed it in v2.8.10...umccr:2.8.10_fix_storage_tier_null <v2.8.10...umccr:2.8.10_fix_storage_tier_null> based on tag v2.8.10... sorry, for the oversight, I was doing a .tostring() on a null field (🤦🏻 , as returned by AWS STANDARD objects). Since our lab does not have many objects on plain/untouched STANDARD tier anymore we didn't notice that :/ Jim, will you fetch that commit and push it for release 2.8.11? Do you want me to pull request against some other specific branch? Whatever incurs the least amount of work for you Jim, open to suggestions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#620 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHD2HDBABMO7VICJDRA52LSLOVV7ANCNFSM4GTFKKEQ> .