Galaxy Communication Server

[bgruening]

Realtime communication server and its user-interface

Galaxy makes it very easy to analyse data but communicating with other researchers
and sharing tricks or asking questions is too hard and a for many people big hurdle.

We should promote https://biostar.usegalaxy.org wherever we can, but I think there is also a need
for quick, instant help ... like developers use IRC.

This PR integrates a realtime communication server and it's user-interface.

This is how it looks like:

It features a chat-application with a gloabl room where all users can communicate with each other.
To offer more specilised rooms for bigger Galaxy instance we have integrated persistent communication rooms that can be configured
by an admin via galaxy.ini. These additional global chat-rooms will stay even if no user is logged in and can be
joined/followed by all interested users. Moreover, we included an option to create personal chat rooms.
These rooms will not be listed to all users.

This is how you can use it:

At first you need to install a few requirements.

• . GALAXY_ROOT/.venv/bin/activate # activate Galaxy's virtualenv
• pip install flask flask-login flask-socketio eventlet # install the requirements

As a next step start the communication server with something like this:

• ./scripts/communication/communication_server.py --port 7070 --host localhost

Please make sure the host and the port matches the ones in ./config/galaxy.ini and
set the secret_id in galaxy.ini

We implemented this as an external implementation as
we were not sure if everyone agrees that such a feature should be deeply integrated into Galaxy.

As a consequence and to protect user-privacy we disabled the communication server and the admin as well as the
user needs to explicitly enable it.

The communication server can be controlled on three different levels:

1. The admin can activate/deactivate the communication server via ./config/galaxy.ini. [off by default]
2. The user can activate/deactivate it in their own personal-settings via Galaxy user preferences. [off by default]
3. The user can activate/deactivate communications for a session directly in the communication window. [on by default]

Credits:

Thanks a lot to Anup Kumar (@anuprulez) a Galaxy first time contributor, Rémi Marenco (@remimarenco) and Eric Rasche (@erasche). It's always a pleasure to work with you guys!

Any comments welcome! What do you think about such a feature, any other way how we should integrate this.

Thanks and see you all at GCC!

[martenson]

@erasche

If things were positive, go back and do tighter integration / better UX.

Once we accept it in the codebase we have to maintain it or disable/remove it. I am not saying it is impossible, I am merely pointing out/highlighting that by diving into this we are creating a job for future us that is not trivial.

My users will not use biostars as it is an external site

This is fairly hard to believe. I believe you though.:)

This is extremely site specific. If their local galaxy admins / staff bioinformatician say "hey, I'll be available on chat" then that site's users will use it.

(topic: large support overhead)
I want to believe but the sceptic in me says that sync communication (especially to discuss complex problems) is not necessarily a benefit due to overhead. However you @erasche and @bgruening have more experience supporting users than I do so I defer to you.

Overall, I think that extending Galaxy with this is of no problem to Main or devteam as long as it is not deployed on Main. Given we do not have a good way how to make this pluggable (other than including it and using config as you have done) the concept is fine.
I would argue strongly against deployment on Main but that is not the topic here.

There is a certain nuance in expanding software this way that could be perceived as making it shallow instead of focused, or complex instead of simple. This is notoriously difficult to identify though.

[hexylena]

Once we accept it in the codebase we have to maintain it or disable/remove it. I am not saying it is impossible, I am merely pointing out/highlighting that by diving into this we are creating a job for future us that is not trivial.

To start, it is already disabled. It's disabled in like 3 different places :) And sure, we understand. I believe the quote

Rule #1 of open-source: no is temporary, yes is forever.

is somewhat apropos here. (xref). I would counter that this feature would not be under any of the guarantees made by the project (like reproducibility) so if the maintenance burden was much too high in the long term it could be removed.

If there are specific maintenance burden concerns maybe we can address those.

My users will not use biostars as it is an external site

This is fairly hard to believe. I believe you though.:)

99% of their questions are specific to unpublished tools and workflows. It would not be a good experience for them nor others, so I do not encourage them to use it. :( I don't know what the best solution is here.

(topic: large support overhead)
I want to believe but the sceptic in me says that sync communication (especially to discuss complex problems) is not necessarily a benefit due to overhead. However you @erasche and @bgruening have more experience supporting users than I do so I defer to you.

My experience here is that my PI and users mostly talk to me through text based google hangouts, and never talk to each other. There's no site-specific forum which is what I would love, somewhere my users can exchange knowledge without me being the intermediary.

Much more relevant to small sites. (other GalaxyAdmins please feel free to chime in here!)

Overall, I think that extending Galaxy with this is of no problem to Main or devteam as long as it is not deployed on Main. Given we do not have a good way how to make this pluggable (other than including it and using config as you have done) the concept is fine.
I would argue strongly against deployment on Main but that is not the topic here.

Yeah, I am not sure what the user experience would be like of this being integrated into main, but I have to side with you that it might be unpleasant and just adding noise.

With that said, I'm going to be optimistic here, irc.freenode.net#bioinformatics is quite active. If there are so many people using main, then maybe a community chat could flourish there as well.

There is a certain nuance in expanding software this way that could be perceived as making it shallow instead of focused, or complex instead of simple. This is notoriously difficult to identify though.

No, no, no. You see, we're on our way to the world's best, general purpose bioinformatics operating system, Galaxy OS. Just as emacs is a great OS lacking an editor, so is Galaxy! ;)

Our users can watch videos, run software, (now) chat, and with docker-compose support we could have a VNC server GIE that lets you browse the internet. Everything you could ever want.

Hmm... given that I'm cult of vi, maybe it's time to start the lean, mean vim alternative... ;)

long story short, I agree. In a perfect world (to me, speaking as a deployer and admin) Galaxy would be perfectly modular and I would say "right, my focus is on X, Y, and Z and I want to provide additional services A, B, and C to my community" and then be able to choose to install only the components that I need. Possibly unfortunately, we are nowhere near that. We have all or nothing, and the nothing brings extra overhead for those who do want a feature that we believe is appropriate and useful to the vast majority of Galaxy instances, i.e. medium sized ones.

[remimarenco]

I think the point is: Does this add enough value to our users to allow it into Galaxy? Does irc #galaxyproject add enough value to galaxy admins / tech users?

Do our users currently need more features or more guidance?

@jennaj could jump into the discussion and share her thoughts about it. Could this help you in your daily main support?

I believe it could also be a pre-room to Biostar redirection. If a question is already answered on Biostar, this could help redirect people (we could even think about a bot looking for keywords in questions and propose Biostar links...but in a galaxy far far away future).

[martenson]

@remimarenco

Do our users currently need more features or more guidance?

I would argue for neither - they need more stability, reliability, and discoverability.

Could this help you in your daily main support?

Imo this would be at least one person's full-time job just to answer chats on Main. We do not have that person. (also if you don't mind let's continue the discussion about chat on Main somewhere else - maybe even a new issue - so this thread is clean)

I believe it could also be a pre-room to Biostar redirection. If a question is already answered on Biostar, this could help redirect people (we could even think about a bot looking for keywords in questions and propose Biostar links...but in a galaxy far far away future).

Here you are designing functionality that is way more complex than what is proposed here. This problem is even globally unsolved (e.g. SO). We have questions on Biostars that have been asked dozens of times - if we want to implement this we should start there. (but we are getting off topic here:))

If it is simple and useful to a subset of deployers(as it seems now) I am in favor of this, but one of my biggest concerns is scope bloating.

[shiltemann]

Just my 2 eurocents here, but I would love to see this available in one way or another.

Our users also don't know about biostars and mainly contact me via mail or chat with questions. Often these are also questions about site-specific things (e.g. the data in our shared libraries, custom tools made specifically for their group, or what were the setting to upload via FTP again? etc) that could not be answered on biostars even if they did know about it.

I would also love to be able to make a chatroom for a course where the participants can help each other out and discuss the materials, also after the course has ended, and keep their momentum going. I feel they are often quite enthusiastic after a course, but then get stuck when they try using it on their own and are perhaps hesitant to 'bother' me with questions and sort of lose interest/confidence

I can understand the concerns for big servers like main, but for smaller ones like ours, where users all know each other, but are spread over different floors/buildings and thus not really talking to each other, I think this could definitely help facilitate communication (and hopefully bypassing me for a lot of it ;P).

don't know the best way to handle this code but I'm definitely excited about the potential here and can't wait to try this out :)

[mvdbeek]

I think my opinion is pretty much the same as @shiltemann's, and I have even heard this suggestion from users before ... I think that the support questions about custom tools (or 'hey, why is my job grey for so long ?') probably applies to whole lot of smaller servers out there.

[martenson]

Those are very valuable insights, thank you @shiltemann @mvdbeek !
The use cases with local data libraries, settings/setups, custom tool usage, and training are all awesome.

[bgruening]

Thanks for this discussion, this is why we labelled it RFC.

I think I understand the usage, some of its benefits, and your enthusiasm however I am at least ambivalent about this PR. A few points that I see as important in the equation are below. Some of these points are more relevant for Main (or a big instance) deployment, others apply to the whole approach.

• One more server to maintain when deployed.
• A feature to maintain that is fairly disjointed with what Galaxy does now. We don't even have public user profiles.

We explicitly disabled this feature by default and "hide" it under ./scripts/. Moreover, we made it as pluggable as possible with as less changes to Galaxy core as possible. If we can reduce this please give us some pointers.
We invested some time in getting the socket.js like server more deeply integrated, but this might be a bigger project and is only worth if this feature should be tightly integrated into Galaxy. Which does not seem to be the case. We have foreseen this, so we have made it the way it is currently.

• This is a barebone/raw implementation of something, that is usually very polished in other instances. Chat apps are often the flagships of UX because people use them intensely and often. The Galaxy implementation would either have to improve on this PR or be always viewed as the subpar one.

Then let's improve it. Not sure if you tried it to call it raw, but some specific points would be nice on which we can improve.

• Duplicates the Biostar channel.

As pointed out in the PR description: This is not a duplicate of Biostar. Biostar serves a different use-case and is not-real-time. I see this feature as instant help for users, from users. I actually have no ambitions to join this chat to answer questions.

• Vast majority of answers on Biostar are from within Galaxy Committers (+3ppl).

For me this means we failed (until now) to get people help each other. Also this does not scale and needs to be changed. And hopefully this PR will help.

With this trend interpolated the chat would either be significant overhead for support (sync communication!) or a dead end for users that will never hear back.

This is not what this PR is about. I don't think we (as Galaxy-Admins, Committers) should, or could answer questions this fast. But maybe users can. We have many users staying on our Website and waiting, trying tools, reading help pages. All this power is lost currently. These people can help other people if we empower them to do so.

Actually, we had the idea to display some text/link inside the chat-window to state that for more elaborated questions biostars is a better place, but here comes the next problem. Biostars is very specific to one domain, at least until know.

We see many very specialised Galaxy instances, including Cheminformatics and Language-Processing. Biostars is not known to these users and yet we want to support them or at least give them tools to support each other.

Given how much of a noob I am in using Galaxy could you please provide me with a fictional 'story' of how this feature was successfully used for users benefit?

Simply: I want that biologist A helps pharmacologist B as easy as possible as concrete as possible - and more importantly without the middleman - me.
I would love to see new friendships between research disciplines, new workflows that are shared, credits that you can gain, the feeling that you can actually help others ... all this which is not easily possible within our current setup.

A completely different aspect is that people are shy. They do not like to post questions on a forum that stays forever. I see this more the default than the exception. With this feature we give them a more encapsulated environment - for small questions.

In my opinion, in an ideal world, this would be an installable community-maintained feature/plugin that would not be distributed with Galaxy by default (like a tool).

Yes and we tried our best to do so, given the current Galaxy code. Maybe we can create an extra folder under scripts, called third-party add-ons.

I completely agree that we need to be very careful of adding too much to Galaxy that is too specific or needs too much maintenance, but we should discuss this as a community and not judging from a usegalaxy.org standpoint. So thanks again to you all for this discussion!

(As a side note: I hope that with these small encapsulated projects, we can attract more JS developers to the project and I'm glad @anuprulez took this opportunity to learn the client side of Galaxy)

[anuprulez]

This discussion and analysis have been instrumental to me (and possibly to all new to the Galaxy) to recognize the crucial pointers where a new feature could impact an entire application and its users.

Once I had read in a blog that in Google, a problem is worthy enough if it has a potential to affect 1 billion people. Along these lines, if a feature is able to cut down the time users take to solve their respective problems, it should be great to adopt it in one way or the other. But at the same time, the incurring cost in terms of man hours to maintain the feature should be kept under check.

Thanks a lot for imparting your views especially @bgruening @erasche @remimarenco :) :)

[bgruening]

We should move this as well into the plugin directory or at least make it communication independent.

[anuprulez]

@bgruening Can we create a separate folder in the config/plugins directory for the communication channel codebase and move files like generic-nav-view.js and the communication folder?
For this +// communication channel bootstrap modal
we can create a new css file and move these styles there. It cannot go within the communication.css file as it loads inside the iframe.

[jgoecks]

Yes, please create a new directory in config/plugins, maybe tabs or menu and move all needed files to there.

[bgruening]

@erasche @guerler @martenson: Please check the latest code. @anuprulez tried to add a more generalized plugin-mechanism for the masthead, that could hopefully be used by other extensions as well. Not sure how to proceed with the css part.

[jgoecks]

Thanks for all the hard work on the @bgruening @anuprulez

IMO, for this to truly be a plugin approach, what's needed is a way to avoid touching menu.js every time a new tab is needed. I imagine something like this in menu.js after all the standard tabs have been loaded:

for each plugin in config/plugins/menu:
  load plugin
  add tab

I wonder if it'd be possible to reuse some code from the visualization plugin framework to make this happen or if @guerler has a better idea of how to approach this.

[jmchilton]

• Is modifying the menu logic to make this feel more "plugin-y" essential before this can be merged, I get the desire and I want that to be more pluggable for other reasons but it seems outside the scope of this PR IMO?
• I know multiple people expressed reservations - is there an actual -1 on this? It has at least 2 +1s from committers if not, so @bgruening I guess can merge as he sees fit.

[martenson]

👍

[bgruening]

Thanks @martenson and everyone for review!
Especially to @anuprulez great work!