-
Notifications
You must be signed in to change notification settings - Fork 270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Accurate and efficient target prediction using a potency-sensitive influence-relevance voter #229
Comments
@swamidass I am reading through the papers you posted and have a couple quick questions. I also edited the original post. Is the IrvPred web server and source code at http://chemdb.ics.uci.edu/cgibin/tools/IrvPredWeb.py the software referenced in this paper? In this comparison with SVM and Random Forest, the features come from fingerprint similarity. This makes sense because it is most similar to the IRV approach. Have you directly compared standard classifiers (e.g. Random Forest) trained with fingerprint similarity features versus using the fingerprint bit vector directly as the features? I haven't yet surveyed everything you posted and am trying to prioritize my reading. |
I believe that is the software, though this is to Baldi's site so I cannot 100% be sure. In this study, we did not compare to RF or fingerprint bit vectors as a direct input to a neural network (I'll abbreviate this as FP -> NN).
In this study, I think there are a couple key things to key in on:
|
Thanks. I did appreciate the advantages from reading this paper and #228. I'm still trying to figure out where it fits in #174 because I think our working defining of "deep learning" may be "high-parameter neural networks". I'll have to see how we've discussed low-parameter NNs in other sections, the options being that we consider them along with high-parameter NNs or treat them as a competing method. Even if it is common knowledge in the field, I would still like to find a reference that directly compares (Tanimoto similarity on fingerprints -> SVM) or (Tanimoto similarity on fingerprints -> random forest) with (fingerprints -> random forest) or (fingerprints -> neural networks) on the same data. That will help readers from outside the cheminformatics field, which I expect to be most readers. |
I think that is a legitimate concern. IRV are low parameter, so they are not quite in the standard group of Deep Learning methods. On that basis, it is possible we may want to exclude them from the review, or at least point out that they are not exactly the current pattern. Deep Learning, however, is more than just "high-parameter". I think a better way to define Deep Learning is: "a collection of new techniques for building neural networks, including higher parameter models, recursive and convolution networks, improved architectures, and improved training strategies." The IRV, by this definition, is a class of Deep Learning. Although it is not high parameter, it (1) uses more hidden layers than normal (there are three hidden layers, plus a kernel layer, between the input and output, (2) it uses extensive weight replication to reduce weights substantially. Of course, it will have limitations compared to the new methods. Honestly, I expect that they will eventually be outclassed by better versions of (for example) #53. We are just not there yet. |
About references that show NN's on fingerprints directly don't work so well. That is a tall request. There was just so much unpublished experience of people trying this approach (albeit with older regularization techniques) that it is offhand mentioned in cheminformatics all the times. Given the bias against publishing negative results, that will be a hard reference to find. Now it is entirely possible that using more advanced regularization that it can work on par with SVMs, RFs and Tanimoto similarity. That has to be established, however, before FP->NN's are a convincing baseline method against which to benchmark improvement over state of the art methods. I think is really the key point. While it is hard to produce a reference that shows FP->NNs are poor, there is really no body literature that demonstrates that they reliably produce results comparable with RF, SVMs and Tanimoto similarity. This alone is enough to discourage use of FP->NNs as a baseline method of comparison. |
@cgreene , as our circus master, can you please comment on the definition that you want to use for deep learning? @agitter offers: "high-parameter neural networks" I think this is more accurate: "a collection of new techniques for building neural networks, including higher parameter models, recursive and convolution networks, improved architectures, and improved training strategies." I think this is important to clarify because high-parameter networks have been around for a long time. They just never worked well, so people avoided them. It is only with new DL techniques (e.g. dropout, resnet, relu, batch-normalization, etc.) that they started to work. This is a pretty fundamental cross-cutting issue to resolve. Can you please weigh in @cgreene ? |
@agitter asks for some benchmark papers. These are some important papers... https://www.ncbi.nlm.nih.gov/pubmed/15961479 This competition on HIV data is pretty important and shows SVMs (from my team) outperforming everything else: http://www.agnostic.inf.ethz.ch/index.php Though we did follow up work that demonstrated the IRV really does better: https://www.ncbi.nlm.nih.gov/pubmed/20378557 I think the competitors in that competition are helpful. You'll see every which algorithm there is there. And MinMax-kernel SVMs (or MinMax-sim IRVs) outperforms everything. |
@swamidass I'm interested in @cgreene's feedback on this as well, but I should say that my definition probably is in line with what you posed above. More thoughts soon. |
I still don't have time articulate my complete thoughts, but my questions about how to classify IRV may not matter much in the end. I plan to include it in this section and am trying to think about where it fits in the new narrative. I have an outline in mind and will write it up as soon as I can for feedback. |
My thoughts are much in line with @swamidass. I just filed #243 to touch on improvements to the introduction to more clearly define what we mean by If you expand the text from that section of the PR, you can see on lines 48-58 of the revised version the definitions that we have been using. We been relatively permissive saying that multi-layer NNs used to construct features, at some stage, count. We also - for what it's worth - note that by this definition such models have existed for more than 50 years in the literature. @swamidass : I'd be thrilled if you want to refine this via a PR on the intro to highlight a more restrictive perspective on deep learning. It will require us to start making harder calls as to what qualifies. |
Side note: I just got back from a trip to UCI where I chatted with Pierre. I should have asked in person, but I'm just catching up on this after my return! (with regards to |
Hope you got to talk to him about this =). He is one of the early leaders in the field that not so many people know about. Any how, I can take a crack at the intro. |
Yea - we chatted a lot about the science but not about this review (missed opportunity - doh!). Do you think you could get him onto github for this? It would be great to get his perspective + feedback! |
I don't think he is an internet "chatter". He generally avoids reviews. But
you can certainly try.
S. Joshua Swamidasshttp://swami.wustl.edu/
On Fri, Feb 17, 2017 1:24 PM, Casey Greene notifications@github.com wrote:
Yea - we chatted a lot about the science but not about this review (missed
opportunity - doh!). Do you think you could get him onto github for this? It
would be great to get his perspective + feedback!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
https://jcheminf.springeropen.com/articles/10.1186/s13321-015-0110-6
Edit:
https://doi.org/10.1186/s13321-015-0110-6
The text was updated successfully, but these errors were encountered: