tag:blogger.com,1999:blog-7844526396210378482.post5941637563903551410..comments2024-01-31T09:23:26.925+00:00Comments on Noel O'Blog: A non-random method to improve your QSAR results - works every time!Noel O'Boylehttp://www.blogger.com/profile/03288289351940689018noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-7844526396210378482.post-14409738551796142412010-06-17T11:57:00.361+01:002010-06-17T11:57:00.361+01:00@Jeremy: Ok, forget the oranges :-)
I defintely a...@Jeremy: Ok, forget the oranges :-)<br /><br />I defintely agree that clustering could be useful for selecting the dataset in the first place.Noel O'Boylehttps://www.blogger.com/profile/03288289351940689018noreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-3380275229509502542010-06-17T11:37:23.566+01:002010-06-17T11:37:23.566+01:00@baoilleach
Hi, well the words are used may be too...@baoilleach<br />Hi, well the words are used may be too much. The molecules will come from the same dataset so they all should be oranges. <br />However using random splitting only can leave molecules which are very similar (like Cl -> Br) in 2 different sets. Then when you look at the performance of the model it performs very well.<br />While if you cluster the molecules and then use the cluster to split in training/test sets you avoid this effect. <br />After it depends also on what you are trying to do , work on well defined space (or try to explain existent results), or try to find a model whcih can be apllied to slightly different chemistry (not too different but more than just aromatic substitution), and it is important to know how the model performs on something different.Jeremynoreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-61011443626728021692010-06-16T16:49:10.487+01:002010-06-16T16:49:10.487+01:00Noel, I see what you mean. Are the distributions o...Noel, I see what you mean. Are the distributions of descriptors in QSAR usually normal?Karolhttps://www.blogger.com/profile/17032498937126368914noreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-8093752445715554272010-06-16T15:09:06.747+01:002010-06-16T15:09:06.747+01:00@Karol: It depends what you mean by "even spa...@Karol: It depends what you mean by "even spacing". Let's say you have only one descriptor, and it has a normal distribution about the mean. A random subset of molecules will also have a normal distribution; however, a diverse subset will be 'even spaced' from the minimum value to the maximum, and have a flat distribution. Any model derived from this will give equal weight to all points on the range, instead of focusing on the central part of the range, where most of the points are. As a result, it will perform poorer on a true test set (as far as I can see).<br /><br />If the training set is small, it is all the more important that you don't intervene or you will have an even greater effect. One solution could be to generate 100 random models and get the mean of the predictions found. <br /><br />Regarding "Is not the point of QSAR is to approximately model a specific, closed set of structures?" Indeed it is - this is the concept of the domain of applicability; in other words, the model won't work for all molecules.<br /><br />The choice of what structures to model is slightly different than the topic of the blog post, which is rather how to accurately assess the quality of a predictive model given a dataset respresentative of "the sort of chemicals you want to model".Noel O'Boylehttps://www.blogger.com/profile/03288289351940689018noreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-49362100413370091622010-06-16T14:54:18.414+01:002010-06-16T14:54:18.414+01:00@Jeremy: I understand, but I don't know if it&...@Jeremy: I understand, but I don't know if it's very useful to see how well a model trained on apples can predict oranges. A separate model trained on oranges might be a good idea.<br /><br />@Egon: I agree. Accurate assessment of predictive ability is not straightforward, and there are some subtle traps.Noel O'Boylehttps://www.blogger.com/profile/03288289351940689018noreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-87627292078192667882010-06-15T21:11:39.136+01:002010-06-15T21:11:39.136+01:00I agree with you. A comment, though... random divi...I agree with you. A comment, though... random division into subsets should also give even spacing in your chemical space, at least for very large sets.<br /><br />If the training set is small, it seems to me that helping it to span the entire relevant space is a good thing. Is not the point of QSAR is to approximately model a <i>specific, closed</i> set of structures? I don't think you expect QSAR to work universally.<br /><br />In any case, the set will always be insignificant in size compared to the entire possible chemical space. The compounds you want QSAR to work for (drug-like) constitute a tiny fraction of it, so too much diversity will be counterproductive for prediction. A well-defined training set, biased towards the sort of chemicals you want to model, seems to be a good thing. I've seen this argumentation in the context of <a href="http://dx.doi.org/10.1038/nchembio.180" rel="nofollow">HTS and docking libraries</a>.Karolhttps://www.blogger.com/profile/17032498937126368914noreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-3343687420163602232010-06-15T16:37:16.448+01:002010-06-15T16:37:16.448+01:00In my thesis I used bootstrapping to 'visualiz...In my thesis I used bootstrapping to 'visualize' the effect of difference test set selections (repeated random selections, btw). The point there was not to estimate the effect of those selections, but to show that your quality measures (R^2, Q^2, RMSEP) are intrinsically variant; in short, a Q^2 of 0.92 is *not* better than one of 0.89. Not for the typical sizes of QSAR data sets. Below 100 compounds the variance is even larges and can easily go up to 0.05.<br /><br />People indeed very often underestimate how their assumptions effect their statistical models.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7844526396210378482.post-28239383348750149922010-06-15T16:25:04.876+01:002010-06-15T16:25:04.876+01:00Another way would be to have non random spiting bu...Another way would be to have non random spiting but use similar compound together in one set. So the training set and test set are normally different.<br />It can be used to see the performance of the model on a different set (without neighbour artefact in the model making it better than it is).<br />One can also use 3 sets, one very different as test set (using cluster, or grid…), then the second set is spitted in two using random for a training and validation set.<br /><br />This could help reduce the optimism of a model; however the results of the test sets could be not very impressive, and people may not publish/use/trust the results.Jeremynoreply@blogger.com