This is my first post and as such, I apologize in advance. This blog is for me by me. I want to improve my science writing skills and abilities to think skeptically. I am not trying to be negative to the articles I review, I am just making an attempt to dig a little deeper and come up with scientific concerns I have. I will probably be wrong and you are free to comment and correct me. This is my learning exercise and I hope others will find it enjoyable as well. This first try, I realize is not written to the lay person. I will attempt to do that in the future, mostly I just wanted to try my hand at it with a fairly low key paper. Thanks for the patience and if you find something here great. If now, I hope I will :)
In a recent study by Cavatorta et al. (2008), they attempt to compare site models of positive molecular selection using the FEL model of HyPhy (Kosakovsky Pond an Frost, 2005) to the M2 and M8 Models of PAML (Yang, 2007). The main question they ask is which group can accurately identify resistance polymorphisms in the plan eIF4E gene with more accuracy.
This gene provides a great opportunity to answer the accuracy question for two reasons: 1) simulated data, the normal first approach, can work but usually lacks biological realism, and 2) empirical data is difficult because the correct answer is not known. eIF4E is one of the most well studied genes for resistance and so offers a background literature of functional data identifying resistance alleles in the gene. This offers a scenario in which the difficulties of empirical studies are overcome for the most part, at least in the opinion of the authors.
They do not put forth a hypothesis as to which method will do better but they do claim that the models should work.
So what are the results, after running both procedures M8 identifies 2 sites, and FEL identifies 3, both identifying site 76 with a 0.9 posterior probability cut off. All three sites identified by FEL are known to confer resistance while only site 76 identified by M8 is known. They then conclude that FEL has higher power and greater precision.
What I find most unappealing in this conclusion is the low level of stringency used. While arbitrary, 0.95 is the normal p-value cut off used in statistics. If 0.9 was selected a priori, I guess it would weaken my argument but no mention of this was made in the methods and the problem was not addressed. Both models identify site 76 with a significant p-value (M8:0.99 and FEL:0.95) and no other site is detected by either method with the normal cut off. The conclusion of precision and power is unfounded either way. The distinction between the two is on a single case (one of the problems of empirical studies) that may favor one method over the other anyway. When normal statistical stringency is used the identified sites are identical between the two methods.
I think this is a great study because it is getting at the problems of model comparison and accuracy, I am just not so sure about their conclusions or if it actually answers the question they were asking. Both methods did identify a site under selection that confers resistence but when so many other sites (16 total) should also have given signal it is concerning that both methods have such low power.
