Reproduced from Plant Science Bulletin 40(1):6-8 (1994), a reprint of Plant Science Bulletin 34(2):5-7 (1988). How to Write an Influential Review (1) Michael L. Rosenzweig (2), Jerrold I. Davis (3) and James H. Brown (4) "It is not the critic who counts. The credit belongs to the (one) who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, and comes short again and again; who knows the great enthusiasm, the great devotions; who at the best knows in the end the triumph of high achievement, and who at the worst, at last fails while daring greatly." -- Theodore Roosevelt. Often we are asked for our anonymous written opinion of a colleague's research proposal or manuscript. Most of us respond whenever we can, because it is one of the most important ways that we contribute to the development of our science. Such peer review is the primary mechanism by which we, the community of scientists, affect the distribution of limited grant funds and journal pages. Our reviews have enormous influence on the direction and rate of progress in our disciplines. It is not much help if we use this influence unwisely, our our review is not taken seriously. How should a review be fashioned so that it is taken at face value and not discounted? What sort of influence should it try to exert? Surely a review should seek to identify and encourage the most promising and innovative research. Yet it is precisely the newest ideas that are the least tested, the most controversial and easiest to criticize. In fact, history reveals that new theories often are incomplete and often do contain serious errors. But wouldn't it have been a tragedy if Natural Selection had been rejected because Darwin founded it upon incorrect mechanisms of inheritance? If we are to advance our science, it is necessary that we take risks and actively encourage the development of new concepts, theories, and methods. If we as reviewers are afraid of mistakes, and insist that our peers write airtight proposals, then who will dare to tackle the difficult questions? If we required that a proposal be so well described that we can visualize every aspect of the research, is the work really likely to produce any surprises or major new discoveries? We have to be honest and point out potential problems, but, above all, we must strive to identify and express our enthusiasm for new ideas and innovative approaches. How do we write a review that will convince panel members and program directors that a proposal, for all its rough edges, is an eminently worthwhile risk, an exciting adventure with a high payoff if it succeeds? The Research Support Liaison Committee in Ecology, Evolution and Systematics took this question to the staff members of NSF's Division of Biotic Systems and Resources. The way we phrased it reflects a nagging perception that we shared with them: With funds scarce relative to need and the average quality of proposals very high, is there a tendency for reviewers, panel members, and program directors to be conservative, avoid risks, and look for grounds to criticize a proposal? Is any fault or loose end likely to prevent funding? Must we avoid negative comments in our reviews if we feel a proposal should be funded? The Correlation Between Detail and Negativity in Reviews The more negative the tenor of a review, the more detail it has. This is a fact. We know no reason why it has to be so, but it is. Negative reviews are often full of well-reasoned objections. Positive reviews are more often brief statements of approval. They may be full of glowing adjectives, but they rarely contain the details and logical arguments which would give them substance. Faced with positive reviews lacking substance and with well-documented negative criticisms, panels are often swayed by the negative comments. What else can we expect? If NSF is to discount negative comments about promising, but risky and even somewhat flawed proposals, it must have good reasons for doing so. Accentuate the Positive We all know what a long, negative review looks like. It is time to produce some long, positive ones. When you encounter a good proposal or manuscript, spend your time on that one. Describe in detail what makes it good. Remember the NSF panels and ad hoc reviewers are drawn from a wide spectrum of our peers. Just because we recognize the merits of a proposal does not mean that other reviewers will appreciate them. State why the problem is important, what contribution the proposed research will make, and why the investigators are well-qualified to do the work. Avoid hyperbole and be specific. Point out weaknesses, too, and explain why they have not put you off. Give the program director or editor a good excuse to follow your advice. Many reviewers seems to think it is their primary responsibility to discover and call attention to all the flaws in a proposal or manuscript. Perhaps they have the attitude that it should be deemed worthy until proven otherwise. They may sound fair, but it is, in truth, pernicious. It makes us over into petty bookkeepers, subtracting the value of each counterfeit penny without noticing that they are coming from a solid gold box. If we adopt the attitude that a proposal does not deserve funding unless the research is daring, novel or interesting, then we should place more emphasis on the positive aspects of a good proposal and write longer, more constructive positive reviews. We need to remember that it is much more damaging to our discipline to suppress an important contribution than to fund or publish a questionable piece of research. New ideas and conflicting data cannot have any influence unless they are developed, whereas serious errors will usually be detected and corrected, either by the investigator before publication or by the scientific community soon afterward. This is why it is essential to be broad-minded, and to consider the potential importance of a piece of research as well as to search for flaws. Regulate the Proportion of Positive Comments Do not undermine what is intended to be a positive review by devoting more attention to criticism than to supporting comments. The overall impression your review makes depends upon the proportion of criticisms which are positive. You cannot counter that fact with an introductory or concluding paragraph. If there are three pages of negatives, and the first sentence says, "This is a manifestly important proposal and should be fully funded," what do you think the panelists notice? Their overall negative impression cannot be reversed. Believe it. If you really think a proposal is that good, fill the review with your reasons and mention the negatives briefly. The score assigned by reviewers to NSF proposals is also very influential. As much as panel members and program officers try to read the reviews carefully, it is a fact that proposals that receive low scores or average less than "very good" have a difficult time. Scores should reflect the priority that we place on supporting a particular program of research, not the number of nits that can be picked from the way the proposal is written. In writing a review we would all do to bear in mind the difficulties we each have in writing a proposal to support the most exciting of our own research. If you feel compelled to note every flaw, send the details directly to the author, and point out in your review that you have done so. The panel will then be even more convinced of your positive opinion because they see you are taking so much time to help. For example, a prescient reviewer of Darwin might have written, "I have my doubts as to the validity of Mr. Darwin's ideas on the process of heredity, and I have written to him in detail about these doubts. But even if I am correct, Mr. Darwin's proposal to visit the Galapagos Islands remains extremely valuable." And just to be sure that you realize Darwin is hardly the only example, consider whether it would have been wise to discourage R. A. Risher's researches because his models lacked the component of genetic drift. Would you have suppressed Eldredge and Gould's work on punctuated equilibria because they first insisted that punctuation had to be connected to speciation? Would you have retired R. H. MacArthur as soon as he made the illogical jump from resource-use overlaps to competitive alphas? Negative Reviews Simultaneously, we should learn to write shorter negative reviews. Simply admit that you found nothing particularly exciting or novel. It is very important, however, to write such reviews, thus calling attention to proposals that are solid but unexciting and unlikely to result in significant advances. If other reviewers have expressed similar concerns, the panel will have little difficulty making an evaluation. The panel needs help ensuring that critically short funds are not wasted on pedestrian projects. The above is not intended to suggest that we should endeavor to be less critical. Serious criticisms and substantial concerns should always be expressed, but this can be done by dispassionate language without indulging in _ad hominem_ assaults. Critics of art and literature, whose criticisms are often published and who earn their keep from them, feel they must entertain their readers with a rapier wit, caustic comments, piercing put-downs, and acid cuts. Many appear to have decided that criticism is a written version of prize fighting, except that in boxing, low blows are against the rules. Leon Wieseltier calls it "aggression as an intellectual instrument". (5) Unfortunately, all too many negative scientific reviews seem to have been written by put-down artists. This is not only cruel and cowardly (at least the literary critic signs his piece), but it minimizes their influence as well. The editor, panelist or program director is driven to sympathize with the victim. This may mean that if you and you alone noticed the flaws, but reported them intemperately, your criticism will be ignored. Moreover, such aggressive attacks leave a lasting impression of unprofessionalism on your part. The Payoff to our Whole Science Basic science in America is often under attack as a social luxury, and an expensive one at that. Usually the attacks are oblique. Politicians ask what direct, immediate benefits to expect from our work, or give golden fleece awards to projects whose titles make them easy targets. Dr. Janet V. Dorigan of the Department of Energy has observed that when scientists are under attack, they circle 'round, wagon train style. The physicists aim outward at their opponents. Biologists, on the other hand, aim inward, at each other. Their weapons, of course, are disparaging reviews and negative comments. The earth pulses with fascinating ecological and evolutionary questions, and threatens with environmental concerns. The questions are as intellectually challenging as those facing any other scientific discipline. The answers are essential to deal with the environmental problems that beset the world. But we cannot convince other scientists (or the public, or government officials) of the importance of our work if we seem to be calling each other incompetents. Why should anyone want to invest in a bunch of incompetents. In order to convey a more accurate impression of the view of our collective labors, we all need to make a conscious effort to tolerate diverse ideas and unconventional approaches, and to promote independence and originality. Robert Reich has written that "Technological innovation is largely a process of imagining radical alternatives to what is currently accepted." (6) In our reviews, we must encourage that dissent and emphasize the advances it will make possible. A famous cartoon character (Pogo) once said, "We have met the enemy and he is us." This may accurately characterize many human activities, such as ethnic and religious prejudice, war, and degradation of the environment. But it cannot be permitted to be true of the scientific enterprise, which is, in its very essence, mutualistic and collaborative. The peer review system has obvious limitations and imperfections, but it is the best means that scientists have devised to evaluate each others' work. It is up to us, the community of ecologists, evolutionists and systematists, to use the peer review system carefully and wisely. One then can make it serve the goal we all share: the rapid advancement of our science. 1 This is the first contribution of the Research Support Liaison Committee in Ecology, Evolution and Systematics (Constituent Societies: Amer. Soc. Natur., Amer. Soc. Zool., Amer. Soc. L&O, Amer Soc. Plant Taxon., Bot. Soc. Amer., Ecol. Soc. Amer., Soc. Study. Evol., Soc. Syst. Zool.). It is in the public domain. 2 Department of Ecology and Evolutionary Biology, University of Arizona, Tuczon AZ 85721 3 L. H. Bailey Hortorium, Cornell University, Ithaca NY 14853 4 Department of Biology, University of New Mexico, Albuquerque 87131 5 The New Republic, 16(12):42, 1987. But notice that even professional critics often make fools of themselves with negative reviews: Henderson, Bill (1986,1987) Rotten Reviews I & II. Pushcart Press, New York. 6 New Republic 3(8):32, 1987. Acknowledgements: The Division of Biotic Systems and Resources, US National Science Foundation, directed by Dr. John Brooks, provided freely of their time to discuss the issues with the committee. Dr. James T. Callahan suggested the main idea of this editorial. The following members of the Committee contributed to its planning or discussed an earlier draft of it or both: Drs. Dave Allan, Walter Eames, Doug Gill, Steven Green, Jim Lawrey, Jane Lubchenco, Norton Miller, Larry Pomeroy, Muriel Poston, Paul Risser, Dick Root, Larry Slobodin, Rick Vari, Elizabeth Wells.