Tag Archives: biology teaching

On Questionable Research Practices

Leave a reply

Ecologists and evolutionary biologists are tarred and feathered along with many scientists who are guilty of questionable research practices. So says this article in “The Conservation” on the web:
https://theconversation.com/our-survey-found-questionable-research-practices-by-ecologists-and-biologists-heres-what-that-means-94421?utm_source=twitter&utm_medium=twitterbutton

Read this article if you have time but here is the essence of what they state:

“Cherry picking or hiding results, excluding data to meet statistical thresholds and presenting unexpected findings as though they were predicted all along – these are just some of the “questionable research practices” implicated in the replication crisis psychology and medicine have faced over the last half a decade or so.

“We recently surveyed more than 800 ecologists and evolutionary biologists and found high rates of many of these practices. We believe this to be first documentation of these behaviours in these fields of science.

“Our pre-print results have certain shock value, and their release attracted a lot of attention on social media.

  • 64% of surveyed researchers reported they had at least once failed to report results because they were not statistically significant (cherry picking)
  • 42% had collected more data after inspecting whether results were statistically significant (a form of “p hacking”)
  • 51% reported an unexpected finding as though it had been hypothesised from the start (known as “HARKing”, or Hypothesising After Results are Known).”

It is worth looking at these claims a bit more analytically. First, the fact that more than 800 ecologists and evolutionary biologists were surveyed tells you nothing about the precision of these results unless you can be convinced this is a random sample. Most surveys are non-random and yet are reported as though they are a random, reliable sample.

Failing to report results is common in science for a variety of reasons that have nothing to do with questionable research practices. Many graduate theses contain results that are never published. Does this mean their data are being hidden? Many results are not reported because they did not find an expected result. This sounds awful until you realize that journals often turn down papers because they are not exciting enough, even though the results are completely reliable. Other results are not reported because the investigator realized once the study is complete that it was not carried on long enough, and the money has run out to do more research. One would have to have considerable detail about each study to know whether or not these 64% of researchers were “cherry picking”.

Alas the next problem is more serious. The 42% who are accused of “p-hacking” were possibly just using sequential sampling or using a pilot study to get the statistical parameters to conduct a power analysis. Any study which uses replication in time, a highly desirable attribute of an ecological study, would be vilified by this rule. This complaint echos the statistical advice not to use p-values at all (Ioannidis 2005, Bruns and Ioannidis 2016) and refers back to complaints about inappropriate uses of statistical inference (Armhein et al. 2017, Forstmeier et al. 2017). The appropriate solution to this problem is to have a defined experimental design with specified hypotheses and predictions rather than an open ended observational study.

The third problem about unexpected findings hits at an important aspect of science, the uncovering of interesting and important new results. It is an important point and was warned about long ago by Medewar (1963) and emphasized recently by Forstmeier et al. (2017). The general solution should be that novel results in science must be considered tentative until they can be replicated, so that science becomes a self-correcting process. But the temptation to emphasize a new result is hard to restrain in the era of difficult job searches and media attention to novelty. Perhaps the message is that you should read any “unexpected findings” in Science and Nature with a degree of skepticism.

The cited article published in “The Conversation” goes on to discuss some possible interpretations of what these survey results mean. And the authors lean over backwards to indicate that these survey results do not mean that we should not trust the conclusions of science, which unfortunately is exactly what some aspects of the public media have emphasized. Distrust of science can be a justification for rejecting climate change data and rejecting the value of immunizations against diseases. In an era of declining trust in science, these kinds of trivial surveys have shock value but are of little use to scientists trying to sort out the details about how ecological and evolutionary systems operate.

A significant source of these concerns flows from the literature that focuses on medical fads and ‘breakthroughs’ that are announced every day by the media searching for ‘news’ (e.g. “eat butter”, “do not eat butter”). The result is almost a comical model of how good scientists really operate. An essential assumption of science is that scientific results are not written in stone but are always subject to additional testing and modification or rejection. But one result is that we get a parody of science that says “you can’t trust anything you read” (e.g. Ashcroft 2017). Perhaps we just need to repeat to ourselves to be critical, that good science is evidence-based, and then remember George Bernard Shaw’s comment:

Success does not consist in never making mistakes but in never making the same one a second time.

Amrhein, V., Korner-Nievergelt, F., and Roth, T. 2017. The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ  5: e3544. doi: 10.7717/peerj.3544.

Ashcroft, A. 2017. The politics of research-Or why you can’t trust anything you read, including this article! Psychotherapy and Politics International 15(3): e1425. doi: 10.1002/ppi.1425.

Bruns, S.B., and Ioannidis, J.P.A. 2016. p-Curve and p-Hacking in observational research. PLoS ONE 11(2): e0149144. doi: 10.1371/journal.pone.0149144.

Forstmeier, W., Wagenmakers, E.-J., and Parker, T.H. 2017. Detecting and avoiding likely false-positive findings – a practical guide. Biological Reviews 92(4): 1941-1968. doi: 10.1111/brv.12315.

Ioannidis, J.P.A. 2005. Why most published research findings are false. PLOS Medicine 2(8): e124. doi: 10.1371/journal.pmed.0020124.

Medawar, P.B. 1963. Is the scientific paper a fraud? Pp. 228-233 in The Threat and the Glory. Edited by P.B. Medawar. Harper Collins, New York. pp. 228-233. ISBN 978-0-06-039112-6

A Few Rules for Giving a Lecture

2 Replies

I’ve discussed some of the rules for graphics in publications and preparing posters before but I feel it’s time for a more general discussion of lecturing for scientists. All of us have suffered through at least one poor lecture at scientific meetings and some of us many more. If you are a scientist or educator and must give a short talk or a long lecture, you should not panic since there are just a few rules that can help in communication and reduce potential suffering for you and the audience.

First, let the audience know what precisely you will be discussing in your talk – what is the problem and what you are going to present about it. The opening 2 minutes of your talk is when you can lose two-thirds of your audience. If you are a politician, this may be what you wish to happen, but if you are a scientist do not go there. You do not need to begin by stating the obvious – we all know that the earth is round and biodiversity is under threat – but dive into the details of the particular problem you are going to resolve.

Second, if you are showing powerpoints, follow a few simple rules or again you will lose your audience. Do not put more than a few dot points on a slide, or more than 1 or at most 2 graphs or maps. You must not spend more than 1-2 minutes on each slide or those of us with a sleep deficit will have a power nap instead of listening. Use writing in large letters only so they can be read from the back of the room.

Thirdly, do not use acronyms anywhere. Most of us do not know that DAE means ‘demographic Allee effect’ or that RR means ‘log response ratio’ so if your slides contain GDD or DOC or HBL or ODE you may be losing your audience. In most cases it is possible to write out the meaning of these acronyms without crowding the slide.

Finally, sum up at the end of your talk what you have achieved and what more might be required to completely answer your opening question or problem. The audience will typically take home one or two points you have raised in your talk. Do not expect miracles.

There is an enormous literature on powerpoints and lecturing, much of it more relevant to medical education than to biology. I have put together 8 specific rules for powerpoints and I list these here:

  1. Never use a dark background for your slides. The reason is that in rooms that have too much light, the audience will be unable to read white printing on a dark background. It is best to use black printing on a white or pastel background.
  2. Use at least 28 point font on every slide. If you think this is too large a font, project your lecture and go back 10 meters in a not-too-dark room, and see if you can read what you have written.
  3. Never have more than one graph on a slide. It is impossible to digest 4 or 8 graphs on one slide, and the audience can never read the labels on the axes.
  4. If you use colour on your slide for different lines or points, make the colours strong and check that you can distinguish them from 10 meters.
  5. Never use a table on a slide with more than 4 columns and 4 rows. No one can read most tables used in most talks because the font size is typically too small.
  6. Allow at least one minute to talk about what is the message on each slide. If you are giving a 15-minute talk, you should have no more than 12 slides.
  7. Do not use a photo as a background for a slide. Use photos as photos to make a particular point, and text as text. Do not in general put several photos on one slide.
  8. Do not use animation in your powerpoints unless you have already gotten an Academy Award for your work. If you need to use a short video, imbed it properly and test that it really works and is clear.

I think these two papers make additional points that are useful in developing lectures. Good luck and an early thank you from your audiences.

Blome, C., H. Sondermann, and M. Augustin. 2017. Accepted standards on how to give a Medical Research Presentation: a systematic review of expert opinion papers. GMS Journal for Medical Education 34: doc11. doi: 10.3205/zma001088

Harolds, J. A. 2012. Tips for giving a memorable presentation, Part IV: Using and composing PowerPoint slides. Clinical Nuclear Medicine 37:977-980. doi: 10.1097/RLU.0b013e3182614219

On Defining a Statistical Population

2 Replies

The more I do “field ecology” the more I wonder about our standard statistical advice to young ecologists to “random sample your statistical population”. Go to the literature and look for papers on “random environmental fluctuations”, or “non-random processes”, or “random mating” and you will be overwhelmed with references and biology’s preoccupation with randomness. Perhaps we should start with the opposite paradigm, that nothing in the biological world is random in space or time, and then the corollary that if your data show a random pattern or random mating or whatever random, it means you have not done enough research and your inferences are weak.

Since virtually all modern statistical inference rests on a foundation of random sampling, every statistician will be outraged by any concerns that random sampling is possible only in situations that are scientifically uninteresting. It is nearly impossible to find an ecological paper about anything in the real world that even mentions what their statistical “population” is, what they are trying to draw inferences about. And there is a very good reason for this – it is quite impossible to define any statistical population except for those of trivial interest. Suppose we wish to measure the heights of the male 12-year-olds that go to school in Minneapolis in 2017. You can certainly do this, and select a random sample, as all statisticians would recommend. And if you continued to do this for 50 years, you would have a lot of data but no understanding of any growth changes in 12-year-old male humans because the children of 2067 in Minneapolis would be different in many ways from those of today. And so, it is like the daily report of the stock market, lots of numbers with no understanding of processes.

Despite all these ‘philosophical’ issues, ecologists carry on and try to get around this by sampling a small area that is considered homogeneous (to the human eye at least) and then arm waving that their conclusions will apply across the world for similar small areas of some ill-defined habitat (Krebs 2010). Climate change may of course disrupt our conclusions, but perhaps this is all we can do.

Alternatively, we can retreat to the minimalist position and argue that we are drawing no general conclusions but only describing the state of this small piece of real estate in 2017. But alas this is not what science is supposed to be about. We are supposed to reach general conclusions and even general laws with some predictive power. Should biologists just give up pretending they are scientists? That would not be good for our image, but on the other hand to say that the laws of ecology have changed because the climate is changing is not comforting to our political masters. Imagine the outcry if the laws of physics changed over time, so that for example in 25years it might be that CO2 is not a greenhouse gas. Impossible.

These considerations should make ecologists and other biologists very humble, but in fact this cannot be because the media would not approve and money for research would never flow into biology. Humility is a lost virtue in many western cultures, and particularly in ecology we leap from bandwagon to bandwagon to avoid the judgement that our research is limited in application to undefined statistical populations.

One solution to the dilemma of the impossibility of random sampling is just to ignore this requirement, and this approach seems to be the most common solution implicit in ecology papers. Rabe et al. (2002) surveyed the methods used by management agencies to survey population of large mammals and found that even when it was possible to use randomized counts on survey areas, most states used non-random sampling which leads to possible bias in estimates even in aerial surveys. They pointed out that ground surveys of big game were even more likely to provide data based on non-random sampling simply because most of the survey area is very difficult to access on foot. The general problem is that inference is limited in all these wildlife surveys and we do not know the ‘population’ to which the numbers derived are applicable.

In an interesting paper that could apply directly to ecology papers, Williamson (2003) analyzed research papers in a nursing journal to ask if random sampling was utilized in contrast to convenience sampling. He found that only 32% of the 89 studies he reviewed used random sampling. I suspect that this kind of result would apply to much of medical research now, and it might be useful to repeat his kind of analysis with a current ecology journal. He did not consider the even more difficult issue of exactly what statistical population is specified in particular medical studies.

I would recommend that you should put a red flag up when you read “random” in an ecology paper and try to determine how exactly the term is used. But carry on with your research because:

Errors using inadequate data are much less than those using no data at all.

—Charles Babbage (1792–1871

Krebs CJ (2010). Case studies and ecological understanding. Chapter 13 in: Billick I, Price MV, eds. The Ecology of Place: Contributions of Place-Based Research to Ecological Understanding. University of Chicago Press, Chicago, pp. 283-302. ISBN: 9780226050430

Rabe, M. J., Rosenstock, S. S. & deVos, J. C. (2002) Review of big-game survey methods used by wildlife agencies of the western United States. Wildlife Society Bulletin, 30, 46-52.

Williamson, G. R. (2003) Misrepresenting random sampling? A systematic review of research papers in the Journal of Advanced Nursing. Journal of Advanced Nursing, 44, 278-288. doi: 10.1046/j.1365-2648.2003.02803.x

 

The Snowshoe Hare 10-year Cycle – A Cautionary Tale

2 Replies

We have been working on the ten-year cycle of snowshoe hares (Lepus americanus) in the southwest Yukon since 1975 trying to answer the simple question of what causes these cyclic fluctuations. I think that we now understand the causes of the cyclic dynamics, which is not to say all things are known but the broad picture is complete. But some misunderstanding persists, hence this one page summary. Some biology first.

The snowshoe hare cycle has been known from Canada lynx fur return data for more than 100 years, and of course known to First Nations people much before that. Hares are herbivores of small trees and shrubs, they reproduce at age 1 and rarely live more than 1-2 years. They have 2-4 litters in a summer, with litter size around 4-6. Juvenile losses are high and at best populations increase about three-to-four-fold per year. Almost everything eats them – lynx, coyotes, great-horned owls, goshawks, a long list of predators on the young. Reproduction collapses with rising density and females reduce their output from 4 litters to 2 in the peak and decline phase.

The obvious driving factors when Lloyd Keith and his students began working on the hare cycle in Alberta in the 1960s were winter food shortage and predation. When there is a high hare peak, damage to shrubs and small trees is obvious. But it was quite clear in Keith’s studies that the decline phase continued well after the vegetation recovered, and so he postulated a two-factor explanation, winter food shortage followed by high predation losses. He looked for disease and parasite problems in hares but found nothing.

Testing the winter food limitation would appear to be simple but is fraught with problems. Everyone believes that food is an ultimate limiting factor, so that it must be involved in the cyclic dynamics. We began testing food limitation in the mid-1970s and found that one could add natural food or artificial food (rabbit chow) and apparently have no effect on cyclic dynamics. Hares came to the food grids so the density increased by immigration, but the decline started at the same time and at the same rate as on control grids. So what is the role of food?

Our next attempt was to do a factorial experiment adding food, reducing predation, and doing both together. The details are important, replication was never enough for the manipulated treatments, we did it only for 10 years rather than 20 or 30. What we found was that there was an interaction between food addition and mammal predator exclusion so that the combined treatment increased to a much higher density than any single treatment. But this result came with a puzzle. What is the role of food? Hares showed no evidence of malnutrition in the peak or decline, fed hares did not increase their reproductive output. What produced the strong interaction between food addition and predator reduction?

The next breakthrough came when Rudy Boonstra suggested that predator-caused stress might underlie these strange dynamics. Because we could now measure stress with faecal cortisol measures we could test for stress directly in free-ranging hares. The surprise was that this idea worked and Michael Sheriff capped off the stress hypothesis by showing that not only does predator-induced stress reduce reproductive rates, but the stress effect is inherited maternally in the next generation.

The bottom line: the whole dynamics of the snowshoe hare cycle are predator-induced. All the changes in mortality and reproduction are direct and indirect effects of predators chasing and eating hares. The experimental food/predator interaction was mechanistically wrong in targeting food as a major limiting factor.

This of course does not mean that food is irrelevant as an important factor to study in hare cycles. In particular very high peak populations damage shrubs and small trees and we do not yet have the details of how this works out in time. Secondary chemicals are certainly involved here.

Why does all this matter? Two points. First, the hare cycle is often trumpeted as an example of a tri-trophic interaction of food – hares – predators, when in fact it seems to be a simple predator-prey system, as Lotka suggested in 1925. Models of the hare cycle have proliferated over time, and there are far more models of the cycle in existence than there are long-term field studies or field experiments. It is possible to model the hare cycle as a predator-prey oscillation, as a food plant-hare oscillation, as a parasite-hare interaction, as a cosmic particle – hare oscillation, as an intrinsic social – maternal effects interaction, and I have probably missed some other combinations of delayed-density dependent factors that have been discussed. That one can produce a formal mathematical model of the hare cycle does not mean that the chosen factor is the correct one.

The other point I would leave you with is the large amount of field work needed to sort out the mechanisms driving the population dynamics of hares. Ecology is not simple. This enigma of the ten-year cycle has always been a classic example in ecology and perhaps it is now solved. Or perhaps not?

Boonstra, R., D. Hik, G. R. Singleton, and A. Tinnikov. 1998. The impact of predator-induced stress on the snowshoe hare cycle. Ecological Monographs 68:371-394.

Boutin, S., C. J. Krebs, R. Boonstra, M. R. T. Dale, S. J. Hannon, K. Martin, A. R. E. Sinclair, J. N. M. Smith, R. Turkington, M. Blower, A. Byrom, F. I. Doyle, C. Doyle, D. Hik, L. Hofer, A. Hubbs, T. Karels, D. L. Murray, V. Nams, M. O’Donoghue, C. Rohner, and S. Schweiger. 1995. Population changes of the vertebrate community during a snowshoe hare cycle in Canada’s boreal forest. Oikos 74:69-80.

Keith, L. B., and L. A. Windberg. 1978. A demographic analysis of the snowshoe hare cycle. Wildlife Monographs 58:1-70.

Keith, L. B. 1990. Dynamics of snowshoe hare populations. Current Mammalogy 4:119-195.

Krebs, C. J., S. Boutin, R. Boonstra, A. R. E. Sinclair, J. N. M. Smith, M. R. T. Dale, K. Martin, and R. Turkington. 1995. Impact of food and predation on the snowshoe hare cycle. Science 269:1112-1115.

Krebs, C. J., S. Boutin, and R. Boonstra, editors. 2001. Ecosystem Dynamics of the Boreal Forest: the Kluane Project. Oxford University Press, New York.

Sheriff, M. J., C. J. Krebs, and R. Boonstra. 2009. The sensitive hare: sublethal effects of predator stress on reproduction in snowshoe hares. Journal of Animal Ecology 78:1249-1258.

Yan, C., N. C. Stenseth, C. J. Krebs, and Z. Zhang. 2013. Linking climate change to population cycles of hares and lynx. Global Change Biology 19:3263-3271.

When Should One Retire from a University Appointment?

2 Replies

In the good old days universities had a hard retirement policy that once you reached age 65 you were retired whether you liked it or not. Then in the age of entitlement it was declared that this was discrimination on the basis of age and thus could not be allowed. Universities bemoaned the fact that they had no firm financial projections under the new policy, and many different policies were introduced partly to solve this problem. In some cases you could gradually go to half-time, and then at some age to quarter time, until you eventually did retire, but most of these policies were voluntary.

It is useful to look at the broad picture that these changes produced in the university community. If there was indeed some general plan of development in a particular discipline like zoology, committees could lay out a future hiring plan but it was usually chaos because the time frame was so uncertain. So in my experience most carefully thought out hiring plans went out the window and hiring became ad hoc with the accompanying ‘departmental drift’. So, as a hypothetical example, if a professor in entomology retired, he or she might get replaced by a young assistant professor in microbial genetics.

On a larger scale, we need to look carefully at the consequences of keeping older professors on the books commandeering relatively large salaries. There are no clear rules but in general one might recognize professors that are worn out at age 55 and ought to retire, others that are happy to stop at 65 and relax more, and others who ask to stay on indefinitely. Every case is an individual one. Some of the age 55 scientists are still vigorous and any concerned department ought to work to make their life easier so they can continue to work. Others of the same age should be encouraged to go. The same should occur at age 65. The worry I have most is about those over 65. I give no names but I can list brilliant scientists who continued to be paid and work until they were 75 or 80. I can also list scientists who were brilliant in their time but had passed the gate by 65 (or even 55) but insisted in taking up a position for many years after age 65. This is a tragedy for the individual, for the department, and in particular for young scientists looking for a university position but finding none because the money is tied up in professors well past their use-by-date. I would expect that the only possible solution to this issue is for the university to evaluate every professor over 55 with strict demands of performance if they wish to remain on the payroll and to do this on a 1 or 2 year timetable. No one likes doing such evaluations so perhaps the university would have to hire one of the many hard-nosed CEOs of companies that are seen to be effective at firing all their workers.

None of this is to say that any and all professors who have retired at age 55, 65, 75, or 85 should not be encouraged to continue research work, but they must do it on their retirement savings. In my youth I met a 98 year old Drosophila researcher who was continuing to do valuable research in his long retirement. In Canada the federal research agencies do not seem to care how old you are when they evaluate the quality of your research work and contributions to science, so they at least do not appear to discriminate in awarding research funds on the basis of age. Scientific journals do not ask you how old you are when you submit a potential scientific paper.

There has always been a paradigm that scientific advances are made entirely by young scientists, so that, as the joke goes, almost all mathematicians should be shot after age 30 (that is a joke….). In at least some of the ecological sciences this age paradigm is not correct, but nevertheless I think it is morally recumbent on older professors to realize that their time on the payroll should be limited in order to release funds for the aspiring young scientists who can rejuvenate university departments.

On the benefits of natural history knowledge

Leave a reply

I am reminded today about the importance of ecologists knowing a good deal of natural history. Every species is in some sense a unique experiment in evolution, and our job as population and community ecologists is to understand how these species operate in the ecosystems in which they live. But this means we must know the details about how the species operates, what it eats and who eats it, and in some sense how it thinks about its world. I suspect that this is easier to do with higher vertebrates than it is with insects or protozoa but we need to do the same with all forms of life if we are to achieve ecological understanding.

There is in my experience a great lack of this approach in the universities I have seen. We no longer tend to teach about angiosperm systematics, or mammalogy, or ornithology. These are completely old fashioned, the world’s most condemning epithet. So we turn out biology students in British Columbia that cannot identify a Douglas fir tree (perhaps the most important forest tree in the province) and California students who think the eucalyptus trees originated in Berkeley. That would all be well if we perfected bar-coding on our iPhones for species IDs so we could spend more time learning about where and how these species live and die. But too often we seem to think there is a short cut to understanding species roles. It is always worth exploring short cuts to understanding if we can effectively make a simpler way to explain the world. But we try and fail at this enterprise again and again. Hope springs eternal. We need to know now, so let us assume that all algae can be grouped as one ‘superspecies’ in our models, and all ‘rats’ are bad and need to be exterminated, and adding CO2 to the air will make all plants grow faster. We learn by a lot of difficult and extended research that these are oversimplifications. But then the problem becomes communicating this complexity to politicians and the public who desire simplicity rather than complexity.

This whole task is much easier if you talk to a birder who being keen on birds knows that they all differ in many interesting ecological characters, that some individuals of the same species behave in quite different ways, and that the ecosystem continues to operate with this amazing complexity. So I think one solution to ecological oversimplification is to quiz those who start to tell you about harvesting whales, or poisoning rats, or bringing in genetically modified crops to find out how much they know about the natural history of the species they talk so confidently about. A dose of humility would not hurt our discussions of the current controversies of wildlife and fisheries management.