Contact site moderator
Dr. Robert E. Belford
Recent attempts by science research consortiums to validate results from contemporary publications have exposed a tremendously low percentage (~10%) of reproducible results. This astoundingly poor reproducibility can partly be attributed to flaws in static publishing format, namely lack of detail in describing complex methodologies. As new technologies emerge, science research and education have adapted to include these new offerings. However, there has been no accompanying evolution in publication format. The scientific publishing model initially introduced in the mid seventeenth century is the same model used to present today’s research findings. Articles are still published as text heavy tomes with the occasional scheme, diagram, or table to demonstrate a point. Why is the science community still employing ancient text standards? How should the publication template be modified to accurately communicate advanced techniques? This presentation describes a new trend in publication format, namely video publication, and its ability to accurately capture complex techniques, thereby increasing reproducibility of results. Recently conducted case studies will be shown in support of video publications as a valid communication venue in scientific publishing.
State of Publishing
Science research is presented using largely the same publication model introduced in the mid seventeenth century. With the exception of color printing and electronic dissemination (cataloguing and searching functions), not much has changed. While these advances have improved user accessibility, they have not improved the ease of application or utility of the presented findings to other areas of research.
Conversely, as new technologies emerge and are applied to various disciplines, research and education is advancing at a rapid rate. Despite these advances, both new research and education models are presented as text heavy documents with the occasional diagram or graph display of obtained results. Why hasn’t the presentation of these new reports evolved to include the concomitant advances in research and education?
Reproducibility on the Decline
A severe consequence of the current stagnate state of research communication presentation platforms, reproducibility in science is on the decline. Irreproducibility of research has surfaced as a serious concern in recent years with numerous studies dedicated to reproducing the findings of contemporary publications. Each of these studies has exposed an alarming percentage of scientific findings that cannot be reproduced.
Two years ago, Bayer Health called attention to the issue internally.1 Interested in quantifying reproducibility from their labs, they selected 67 published experiments and repeated them. Comparing their new results to those previously published, they found that a staggering 64.2% could not be replicated. Of the rest, a mere 20.9% were fully replicated, 11.9% results were partially replicated, and the remainder was not applicable.
A separate study conducted by C. Glenn Begly while he served as the head of global cancer research at Amgen Inc. calls attention to the reproducibility in cancer research.2 Begly’s research group identified 53 papers from high profile journals and tested their claims. Similar to Bayer’s findings, Begley’s group found that only 89% of these selected works could not be replicated.
Deleterious Effects of Irreproducibility
Obvious negative impacts of irreproducible published results are numerous: contamination of the literature with potentially false findings, assumptions based on these reports, and a backward trajectory of science advancement, among others. A more downstream consequence, research budgets allocated through government grants and independent industries are at risk. As the number of studies investigating the reproducibility of results in published research climbs, funding agencies are beginning to apply more stringent reproducibility guidelines to their applications.
Echoing this concern, an article in the Journal of American Medical Association provides a break-down of the biomedical research budget and funding.3 The report reveals a recent decline in budget trends despite the steady incline in the number of researchers. An updated study using recent fiscal calendars shows that research budgets continue to decline.4
The authors of the study posit high failure rate of new technologies as one possible cause for the budget decrease. Failings in technology can be traced back to the poor reproducibility of published research and the inability to translate basic research findings into advanced applications. Though some of these failings may be due in part to complications of scaling factors, lack of reproducibility certainly plays a large role.
Considering funding availability directly impacts science progress, science will suffer greatly if reproducibility is not addressed and remains an instigator in declining research budgets. To mitigate this serious and immanent threat, the science community must acknowledge the problem and alter the way in which information is shared for increased reproducibility.
Revolution in Science Publishing
The reproducibility epidemic has not gone without attention by the publishing groups responsible for distributing research reports. As the number of reproducibility investigations and published reports climbs, irreproducibility awareness has increased. Just this past spring, Nature published an editorial commentating on ways in which researchers can alter their published works to increase reproducibility.5 Among these suggestions was increased methodology details and specific reagent information, as the lack of sufficient detail in published reports was implicated as a main contributor of the irreproducibility problem.
What was not addressed in the Nature editorial was how researchers could provide greater level of detail when describing their recent findings. If publication methods or the ways in which information is presented to the community have not changed in centuries, how can scientists be expected to present their advanced findings with the required level of detail needed for replication? Publishers must adapt and provide new outlets for sharing methods and results to the science community. These new outlets must facilitate detail-oriented presentation.
Here at JoVE, the Journal of Visualized Experiments, we have approached this problem and developed a novel publication format, peer reviewed video publication. Using the traditional text manuscript model with all standard sections present (Abstract, Introduction, Protocol, Representative Results, Discussion, and References), JoVE then converts the text into a script. Next, JoVE films the entire contents of the manuscript in the researcher’s own lab or clinic. Dynamic video presentation provides a visual level of detail that mirrors that of an in-person training experience. Through observing how recent findings are obtained and analyzed, the time and resources spent reproducing and learning new published methods will drastically decrease while also increasing the reproducibility of results.
JoVE publishes 70 video articles a month in both the physical and life sciences with sections in Chemistry and Applied Physics, among others. In addition to citations, the video articles receive several thousand views with total monthly web traffic typically reaching 300,000 visitors. The high number of web visitors shows that the science community is interested in video publication.
Recently published video article in JoVE Chemistry
Does Video Publication Deliver?
Given the high usage statistics, we were interested in determining how usage translates to increased utilization of published methods. To this end, we interviewed several researchers at various institutions to inquire about their interaction with JoVE. A postdoctoral researcher at Baylor College of Medicine, Nikolaos Giagtzoglou, shared his JoVE experience with us.6 While developing a new application of his research, Dr. Giagtzoglou looked to the literature to learn three techniques for working with Drosophila (fruit flies). After spending time reading several traditional text publications, Dr. Giagzoglou was unable to learn from these reports and utilize the techniques in his experiment.
The obvious next step was to contact the author of the paper and request a training session. Unfortunately, when Dr. Giagtzoglou reached out to the authors, he found that “it can be hard to coordinate busy schedules to travel and learn the method.6” During his search for more literature sources, Dr. Giagtzoglou stumbled upon a JoVE article presenting the technique in video format and immediately recognized its value. Using the JoVE video article, Dr. Giagtzoglou shares with us that “I really had no starting point to learn these techniques, and JoVE was invaluable...Watching a JoVE video-article is so much more helpful than reading just materials or methods, which can have grammatical mistakes, bad syntax, or may be hard to interpret.6”
Similarly, Dr. Casey, an Assistant Professor at Purdue University, found a JoVE article when searching for methods describing how to dissect the suprachiasmatic nucleus in mice, a complicated neuroscience procedure.6 About her JoVE experience, she said, “I had a collaborator in Buffalo who knew the SCN surgery, and I’ve seen it done before. By using the JoVE video, we saved money in travel costs to go to Buffalo repeatedly to learn the technique.6” A cost analysis showed that Dr. Casey saved over six thousand dollars that would have been spent on travel expenses and reagents because the JoVE video article provided the necessary visual training needed to learn and perform the procedure independently.
In addition to saving valuable time, money and resources, video articles have the potential to help build the foundation for new ideas and research by providing the necessary instructions for non-experts to learn and implement new techniques in their own laboratory. In fact, Dr. Casey’s testimony reflects just that, “I’ve been doing research for 20 years, and having JoVE makes things so much easier. You can educate yourself on research other scientists are doing around you and get familiarized on a technique before you try it. I like to watch techniques and refresh myself on experiments I haven’t conducted in 18 years but need now.6” In this way, video articles facilitate the transfer of knowledge from one research lab to another from anywhere around the globe without the need to travel.
Researchers have also used video articles to validate results. After publishing novel results in a high impact journal, Dr. Jonathan T. Butcher at Cornell University explains that he received numerous inquiries from researchers in the field questioning the validity of the results within since “these other labs were not able to reproduce our results using the written instructions in the methods section of our novel research paper.6” Frustrated with having to defend his research in response to these claims, Dr. Butcher published this same method in JoVE and no longer receives correspondence disputing the validity or reproducibility of the lab’s results. Dr. Butcher believes this is because “the video format conveys complicated methods significantly better than text alone and helped to validate our novel results.6” Dr. Butcher’s testimony confirms that traditional text manuscripts are no longer sufficient to describe the complexity of contemporary research methods.
Video Publication for the Advancement of Science
Despite a decrease in reproducibility, science journals are publishing an increasing number of articles. This opposite trend suggests that peer reviewed journals are not assisting the research community with the tools necessary to accurately present complex research. While science continues to advance at a rapid rate, publishing has not kept pace with these advances.
The first major change to science publishing in 350 years, JoVE utilizes video technology to capture complex research in both physical and life sciences. These video accounts provide the viewers with the necessary level of detail for understanding, learning, and reproducing, the research presented. Increased transparency through visualization also serves to validate methods and their results. Building confidence in research through increased validation will encourage funding agencies to grant larger budgets to research labs and institutions. The culmination of these myriad benefits will provide a springboard for the advancement of science.