Genomics in the Cloud: Using Docker, GATK, and WDL in Terra

Front Cover
"O'Reilly Media, Inc.", Apr 2, 2020 - Computers - 496 pages

Data in the genomics field is booming. In just a few years, organizations such as the National Institutes of Health (NIH) will host 50+ petabytes—or over 50 million gigabytes—of genomic data, and they’re turning to cloud infrastructure to make that data available to the research community. How do you adapt analysis tools and protocols to access and analyze that volume of data in the cloud?

With this practical book, researchers will learn how to work with genomics algorithms using open source tools including the Genome Analysis Toolkit (GATK), Docker, WDL, and Terra. Geraldine Van der Auwera, longtime custodian of the GATK user community, and Brian O’Connor of the UC Santa Cruz Genomics Institute, guide you through the process. You’ll learn by working with real data and genomics algorithms from the field.

This book covers:

  • Essential genomics and computing technology background
  • Basic cloud computing operations
  • Getting started with GATK, plus three major GATK Best Practices pipelines
  • Automating analysis with scripted workflows using WDL and Cromwell
  • Scaling up workflow execution in the cloud, including parallelization and cost optimization
  • Interactive analysis in the cloud using Jupyter notebooks
  • Secure collaboration and computational reproducibility using Terra
 

Contents

Chapter 1 Introduction
1
A Primer for Newcomers to the Field
13
Chapter 3 Computing Technology Basics for Life Scientists
53
Chapter 4 First Steps in the Cloud
79
Chapter 5 First Steps with GATK
115
Chapter 6 GATK Best Practices for Germline Short Variant Discovery
147
Chapter 7 GATK Best Practices for Somatic Variant Discovery
183
Chapter 8 Automating Analysis Execution with Workflows
209
Chapter 10 Running Single Workflows at Scale with Pipelines API
269
Chapter 11 Running Many Workflows Conveniently in Terra
295
Chapter 12 Interactive Analysis in Jupyter Notebook
331
Chapter 13 Assembling Your Own Workspace in Terra
373
Chapter 14 Making a Fully Reproducible Paper
413
Glossary
441
Index
445
About the Authors
468

Chapter 9 Deciphering Real Genomics Workflows
245

Other editions - View all

Common terms and phrases

About the author (2020)

Dr. Geraldine A. Van der Auwera is the Director of Outreach and Communication for the Data Sciences Platform (DSP) at the Broad Institute of MIT and Harvard. As part of her outreach role, she serves as an educator and advocate for researchers who use DSP software and services including GATK, the Broad's industry-leading toolkit for variant discovery analysis; the Cromwell/WDL workflow management system; and Terra.bio, a cloud-based analysis platform that integrates computational resources, methods repository and data management in a user-friendly environment. Van der Auwera was originally trained as a microbiologist, earning her Ph.D. in Biological Engineering from the Université catholique de Louvain (UCL) in Belgium in 2007, then surviving a 4-year postdoctoral stint at Harvard Medical School. She joined the Broad Institute in 2012 to become Benevolent Dictator For Life of the GATK user community, leaving behind the bench and pipette work forever.

Dr. Brian O’Connor is the Director of the Computational Genomics Platform at the University of California Santa Cruz (UCSC) Genomics Institute. There, he focuses on the development and deployment of large-scale, cloud-based systems for analyzing genomic data. These include the NHGRI AnVIL and NHLBI Bio Data Catalyst platformsas well as the Dockstore site for workflow and tool sharing. Brian is active in standards efforts and is the cochair of the Global Alliance for Genomics and Health Cloud Work Stream where he works on API standards for cloud interoperability. Brian joined UCSC from the Ontario Institute for Cancer Research where his previous projects included leading the technical implementation of worldwide, cloud-based analysis systems for the PanCancer Analysis of Whole Genomes project, creating the Dockstore, and managing a successful rebuild of the International Cancer Genome Consortium’s Data Portal.

Bibliographic information