August 30, 2019 | By: Julie Poucher Harbin, Senior Writer, DCI
Randles' team is one of 15 investigator-led teams being given priority to run their world-changing projects on the U.S. Department of Energy's “Aurora” exascale computer when it debuts in 2021. Randles' simulations could shed light on the mechanisms driving cancer metastasis.
Amanda Randles, PhD, earned her Bachelor’s Degree in computer science and physics from Duke. Upon graduation, she spent three years as a software engineer at IBM, before earning her MS and PhD from Harvard University. She returned to Duke as faculty in 2015 and was named the Alfred Winborne and Victoria Stover Mordecai Assistant Professor of Biomedical Sciences, Biomedical Engineering, Pratt School of Engineering, in 2018.
Amanda Randles, PhD, assistant professor of biomedical sciences, computer science and mathematics at Duke University, has spent the greater part of the past decade building computer simulations of the circulatory system to both tackle the risks for and progression of cardiovascular disease and to chart the movements and spread of cancer.
On August 13, she was named a senior member of the National Academy of Inventors (NAI) — an organization that recognizes, with this honor, active faculty, scientists and administrators “with success in patents, licensing and commercialization” who have “produced technologies that have brought, or aspire to bring, real impact on the welfare of society.”
With 114 U.S. patents and 16 international patents in her name, and counting (mainly in parallel computing), the Pratt School of Engineering professor and Duke Cancer Institute member made international headlines in 2015 for creating the first three-dimensional simulation of blood flow through the human body.
The model — built from full-body CT and MRI scans and angiograms from a single patient — compared extremely well with real-life blood flow measurements (in vivo) and lab-based scenarios (in vitro). The simulation, for which Randles wrote the code (HARVEY), was able to accurately predict and follow the path of hundreds of millions of blood cells as they moved through the 10,000-mile long vascular system.
At the time of the model’s debut at a computer science conference, Randles had just joined Duke and was running her own lab, having completed a 2.5-year post-doctoral fellowship at Livermore National Laboratory and Dana Farber Cancer Institute and a PhD in applied physics and computer science at Harvard University where her focus was on large-scale computer simulations of fluid dynamics.
Her 3D simulation of blood flow through the body consumed the entire Sequoia supercomputer system at the Lawrence Livermore National Laboratory — a 20-petaflop system capable of 20 quadrillion calculations per second and one of the top 10 supercomputers at the time. It included all of the arteries greater than one millimeter in diameter.
But Randles quickly understood that she was going to need a computer with at least a thousand-fold more processing power both to track individual red blood cells and to advance her work (supported by a $2.5 million NIH grant) simulating cancer cell movement through the circulatory system.
She just had to wait.
High Risk, High Reward
In this image, 3 million red blood cells are being simulated moving through a microvasculature topology. A circulating tumor cell is shown in white. 1586 MPI tasks were used on the Duke Compute Cluster. Credit: John Gounley, Duke
In the summer of 2018, the U.S. Department of Energy’s Argonne National Laboratory selected Randles’ computational fluid dynamics model and 14 other projects to ready their codes for the nation’s first exascale computer “Aurora” — expected to be delivered to the lab sometime in 2021.
The investigator-led project teams — hailing from five universities and two national labs — are now preparing key applications, libraries, and infrastructure for the architecture and scale of Aurora. These include projects that could lead to the development of alternative energy sources, safer vehicles, and new materials, as well as a greater understanding of the origin and nature of the universe and how the brain works. Randles' team project Extreme-Scale In-Situ Visualization and Analysis of Fluid-Structure-Interaction Simulations uses data science to create a cellular-level model that is expected to be "a critical step towards elucidating the mechanisms driving cancer metastasis."
Randles and her co-PI on the project Erik Draeger, PhD, from Lawrence Livermore National Laboratory, are currently working with Jeffrey Ames, a PhD student at Duke, and Argonne researchers Joseph Insley, MFA, MS; Silvio Rizzi, PhD, and post-doc Saumil Patel, PhD, from Argonne National Laboratory, to prepare the computational studies for Aurora.
“We’re working on numerical techniques to scale to large regions,” explained Randles. “While we won’t be able to simulate all of the red blood cells in the human vasculature with Aurora, due to memory constraints, we’ll be able to track individual cancer cells and red blood cells at unprecedented lengths, scales and densities.”
According to the Aurora website, the $500 million supercomputer will be capable of performing “a quintillion calculations per second.” It's being built on a future generation of Intel® Xeon® Scalable processor accelerated by Intel’s Xe compute architecture. The Cray Slingshot™ fabric and Shasta™ platform will form the backbone of the system. Argonne staff, in collaboration with experts from Intel and Cray, are training the teams on the Aurora hardware design and how to program it. Intel is even putting together a hackathon focused on the team’s code to help optimize it for Aurora.
Argonne will also be hiring a post-doc in the next year or so, to be based there, who will work to get the code running and help bridge the collaboration between Duke and the national lab.
Randles doesn’t know exactly how much time they’ll get on Aurora, but like the other Early Science Program teams, her team will get “first dibs access and priority in their queue.”
They’re already getting millions of hours on the Summit supercomputer at Oak Ridge National Laboratory (the current most powerful) through the Director’s Discretionary Allocation, and on Argonne’s supercomputer Theta, through the Aurora Early Science Program.
“Our simulations are running hundreds of millions of time-steps,” Randles explained. “We’re generating a petabyte of memory, every time step, adding up to hundreds of millions of petabytes. We’re using the entire memory footprint.”
Even with the anticipated computing power of Aurora, Randles said it won’t be feasible to output all the data, post-process it, and then look for something interesting later. So the team is currently tackling techniques to visualize and analyze the data in Aurora on-the-fly.
Call for Collaborators
Half of the work of Randles’ lab is focused on cardiovascular disease. Using the blood flow simulation model, they’ve so far been able to create a personalized framework for modeling how vascular changes influence the ankle-brachial index; establish that for accurate coronary flow analysis side branches must be included; and develop a non-invasive tool for assessing endothelial shear stress.
She's also active on the faculty committee for the Chancellor’s Translating Duke Health Cardiovascular Disease Initiative “Keeping the Heart Young” and works closely with colleagues at Duke Cardiology, Duke Pediatric Cardiology, Duke Radiology, and the Duke Department of Biomedical Engineering.
In cancer, up until this point, her lab’s research has been more basic-science focused.
The lab is not specifically tracking cancer cells in one patient at this stage (like they were able to do with red blood cells), but in collaboration with cancer researchers at MIT, they’ve matched their data to several cell lines, including in kidney cancer, leukemia and breast cancer.
They’ve been using human in vitro models to characterize the mechanical properties of cancer cells, like bending and dilational elasticity.
“We’ve been looking at cancer cells moving through constricted micro-fluidic devices and observing how they change as they pass through,” said Randles. “Our hope is to study the mechanical properties of cancer cells that influence where they’re more likely to metastasize, what their residence time is at different locations, what their hit rates are, and what properties we should target for future studies.”
Now, the team is looking to get more data for their models, specifically human patient-derived or patient-inspired geometries.
“We’ve been building the computational tools for a long time and I think we’re at the point where we could start getting more collaborators that are not just doing the mechanical properties with in vitro models,” she said. “It would be great to start touching base with researchers at DCI now.”
For their cancer-specific studies, the team is looking to identify partners who can provide access to human data. In particular, they’re are interested in investigating the interplay between hemodynamics and cell-specific properties.
“For example, using scans of a patient’s vasculature after diagnosis of the primary tumor, we could create personalized 3D flow profiles,” Randles explained. “We’re interested in studying whether this data is predictive of progression or metastasis formation. We want to assess whether we could use our tools to identify such a predictive metric between the morphology or the flow characteristics. For this project, we’d be interested in talking with researchers working on small cell lung cancer.”
Another objective is to look at the clustering of circulating tumor cells.
“Can we use these studies to see if one of those clusters is going to break down? How does that influence where they’re moving through the vasculature, and how do clusters form?” said Randles. “Once we get the infrastructure in place, and get on Aurora, we can start asking those questions.”