Enabling Large Model of Reverse Time Migration

The extreme computational complexity of modern seismic imaging techniques has led to a decades-long search for the most efficient algorithms, the best optimization techniques and the most suitable hardware platforms on which to run these algorithms. The purpose of our study is to advance this search by exploring the potential of the new Blue Gene/Q (BG/Q) supercomputing system for Reverse Time Migration (RTM) imaging. Our study analyzes BG/Q's performance scalabilities over both imaging model size and number of computing nodes. Our end-to-end RTM implementation includes a 3D isotropic model, absorbing boundary conditions, a 9x9x9 stencil, forward and backward passes with application of source and received terms, snapshot storage and imaging condition. Our implementation uses domain partitioning of a single model over hundreds of nodes, which allows us to run models much larger than are common today. For example, we can easily run a 3D model of 20483 points. For the models tested, we have observed nearly perfect performance scaling over hundreds of nodes and are exploring scaling beyond that point. Even with initialization, data input, and image output, we have achieved an end-toend throughput of 580 million stencils per second per node (or 590 billion stencils per second per rack). Furthermore, we achieved a 14.93x rack-to-rack performance improvement over the previous generation, BG/P. Our implementation exploits all levels of parallelism available, including MPI, OpenMP for multithreading, and built-in intrinsics for vector operations. We avoid the need for scratch space on disk by storing snapshots in memory. Our paper shares insights and experience on how to efficiently make use of BG/Q's HW resources to advance the state-of-the art of industry practice. It provides a guiding example for enabling seismic imaging algorithms using BG/Q to achieve an unprecedented level of performance.