Salvus: An open-source code package for large-scale waveform modelling and inversion

PI: Andreas Fichtner (ETH Zurich)

Co-PIs: Dave A. May, Matthew G. Knepley, Johan O. A. Robertsson, Martin O. Saar

July 1, 2017 - June 30, 2020

Project Summary

We propose to develop Salvus, the first open-source code package for large-scale waveform modelling and inversion that is (1) built on the basis of modern programming principles, (2) modular and easily extensible, (3) capable of harnessing large homogeneous and various heterogeneous HPC architectures that are available today, (4) easily adaptable to future architectures, requiring minimal code modifications, (5) easy to use, and (6) continuously unit-, integration- and performance-tested.

This project will position Salvus as a top wavefield modelling and inversion package in the exascale era. To ensure performance of Salvus on today's and tomorrow's supercomputing platforms, our work will focus on cross-architecture developments, code and I/O optimisation, and systematic testing and validation. This will be complemented by actions to increase and broaden the usability and impact of Salvus. They include workflow developments, the implementation of frequency-domain solvers, and extensions of the physics that can be modelled.

The team of proposers and Salvus developers covers fields including Computational Science, Applied Mathematics, Seismology, Exploration and Environmental Geophysics, Geothermal Energy, and Geofluids. This team is ideally prepared to achive the goals of this project, and to ensure that Salvus becomes a code package that is widely used to solve real-world problems in the wave-based imaging sciences.

Imaging with waves: The propagation of mechanical waves is the foundation of numerous imaging techniques in a wide range of disciplines, including seismic tomography for the Earth's internal structure, non-destructive testing of infrastructure and buildings, and medical ultrasound tomography.

State of the art: Driven by the need to increase imaging resolution, several wave propagation solvers have been developed in the early 2000s for the supercomputers of that era. Naturally, these solvers were developed without today's modern software engineering principles, such as unit and integration testing, continuous integration, and a librarisation that facilitates the transition from one generation of supercomputers to the next. Furthermore, changing the parallel configuration or other solver parameters cannot be done at runtime, but requires recompilation. This constitutes a severe limitation.

Salvus: Responding to the need for a modular and extensible wave propagation code built with state-of-the-art, we developed the spectral-element package Salvus. By leveraging the PETSc library and its DMPlex finite-element API, Salvus is running in parallel with performance already comparable to established packages.

Written in C++, Salvus has easy access to the PETSc API and to the linear algebra template library Eigen. This allows us to model a variety of element types in a general way that minimises redundant code. C++ enables us to utilise modern testing frameworks, and to model a variety of element and physics types succinctly and with minimal code repetition. We achieve this through two techniques; (1) classical C++ inheritance and, (2) C++ template mixins, which is a newer technique that acts similarly to subclassing but avoids virtual function calls as the functional dependencies are determined at compile time.

Salvus simulates wave propagation in fluids, solids, as well as coupled fluids and solids. It operates on both hexahedral and tetrahedral meshes in 3D, and quadrilateral and triangular meshes in 2D. Current applications include wave propagation through the whole Earth, the Japanese islands region, the Greenlandic ice sheet, seismic exploration-scale models, machine elements, and the human breast. As expected for a well-written wave propagation solver, the scaling of Salvus is nearly perfect.

Proposed work: Here we propose to position Salvus as a top wavefield modelling and inversion package in the exascale era. To ensure performance of Salvus on today's and tomorrow's supercomputing platforms, our work will focus on the following tasks:

  1. Cross-architecture developments, leveraging Salvus' mixin-based design to implement hardware-specific versions of compute-intensive code segments, while leaving most of the code unchanged.
  2. Workflow developments to facilitate the solution of large-scale inverse problems, including the automatic orchestration of a large number of HPC jobs.
  3. General code opimisations to achieve maximal performance from single nodes to full machine runs This involves the implementation of optimal degree of freedom ordering in PETSc DMPlex, optimal load balancing, and hybrid parallelism.
  4. I/O optimisation to handle the enormous data volumes needed in adjoint simulations. Sub-tasks include the incorporation and extension of a previously developed wavefield compression library, and the interfacing to modern parallel seismic data formats.
  5. Incorporation of frequency-domain solvers using Krylov methods together with geometric-based multi-level preconditioners. This is intended to open Salvus for emerging applications in exploration geophysics with very large numbers of sources (hundreds of thousands).
  6. Extension of an automatic testing framework that performs unit, integration and performance tests at least on a nightly basis.
  7. Extensions of the physics that Salvus is able to simulate in order to broaden is applicability and impact. This includes the incorporation of distributed sources, full self-gravitation, coupled systems with ocean tidal loading, and the solution of the diffusion equation.

Deliverables: The main deliverable of this project is a wave propagation and inversion package that is portable and optimised across various node architectures, and scalable at least to the number of nodes provided by current CSCS resources. Further deliverables include (1) an integrated wavefield compression module to minimise I/O in adjoint simulations, (2) Asynchronous I/O of wavefields, (3) a framework for automatic unit, integration and performance testing, (4) optimised workflows for waveform inversion, (5) a frequency-domain (Helmholtz) solver based on geometric pre-conditioners, and (6) Extended physics (diffusion equation, full gravity, tidal loading, ...).

Impact: Salvus will be the first open-source wave propagation code package built on modern software engineering principles, and fit to be used with today's and tomorrow's large-scale heterogeneous HPC infrastructures. The modular nature and maintainability of Salvus, combined with its performance will position it well in the emerging exascale era.

Salvus will be essential for research and applications in wave-based imaging. Covered domains - represented by the proposers and their respective communities - include whole-Earth tomography, seismic exploration, medical imaging, and geothermal energy.

Through the Salvus developments, we will unavoidably encounter limitations related to today's supercomputing architectures. These limitations may be related to the need to orchestrate a large number of forward and adjoint solves in the solution of inverse problems, or limits in job submission and I/O. We hope that these issues will influence the design of future CSCS systems, thereby allowing us to solve problems in the future that we may not be able to solve today.