1887

Abstract

Summary

Digital rock physics (DRP) is a rapidly evolving technology targeting fast turnaround times for repeatable core analysis and multi-physics simulation of rock properties. We develop a rapid and scalable distributed-parallel single-phase pore-scale flow simulator for permeability estimation on real 3D pore-scale micro-CT images using a novel variant of the lattice Boltzmann method (LBM). The LBM code implementation is designed to take maximum advantage of distributed computing on multiple general-purpose graphics processing units (GPGPUs). We describe and extensively test the distributed parallel implementation of an innovative LBM algorithm for simulating flow in pore-scale media based on the multiple-relaxation-time (MRT) model. The novel contributions of this work are (1) integration of mathematical and high-performance computing components together with a highly optimized implementation and (2) quantitative results with the resulting simulator in terms of robustness, accuracy, and computational efficiency for a variety of flow geometries including various types of real rock images. We report on extensive tests with the simulator in terms of accuracy and provide near-ideal distributed parallel scalability results on large pore-scale image volumes that were largely computationally inaccessible prior to our implementation. Permeability estimation results are provided on large 3D binary microstructures including real rocks from various sandstone and carbonate formations. We quantify the scalability behavior of the distributed parallel implementation of MRT-LBM as a function of model type/size and the number of utilized new-generation NVIDIA V100 GPGPUs for a panoply of permeability estimation problems.

Loading

Article metrics loading...

/content/papers/10.3997/2214-4609.201802184
2018-09-03
2020-08-15
Loading full text...

Full text loading...

References

  1. Adler, P., Jacquin, C. and Quiblier, J.
    [1990] Flow in simulated porous media.International Journal of Multiphase Flow, (16), 691–712.
    [Google Scholar]
  2. Alpak, F.O., Gray, F., Saxena, N., Dietderich, J., Hofmann, R. and Berg, S.
    [2018] A distributed parallel multiple-relaxation-time lattice Boltzmann method on general-purpose graphics processing units for the rapid and scalable computation of absolute permeability from high-resolution 3D micro-CT images.Computational Geosciences, 22, 815–832.
    [Google Scholar]
  3. Amdahl, G.M.
    [1967] Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities. In: Proceedings of the April 18–20, 1967, Spring Joint Computer Conference,AFIPS ’67 (Spring). ACM, New York, NY, USA, 483–485.
    [Google Scholar]
  4. Andrä, H., Combaret, N., Dvorkin, J. E. G. J. H., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E., Sain, R., Saxena, N., Ricker, S., Wiegmann, A. and Zhan, X.
    [2013a] Digital rock physics benchmarks-Part I: Imaging and segmentation.Computers & Geosciences, (50), 25–32.
    [Google Scholar]
  5. [2013b] Digital rock physics benchmarks-part II: Computing effective properties.Computers & Geosciences, (50), 33–43.
    [Google Scholar]
  6. ANSYS
    [2010] ANSYS FLUENT Theory Guide.
    [Google Scholar]
  7. Arns, C., Bauget, F., Limaye, A., Sakellariou, A., Senden, T., Sheppard, A., Sok, R., Pinczewski, V., Bakke, S., Berge, L., Oren, P. and Knackstedt, M.
    [2005] Pore-Scale Characterization of Carbonates Using X-Ray Microtomography.SPE Journal, (10), 26–29.
    [Google Scholar]
  8. Arns, C., Knackstedt, M., Pinczewski, W. and Garboczi, E.
    [2002] Computation of linear elastic properties from microtomographic images: Methodology and agreement between theory and experiment.Geophysics, (67), 1396–1405.
    [Google Scholar]
  9. Benzi, R., Succi, S. and Vergassola, M.
    [1992] The lattice Boltzmann-equation-theory and applications.Physics Reports, (222), 145–197.
    [Google Scholar]
  10. Berg, S., Ott, H., Klapp, S., Schwing, A., Neiteler, R., Brussee, N., Makurat, A., Leu, L., Enzmann, F., Schwarz, J.O., Kersten, M., Irvine, S. and Stampanoni, M.
    [2013] Real-time 3D imaging of Haines jumps in porous media flow.Proceedings of the National Academy of Sciences (PNAS), (110), 3755–3759.
    [Google Scholar]
  11. Berg, S., Rücker, M., Ott, H., Georgiadis, A., van der Linde, H., Enzmann, F., Kersten, M., Armstrong, R., de With, S., Becker, J. and Wiegmann, A.
    [2016] Connected pathway relative permeability from pore-scale imaging of imbibition.Advances in Water Resources, (90), 24–35.
    [Google Scholar]
  12. Berryman, J. and Blair, S.
    [1986] Use of digital image analysis to estimate fluid permeability of porous materials: Application of two-point correlation functions.Journal of Applied Physics, (60), 1930–1938.
    [Google Scholar]
  13. Bhatnagar, P., Gross, E. and Krook, M.
    [1954] A Model for Collision Processes in Gases. I. Small Amplitude Processes in Charged and Neutral One-Component Systems.Physical Review, (94), 511–525.
    [Google Scholar]
  14. Blunt, M., Bijeljic, B., Dong, H., Gharbi, O., Iglauer, S., Mostaghimi, P., Paluszny, A. and Pentland, C.
    [2013] Pore-scale imaging and modelling.Advances in Water Resources, (51), 197–216.
    [Google Scholar]
  15. Bouzidi, M., Firdaouss, M. and Lallemand, P.
    [2001] Momentum transfer of a Boltzmann-lattice fluid with boundaries.Physics of Fluids, (13), 3452–3459.
    [Google Scholar]
  16. Cabezas, J., Araya-Polo, M., Gelado, I., Navarro, N., Morancho, E. and Cela, J.M.
    [2009] High-Performance Reverse Time Migration on GPU. In: 2009 International Conference of the Chilean Computer Science Society.77–86.
    [Google Scholar]
  17. Calore, E., Gabbana, A., Kraus, J., Pellegrini, E., Schifano, S. and Tripiccione, R.
    [2016] Massively parallel lattice-Boltzmann codes on large GPU clusters.Parallel Computing, 58, 1–24.
    [Google Scholar]
  18. Cancelliere, A., Chang, C., Foti, E., Rothman, D. and Succi, S.
    [1990] The permeability of a random medium: comparison of simulation with theory.Physics of Fluids, (2), 2085–2088.
    [Google Scholar]
  19. Chen, H., Teixeira, C. and Molvig, K.
    [1997] Digital physics approach to computational fluid dynamics: some basic theoretical features.International Journal of Modern Physics, (8), 675–684.
    [Google Scholar]
  20. COMSOL
    [2017] COMSOL Multiphysics Documentation. Version 5.3, Pore-Scale Flow. https://www.comsol.com/model/download/407771/models.ssf.pore_scale_flow.pdf. [Online; Accessed on June 4, 2017].
  21. Cook, S.
    [2013] CUDA Programming: A Developer’s Guide to Parallel Computing with GPUs.Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1st edn.
    [Google Scholar]
  22. d’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand, P. and Luo, L.
    [2002] Multiple-Relaxation-Time lattice Boltzmann models in three dimensions.Philosophical Transactions of the Royal Society, (360), 437–451.
    [Google Scholar]
  23. Dvorkin, J., Armbruster, M., Baldwin, C., Fang, Q., Derzhi, N., Gomez, C., Nur, B., Nur, A. and Mu, Y.
    [2008] The future of rock physics: Computational methods vs. lab testing.First Break, (26), 63–68.
    [Google Scholar]
  24. Dvorkin, J., Derzhi, N., Diaz, E. and Fang, Q.
    [2011] Relevance of computational rock physics.Geophysics, (76), E141–E153.
    [Google Scholar]
  25. Ferrèol, B. and Rothman, D.
    [1995] Lattice-Boltzmann simulations of flow through Fontainebleau sandstone.Transport in Porous Media, (20), 3–20.
    [Google Scholar]
  26. Ginzburg, I. and d’Humières, D.
    [2003] Multireflection boundary conditions for lattice Boltzmann models.Phys. Rev. E, 68, 066614.
    [Google Scholar]
  27. Gray, F. and Boek, E.
    [2016] Enhancing Computational Precision for Lattice Boltzmann Schemes in Porous Media Flows.Computation, (4), 11.
    [Google Scholar]
  28. Grossman, M., Araya-Polo, M., Alpak, F., Frank, F., Limbeck, J. and Sarkar, V.
    [2016] Analysis of Sparse Matrix-vector Multiply for Large Sparse Linear Systems. In: ECMOR XV - 15th European Conference on the Mathematics of Oil Recovery.EAGE, Amsterdam, Netherlands.
    [Google Scholar]
  29. Guo, Z. and Zheng, C.
    [2008] Analysis of lattice Boltzmann equation for microscale gas flows: Relaxation times, boundary conditions and the Knudsen layer.International Journal of Computational Fluid Dynamics, (22), 465–473.
    [Google Scholar]
  30. Guo, Z., Zheng, C. and Shi, B.
    [2002] Discrete lattice effects on the forcing term in the lattice Boltzmann method.Phys. Rev. E, 65, 046308.
    [Google Scholar]
  31. He, X., Zou, Q., Luo, L. and Dembo, M.
    [1997] Analytic solutions of simple flows and analysis of nonslip boundary conditions for the lattice Boltzmann BGK model.Journal of Statistical Physics, (87), 115–136.
    [Google Scholar]
  32. Karp, A.H. and Flatt, H.P.
    [1990] Measuring Parallel Processor Performance.Commun. ACM, 33(5), 539–543.
    [Google Scholar]
  33. Keehm, Y.
    [2003] Computational rock physics: Transport properties in porous media and applications. Ph.D. Dissertation, Stanford University, Stanford, California, U.S.A.
    [Google Scholar]
  34. Kendon, V., Cates, M., Desplat, J.C., Pagonabarraga, I. and Bladon, P.
    [2000] Inertial effects in three-dimensional spinodal decomposition of a symmetric binary fluid mixture: a lattice Boltzmann study.Journal of Fluid Mechanics, (440), 147–203.
    [Google Scholar]
  35. Knackstedt, M. A.P. S. and Sahimi, M.
    [2001] Pore network modeling of two-phase flow in porous rock: The effect of correlated heterogeneity.Advances in Water Resources, (24), 257–277.
    [Google Scholar]
  36. Knackstedt, M., Arns, C., Madadi, M., Sheppard, A., Latham, S., Sok, R., Bächle, G. and Eberli, G.
    [2008] Elastic and flow properties of carbonate core derived from 3D Xray-CT images.SEG Technical Program Expanded Abstracts, (27), 1804–1809.
    [Google Scholar]
  37. Knackstedt, M., Latham, S., Madadi, M., Sheppard, A., Varslot, T. and Arns, C.
    [2009] Digital rock physics: 3D imaging of core material and correlations to acoustic and flow properties.The Leading Edge, (28), 28–33.
    [Google Scholar]
  38. Kuzmin, A., Guo, Z. and Mohamad, A.
    [2011] Simultaneous incorporation of mass and force terms in the multi-relaxation-time framework for lattice Boltzmann schemes.Philosophical Transactions of the Royal Society A Mathematical, Physical and Engineering Sciences, (369), 2219–2227.
    [Google Scholar]
  39. Ladd, A.
    [1994a] Numerical simulations of particulate suspensions via a discretized Boltzmann equation: Part 1: theoretical foundation.Journal of Fluid Mechanics, (271), 285–309.
    [Google Scholar]
  40. [1994b] Numerical simulations of particulate suspensions via a discretized Boltzmann equation: Part 2. Numerical results.Journal of Fluid Mechanics, (271), 311–339.
    [Google Scholar]
  41. Lallemand, P. and Luo, L.
    [2000] Theory of the lattice Boltzmann method: dispersion, dissipation, isotropy, Galilean invariance, and stability.Physical Review, (61), 6546–6562.
    [Google Scholar]
  42. Leu, L., Berg, S., Enzmann, F., Armstrong, R. and Kersten, M.
    [2014] Fast X-ray Micro-Tomography of Multiphase Flow in Berea Sandstone: A Sensitivity Study on Image Processing.Transport in Porous Media, (105), 451–469.
    [Google Scholar]
  43. Llwellin, E.
    [2010a] LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part I: theory and implementation.Computers & Geosciences, (36), 115–122.
    [Google Scholar]
  44. [2010b] LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part II: usage and validation.Computers & Geosciences, (36), 123–132.
    [Google Scholar]
  45. Martys, N. and Chen, H.
    [1996] Simulation of multicomponent fluids in complex three dimensional geometries by the lattice Boltzmann method.Physical Review, (53), 743–750.
    [Google Scholar]
  46. Meakin, P. and Tartakovsky, A.
    [2009] Modeling and simulation of pore-scale multiphase fluid flow and reactive transport in fractured and porous media.Reviews of Geophysics, (RG3002), 1–47.
    [Google Scholar]
  47. Micikevicius, P.
    [2012] GPU performance analysis and optimization. In: NVIDIA’s GPU Technology Conference.
    [Google Scholar]
  48. NVIDIA
    [2018] NVIDIA Visual Profiler. https://developer.nvidia.com/nvidia-visual-profiler. [Online; Accessed on January 24, 2018].
  49. Oeren, P., Bakke, S. and Rueslatten, H.
    [2006] Digital core laboratory: Rock and flow properties derived from computer generated rocks.International Symposium of the Society of Core Analysts held in Trondheim, Norway 12–16 September, 1–12.
    [Google Scholar]
  50. Otomo, H., Fan, H., Li, Y., Dressler, M., Staroselsky, I., Zhang, R. and Chen, H.
    [2016] Studies of accurate multi-component lattice Boltzmann models on benchmark cases required for engineering applications.Journal of Computational Science, (17), 334–339.
    [Google Scholar]
  51. Owens, J.D., Houston, M., Luebke, D., Green, S., Stone, J.E. and Phillips, J.C.
    [2008] GPU Computing.Proceedings of the IEEE, 96(5), 879–899.
    [Google Scholar]
  52. Palabos
    [2017] Palabos Documentation. http://www.palabos.org/documentation/userguide. [Online; Accessed on June 4, 2017].
  53. Pan, C., Luo, L.S. and Miller, C.
    [2006] An evaluation of lattice Boltzmann schemes for porous medium flow simulation.Computational Fluids, (35), 898–909.
    [Google Scholar]
  54. Paul, S.R., Araya-Polo, M., Mellor-Crummey, J. and Hohl, D.
    [2016] Performance Analysis and Optimization of a Hybrid Seismic Imaging Application.Procedia Comput. Sci., 80(C), 8–18.
    [Google Scholar]
  55. Piller, M., Casagrande, D., Schena, G. and Santini, M.
    [2014] Pore-scale simulation of laminar flow through porous media.Journal of Physics: Conference Series, 501(1), 012010.
    [Google Scholar]
  56. Premnath, K. and Abraham, J.
    [2007] Three-dimensional multi-relaxation-time (MRT) lattice-Boltzmann models for multiphase flow.Journal of Computational Physics, (224), 539–559.
    [Google Scholar]
  57. Premnath, K., Pattison, M. and Banerjee, S.
    [2013] An Investigation of the Lattice Boltzmann Method for Large Eddy Simulation of Complex Turbulent Separated Flow.Journal of Fluids Engineering, (135), 051401–051401–12.
    [Google Scholar]
  58. Robertsén, F., Westerholm, J. and Mattila, K.
    [2015] Lattice Boltzmann Simulations at Petascale on Multi-GPU Systems with Asynchronous Data Transfer and Strictly Enforced Memory Read Alignment. In: 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing.604–609.
    [Google Scholar]
  59. Saenger, E., Enzmann, F., Keehm, Y. and Steeb, H.
    [2011] Digital rock physics: Effect of fluid viscosity on effective elastic properties.Journal of Applied Geophysics, (74), 236–241.
    [Google Scholar]
  60. Saxena, N., Hofmann, R., Alpak, F., Dietderich, J., Hunter, S. and Day-Stirrat, R.
    [2017a] Effect of image segmentation & voxel size on micro-CT computed effective transport & elastic properties.Marine and Petroleum Geology, (86), 972–990.
    [Google Scholar]
  61. Saxena, N., Hofmann, R., Alpak, F., Berg, S., Dietderich, J., Agarwal, U., Tandon, K., Hunter, S., Freeman, J. and Wilson, O.
    [2017b] References and benchmarks for pore-scale flow simulated using micro-CT images of porous media and digital rocks.Advances in Water Resources, (109), 211–235.
    [Google Scholar]
  62. Saxena, N. and Mavko, G.
    [2016] Estimating elastic moduli of rocks from thin sections: Digital rock study of 3D properties from 2D images.Computers and Geosciences, (88), 9–21.
    [Google Scholar]
  63. Saxena, N., Mavko, G., Hofmann, R. and Srisutthiyakorn, N.
    [2017c] Estimating permeability from thin sections without reconstruction: Digital rock study of 3D properties from 2D images.Computers and Geosciences, (102), 79–99.
    [Google Scholar]
  64. Saxena, N., R., H., A., H., Alpak, F.
    and S., H. [2017d] Imaging and computational considerations for digital rocks to resolve pore throats and achieve representative flow volume. Advances in Water Resources, in review.
    [Google Scholar]
  65. Shah, S., Gray, F., Crawshaw, J. and Boek, E.
    [2016] Micro-Computed Tomography Pore-scale Study of Flow in Porous Media: Effect of Voxel Resolution.Advances in Water Resources, (95), 276–287.
    [Google Scholar]
  66. Skordos, P.
    [1993] Initial and boundary conditions for the lattice Boltzmann method.Physical Review, (48), 4823–4842.
    [Google Scholar]
  67. Succi, S.
    [2003] The Lattice Boltzmann Equation - For Fluid Dynamics and Beyond. 22, 368.
    [Google Scholar]
  68. Sukop, M.C. and Thorne, D.T.
    [2010] Lattice Boltzmann Modeling: An Introduction for Geoscientists and Engineers.Springer Publishing Company, Incorporated, 1st edn.
    [Google Scholar]
  69. Toelke, J., Baldwin, C., Mu, Y., Derzhi, N., Fang, Q., Grader, A. and Dvorkin, J.
    [2010] Computer simulations of fluid flow in sediment: From images to permeability.The Leading Edge, (29), 68–74.
    [Google Scholar]
  70. Wiegmann, A.
    [2007] Computation of the Permeability of Porous Materials from Their Microstructure by FFF-Stokes.Berichte des Fraunhofer ITWM, (29).
    [Google Scholar]
  71. Youngseuk, K., Tapan, M. and Amos, N.
    [2004] Permeability prediction from thin sections: 3D reconstruction and Lattice-Boltzmann flow simulation.Geophysical Research Letters, 31(4).
    [Google Scholar]
http://instance.metastore.ingenta.com/content/papers/10.3997/2214-4609.201802184
Loading
/content/papers/10.3997/2214-4609.201802184
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error