1887
  1. Home
  2. A-Z Publications
  3. Near Surface Geophysics
  4. Volume 22, Issue 6
  5. Article
Volume 22, Issue 6
  • ISSN: 1569-4445
  • E-ISSN: 1873-0604
PDF

Abstract

Abstract

Geophysical and remote sensing products that rely on Poisson‐distributed measurement signals, such as cosmic‐ray neutron sensing (CRNS) and gamma spectrometry, often face challenges due to inherent Poisson noise. Common techniques to enhance signal stability include data aggregation or smoothing (e.g., moving averages and interpolation). However, these methods typically reduce the ability to resolve detailed temporal (stationary data) and spatial (mobile data) features. In this study, we introduced a method for contextual noise suppression tailored to Poisson‐distributed data, utilizing a discrete score attribution system. This evaluates each observation against eight different criteria to assess its consistency with surrounding values, assigning a score between 0 (very unlikely) and 8 (very likely) to indicate whether the observation is likely to act as noise. These scores can then be used to flag or remove data points based on user‐defined thresholds. We tested the effectiveness on both stationary and mobile CRNS data, as well as on gamma‐ray spectrometry and electromagnetic induction (EMI) recordings. In our examples, the consistently outperformed established filters, for example Savitzky–Golay and Kalman, in direct competition when applied to CRNS time series data. Additionally, the substantially reduced Poisson noise in mobile CRNS, gamma‐ray spectrometry and EMI data. The scoring system also provides a context‐sensitive evaluation of individual observations or aggregates, assessing their conformity within the dataset. Given its general applicability, customizable criteria and very low computational demands, the proposed filter is easy to implement and holds promise as a valuable tool for denoising geophysical data and applications in other fields.

© 2024 European Association of Geoscientists & Engineers. © 2024 The Author(s). Near Surface Geophysics published by John Wiley & Sons Ltd on behalf of European Association of Geoscientists and Engineers.
Loading

Article metrics loading...

/content/journals/10.1002/nsg.12324
2024-11-18
2024-12-06

  • From This Site

    /content/journals/10.1002/nsg.12324
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

/deliver/fulltext/nsg/22/6/nsg12324.html?itemId=/content/journals/10.1002/nsg.12324&mimeType=html&fmt=ahah

References

  1. Altdorff, D., Bechtold, M., van der Kruk, J., Vereecken, H. & Huisman, J.A. (2016) Mapping peat layer properties with multi‐coil offset electromagnetic induction and laser scanning elevation data. Geoderma, 261, (Supplement C), 178–189. Available from: https://doi.org/10.1016/j.geoderma.2015.07.015
     [Google Scholar]
  2. Altdorff, D., Galagedara, L., Nadeem, M., Cheema, M. & Unc, A. (2018) Effect of agronomic treatments on the accuracy of soil moisture mapping by electromagnetic induction. Catena, 164, 96–106. Available from: https://doi.org/10.1016/j.catena.2017.12.036
     [Google Scholar]
  3. Altdorff, D., Oswald, S.E., Zacharias, S., Zengerle, C., Dietrich, P., Mollenhauer, H. et al. (2023) Toward large‐scale soil moisture monitoring using rail‐based cosmic ray neutron sensing. Water Resources Research, 59(3), e2022WR033514. Available from: https://doi.org/10.1029/2022wr033514
     [Google Scholar]
  4. Avery, W.A., Finkenbiner, C., Franz, T.E., Wang, T.J., Nguy‐Robertson, A.L., Suyker, A. et al. (2016) Incorporation of globally available datasets into the roving cosmic‐ray neutron probe method for estimating field‐scale soil water content. Hydrology and Earth System Sciences, 20(9), 3859–72. Available from: https://doi.org/10.5194/hess‐20‐3859‐2016
     [Google Scholar]
  5. Blöschl, G., Blaschke, A.P., Broer, M., Bucher, C., Carr, G., Chen, X. et al. (2016) The Hydrological Open Air Laboratory (HOAL) in Petzenkirchen: a hypothesis‐driven observatory. Hydrology and Earth System Sciences, 20(1), 227–55. Available from: https://doi.org/10.5194/hess‐20‐227‐2016
     [Google Scholar]
  6. Bogena, H.R., Schrön, M., Jakobi, J., Ney, P., Zacharias, S., Andreasen, M. et al. (2022) COSMOS‐Europe: a European network of cosmic‐ray neutron soil moisture sensors. Earth System Science Data, 14(3), 1125–1151. Available from: https://doi.org/10.5194/essd‐14‐1125‐2022
     [Google Scholar]
  7. Box, G.E.P., Jenkins, G.M. & Reinsel, G.C. (1994) Time series analysis: forecasting and control, 3rd Edition. Saddle River, NJ: Prentice Hall.
     [Google Scholar]
  8. Brogi, C., Pisinaras, V., Koehli, M., Dombrowski, O., Franssen, H.J.H., Babakos, K. et al. (2023) Monitoring irrigation in small orchards with cosmic‐ray neutron sensors. Sensors, 23(5), 2378. Available from: https://doi.org/10.3390/s23052378
     [Google Scholar]
  9. Chrisman, B. & Zreda, M. (2013) Quantifying mesoscale soil moisture with the cosmic‐ray rover. Hydrology and Earth System Sciences, 17(12), 5097–5108. Available from: https://doi.org/10.5194/hess‐17‐5097‐2013
     [Google Scholar]
  10. Dalan, R.A. (2008) A review of the role of magnetic susceptibility in archaeogeophysical studies in the USA: recent developments and prospects. Archaeological Prospection, 15, 1–31. Available from: https://doi.org/10.1002/arp.323
     [Google Scholar]
  11. Davies, P., Baatz, R., Bogena, H.R., Quansah, E. & Amekudzi, L.K. (2022) Optimal temporal filtering of the cosmic‐ray neutron signal to reduce soil moisture uncertainty. Sensors, 22(23), 9143. Available from: https://doi.org/10.3390/s22239143
     [Google Scholar]
  12. De Mello, D.C., Veloso, G.V., Moquedace, C.M., de Angeli Oliveira, I., De Oliveira, F.S., Gomes, L.C. et al. (2023) Radiometric and magnetic susceptibility characterization of soil profiles: geophysical data and their relationship with Antarctic periglacial processes, pedogenesis, and lithology. Catena, 232, 107427. Available from: https://doi.org/10.1016/j.catena.2023.107427
     [Google Scholar]
  13. Dega, S., Dietrich, P., Schroen, M. & Paasche, H. (2023) Probabilistic prediction by means of the propagation of response variable uncertainty through a Monte Carlo approach in regression random forest: application to soil moisture. Frontiers in Environmental Science, 11, 1009191. Available from: https://doi.org/10.3389/fenvs.2023.1009191
     [Google Scholar]
  14. Doolittle, J.A. & Brevik, E.C. (2014) The use of electromagnetic induction techniques in soils studies. Geoderma, 223–225, 33–45. Available from: https://doi.org/10.1016/j.geoderma.2014.01.027
     [Google Scholar]
  15. Fersch, B., Jagdhuber, T., Schrön, M., Volksch, I. & Jäger, M. (2018) Synergies for soil moisture retrieval across scales from airborne polarimetric sar, cosmic ray neutron roving, and an in situ sensor network. Water Resources Research, 54(11), 9364–83. Available from: https://doi.org/10.1029/2018wr023337
     [Google Scholar]
  16. Franz, T.E., Zreda, M., Rosolem, R. & Ferre, T.P.A. (2012) Field validation of a cosmic‐ray neutron sensor using a distributed sensor network. Vadose Zone Journal, 11(4), 1–16. Available from: https://doi.org/10.2136/vzj2012.0046
     [Google Scholar]
  17. Franz, T.E., Wang, T.J., Avery, W., Finkenbiner, C. & Brocca, L. (2015) Combined analysis of soil moisture measurements from roving and fixed cosmic ray neutron probes for multiscale real‐time monitoring. Geophysical Research Letters, 42(9), 3389–96. Available from: https://doi.org/10.1002/2015gl063963
     [Google Scholar]
  18. Franz, T.E., Wahbi, A., Zhang, J., Vreugdenhil, M., Heng, L., Dercon, G. et al. (2020) Practical data products from cosmic‐ray neutron sensing for hydrological applications. Frontiers in Water, 2(9), 1–13. Available from: https://doi.org/10.3389/frwa.2020.00009
     [Google Scholar]
  19. Gianessi, S., Polo, M., Stevanato, L., Lunardon, M., Francke, T., Oswald, S.E. et al. (2022) Testing a novel sensor design to jointly measure cosmic‐ray neutrons, muons and gamma rays for non‐invasive soil moisture estimation. Geoscientific Instrumentation, Methods and Data Systems, 13(1), 9–25. Available from: https://doi.org/10.5194/gi‐13‐9‐2024
     [Google Scholar]
  20. Heistermann, M., Bogena, H., Francke, T., Güntner, A., Jakobi, J., Rasche, D. et al. (2022) Soil moisture observation in a forested headwater catchment: combining a dense cosmic‐ray neutron sensor network with roving and hydrogravimetry at the Tereno Site Wustebach. Earth System Science Data, 14(5), 2501–19. Available from: https://doi.org/10.5194/essd‐14‐2501‐2022
     [Google Scholar]
  21. Heistermann, M., Francke, T., Schrön, M., & Oswald, S.E. (2024) Revisiting the general calibration of cosmic‐ray neutron sensors to estimate soil water content. Hydrology and Earth System Sciences, 28(4), 989–1000. Available from: https://doi.org/10.5194/hess‐28‐989‐2024
     [Google Scholar]
  22. IAEA (International Atomic Energy Agency) . (2003). Guidelines for radioelement mapping using gamma ray spectrometry data. Vienna, IAEA.
  23. Iwema, J., Schrön, M., Silva, J.K.D., Lopes, R.S.D.P., & Rosolem, R. (2021) Accuracy and precision of the cosmic-ray neutron sensor for soil moisture estimation at humid environments. Hydrological Processes, 35, https://doi.org/10.1002/hyp.14419
     [Google Scholar]
  24. Jakobi, J., Huisman, J.A., Schrön, M., Fiedler, J., Brogi, C., Vereecken, H. et al. (2020) Error estimation for soil moisture measurements with cosmic ray neutron sensing and implications for rover surveys. Frontiers in Water, 2, 1–14. Available from: https://doi.org/10.3389/frwa.2020.00010
     [Google Scholar]
  25. Jakobi, J., Huisman, J.A., Köhli, M., Rasche, D., Vereecken, H. & Bogena, H.R. (2021) The footprint characteristics of cosmic ray thermal neutrons. Geophysical Research Letter, 48(15), e2021GL094281. Available from: https://doi.org/10.1029/2021GL094281
     [Google Scholar]
  26. Kalka, A.J. & Turek, A.M. (2023) Searching for alternatives to the Savitzky–Golay filter in the spectral processing domain. Applied Spectroscopy, 77(4), 426–32. Available from: https://doi.org/10.1177/00037028231154278
     [Google Scholar]
  27. Kalman, R.E. (1960) A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82, 35–45. Available from: https://doi.org/10.1115/1.3662552
     [Google Scholar]
  28. Keller, G.V. & Frischknecht, F.C. (1966) Electrical methods of geophysical prospecting (Inter‐national series of monographs in electromagnetic waves). Oxford, New York: Pergamon Press. Available from: https://archive.org/details/electricalmethod00kell/page/n7/mode/2up
     [Google Scholar]
  29. Kirby, G.E. (1988) Soils of the Pasadena‐Deer Lake Area, Newfoundland. Newfoundland soil survey, land resource research centre contribution, 17(85‐5). Ottawa: Minister of Supply and Services Canada. Cat. No. A57‐181/1988E, ISBS: 0‐662‐16377‐X. Available from: http://sis.agr.gc.ca/cansis/publications/surveys/nf/nf17/nf17_report.pdf
  30. Knoll, G.F. (2010) Radiation detection and measurement. Hoboken, NJ: John Wiley and Sons Inc. Available from: https://www.wiley.com/en‐us/Radiation+Detection+and+Measurement%2C+4th+Edition‐p‐9780470131480
     [Google Scholar]
  31. Köhli, M., Schrön, M., Zreda, M., Schmidt, U., Dietrich, P. & Zacharias, S. (2015) Footprint characteristics revised for field‐scale soil moisture monitoring with cosmic‐ray neutrons. Water Resources Research, 51(7), 5772–5790. Available from: https://doi.org/10.1002/2015wr017169
     [Google Scholar]
  32. Köhli, M., Weimar, J., Schrön, M., Baatz, R. & Schmidt, U. (2021) Soil moisture and air humidity dependence of the above‐ground cosmic‐ray neutron intensity. Frontiers in Water, 2(66), Available from: https://doi.org/10.3389/frwa.2020.544847
     [Google Scholar]
  33. Li, C. & Mao, Z. (2023) A label noise filtering method for regression based on adaptive threshold and noise score. Expert Systems with Applications, 228, 120422. Available from: https://doi.org/10.1016/j.eswa.2023.120422
     [Google Scholar]
  34. McNeil, J.D. (1980) Electromagnetic terrain conductivity measurement at low induction numbers. Mississauga, Canada: Geonics. Ltd., Technical Note TN‐6. Available from: https://geonics.com/pdfs/technicalnotes/tn6.pdf
     [Google Scholar]
  35. McJannet, D., Hawdon, A., Baker, B., Renzullo, L. & Searle, R. (2017) Multiscale soil moisture estimates using static and roving cosmic‐ray soil moisture sensors. Hydrology and Earth System Sciences, 21(12), 6049–67. Available from: https://doi.org/10.5194/hess‐21‐6049‐2017
     [Google Scholar]
  36. Morris, E.R. (2009) Height‐above‐ground effects on penetration depth and response of electromagnetic induction soil conductivity meters. Computers and Electronics in Agriculture, 68(2), 150–156. Available from: https://doi.org/10.1016/j.compag.2009.05.009
     [Google Scholar]
  37. Okyay, S. & Adar, N. (2022) Dementia‐related user‐based collaborative filtering for imputing missing data and generating a reliability scale on clinical test scores. PeerJ, 10, 13425. Available from: https://doi.org/10.7717/peerj.13425
     [Google Scholar]
  38. Othman, A., Naveed, I., Sherif, M.H. & Umair, B.W. (2021) Automated event detection and denoising method for passive seismic data using residual deep convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–11. Available from: https://doi.org/10.1109/TGRS.2021.3054071
     [Google Scholar]
  39. Pracilio, G., Adams, M.L., Smettem, K.R.J. & Harper, R.J. (2006) Determination of spatial distribution patterns of clay and plant available potassium contents in surface soils at the farm scale using high resolution gamma ray spectrometry. Plant and Soil, 282, 67–82. Available from: https://doi.org/10.1007/s11104‐005‐5229‐1
     [Google Scholar]
  40. Power, D., Rico‐Ramirez, M.A., Desilets, S., Desilets, D. & Rosolem, R. (2021) Cosmic‐ray neutron sensor PYthon tool (crspy 1.2.1): an open‐source tool for the processing of cosmic‐ray neutron and soil moisture data. Geoscientific Model Development, 14(12), 7287–7307. Available from: https://doi.org/10.5194/gmd‐14‐7287‐2021
     [Google Scholar]
  41. Rao, C.R. (1947) Factorial experiments derivable from combinatorial arrangements of arrays. Supplement to the Journal of the Royal Statistical Society. Oxford University Press, 9(1), 128–139. p12. Available from: https://doi.org/10.2307/2983576
     [Google Scholar]
  42. Rentschler, T., Werban, U., Ahner, M., Behrens, T., Gries, P., Scholten, T. et al. (2020) 3D mapping of soil organic carbon content and soil moisture with multiple geophysical sensors and machine learning. Vadose Zone Journal, 19(1), 1–15. Available from: https://doi.org/10.1002/vzj2.20062
     [Google Scholar]
  43. Savitzky, A. & Golay, M.J.E. (1964) Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry, 36(8), 1627–1639. Available from: https://doi.org/10.1021/ac60214a047
     [Google Scholar]
  44. Sadatcharam, K., Altdorff, D., Unc, A. & Galagedara, L. (2020) Depth sensitivity of apparent magnetic susceptibility measurements using multi‐coil and multi‐frequency electromagnetic induction. Journal of Environmental and Engineering Geophysical, 25(3), 301–314. Available from: https://doi.org/10.32389/jeeg20‐001
     [Google Scholar]
  45. Shirzaditabar, F. & Heck, R.J. (2022) Characterization of soil magnetic susceptibility: a review of fundamental concepts, instrumentation, and applications. Canadian Journal of Soil Science, 102(2), 231–251. Available from: https://doi.org/10.1139/cjss‐2021‐0040
     [Google Scholar]
  46. Schmid, M., Rath, D. & Diebold, U. (2022) Why and how Savitzky‐Golay filters should be replaced. ACS Measurement Science Au, 2(2), 185–96. Available from: https://doi.org/10.1021/acsmeasuresciau.1c00054
     [Google Scholar]
  47. Schreiner‐McGraw, A.P., Vivoni, E.R., Mascaro, G. & Franz, T.E. (2016) Closing the water balance with cosmic‐ray soil moisture measurements and assessing their relation to evapotranspiration in two semiarid watersheds. Hydrology and Earth System Sciences, 20(1), 329–45. Available from: https://doi.org/10.5194/hess‐20‐329‐2016
     [Google Scholar]
  48. Schrön, M., Köhli, M., Scheiffele, L., Iwema, J., Bogena, H.R., Lv, L. et al. (2017) Improving calibration and validation of cosmic‐ray neutron sensors in the light of spatial sensitivity. Hydrology and Earth System Sciences, 21(10), 5009–5030. Available from: https://doi.org/10.5194/hess‐21‐5009‐2017
     [Google Scholar]
  49. Schrön, M., Rosolem, R., Köhli, M., Piussi, L., Schröter, I., Iwema, J. et al. (2018a) Cosmic‐ray neutron rover surveys of field soil moisture and the influence of roads. Water Resources Research, 54(9), 6441–59. Available from: https://doi.org/10.1029/2017wr021719
     [Google Scholar]
  50. Schrön, M., Zacharias, S., Womack, G., Köhli, M., Desilets, D., Oswald, S.E. et al. (2018b) Intercomparison of cosmic‐ray neutron sensors and water balance monitoring in an urban environment. Geoscientific Instrumentation Methods and Data Systems, 7(1), 83–99. Available from: https://doi.org/10.5194/gi‐7‐83‐2018
     [Google Scholar]
  51. Schrön, M., Oswald, S.E., Zacharias, S., Kasner, M., Dietrich, P. & Attinger, S. (2021) Neutrons on rails: transregional monitoring of soil moisture and snow water equivalent. Geophysical Research Letters, 48(24), e2021GL093924. Available from: https://doi.org/10.1029/2021gl093924
     [Google Scholar]
  52. Schrön, M., Köhli, M. & Zacharias, S. (2023) Signal contribution of distant areas to cosmic‐ray neutron sensors—implications for footprint and sensitivity. Hydrology and Earth System Sciences, 27(3), 723–738. Available from: https://https://doi.org/10.5194/hess‐27‐723‐2023
     [Google Scholar]
  53. Song, H. & Hu, S. (2019) Open problems in applications of the Kalman filtering algorithm. In Proceedings of the 2019 International Conference on Mathematics, Big Data Analysis and Simulation and Modelling (MBDASM 2019), Changsha, China: Advances in Computer Science Research, 92. Atlantis Press: Amsterdam, The Netherlands, pp. 185–190. Available from: https://doi.org/10.2991/mbdasm‐19.2019.43
  54. Taylor, M.J. & Smettem, K. (2002) Relationships between soil properties and high‐resolution radiometrics, central eastern Wheatbelt, Western Australia. Exploration Geophysics, 33, 95–102. Available from: https://doi.org/10.1071/EG02095
     [Google Scholar]
  55. Thiesson, J., Kessouri, P., Schamper, C. & Tabbagh, A. (2014) Calibration of frequency‐domain electromagnetic devices used in near‐surface surveying. Near‐Surface Geophysics, 12(4), 481–491. Available from: https://doi.org/10.3997/1873‐0604.2014012
     [Google Scholar]
  56. Tsagris, M., Alenazi, A. & Fafalios, S. (2020) Computationally efficient univariate filtering for massive data. Electronic Journal of Applied Statistical Analysis, 13(2), 390–412. Available from: https://doi.org/10.48550/arXiv.2002.04691
     [Google Scholar]
  57. Weimar, J., Köhli, M., Budach, C. & Schmidt, U. (2020) Large‐scale boron‐lined neutron detection systems as a 3He alternative for cosmic ray neutron sensing. Frontiers in Water, 2(16), 1–12. Available from: https://doi.org/10.3389/frwa.2020.00016
     [Google Scholar]
  58. Yin, X.F., Chen, T.Y., Liu, W.R., Xiao, R., Ma, C.X. & Fu, Z.W. (2016) A reinforced collaborative filtering approach based on similarity propagation and score predication graph. Eurasip Journal on Wireless Communications and Networking, 210, 1–20. Available from: https://doi.org/10.1186/s13638‐016‐0710‐5
     [Google Scholar]
  59. Zhang, H., Wang, K., Zhou, J., Chen, J., Xu, Y., Wang, D. et al. (2019) Varifast: a variant filter by automated scoring based on tagged‐signatures. BMC Bioinformatics [Electronic Resource], 20(1), 713. Available from: https://doi.org/10.1186/s12859‐019‐3226‐2
     [Google Scholar]
  60. Zreda, M., Shuttleworth, W.J., Zeng, X., Zweck, C., Desilets, D., Franz, T. et al. (2012) COSMOS: the cosmic‐ray soil moisture observing system. Hydrology and Earth System Sciences, 16(11), 4079–4099. Available from: https://doi.org/10.5194/hess‐16‐4079‐2012
     [Google Scholar]
/content/journals/10.1002/nsg.12324
Loading
Score filtering for contextualized noise suppression of Poisson‐distributed geophysical signals
Near Surface Geophysics 22, 599 (2024); https://doi.org/10.1002/nsg.12324
/content/journals/10.1002/nsg.12324
/content/journals/10.1002/nsg.12324
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): CRNS; data smoothing; EMI; gamma spectrometry; noise suppression

Most Cited This Month Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error