1887

Abstract

Summary

In this paper, we presented our efforts towards advancing the geological text understanding capability of language models through domain-adapted training. By leveraging our in-house geological text corpus and innovative training strategies, we developed two geological language models. The geological BERT model significantly enhanced its ability to capture the semantic characteristics of words when used in geological contexts, leading to improved performance on the critical NER task. The geological GTE model, adapted with a topic classification task, showed promise in categorizing geological content despite limited domain-specific training data. These efforts highlight the importance of domain adaptation in achieving state-of-the-art performance in specialized fields like geoscience.

© EAGE Publications BV
Loading

Article metrics loading...

/content/papers/10.3997/2214-4609.202539036
2025-03-24
2026-02-06

  • From This Site

    /content/papers/10.3997/2214-4609.202539036
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. [2020] Language Models are Few-Shot Learners. 34th Conference on Neural Information Processing Systems, Expanded Abstracts.
     [Google Scholar]
  2. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. [2019] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics, 4171–4186.
     [Google Scholar]
  3. Grangier, D., Katharopoulos, A., Ablin, P. and Hannun, A. Y. [2024] Need a Small Specialized Language Model? Plan Early!. arXiv preprint arXiv:2402.01093.
     [Google Scholar]
  4. Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P. and Zhang, M. [2023] Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv preprint arXiv:2308.03281.
     [Google Scholar]
  5. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. [2019] RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv: 1907.11692.
     [Google Scholar]
  6. Lun, C. H., Hewitt, T. and Hou, S. [2022] A Machine Learning Pipeline for Document Extraction. First Break, 40(2), 73–78.
     [Google Scholar]
/content/papers/10.3997/2214-4609.202539036
Loading
Advancing Geological Text Understanding with Domain-Specific Language Modelling
Conference Proceedings 2025, 1 (2025); https://doi.org/10.3997/2214-4609.202539036
/content/papers/10.3997/2214-4609.202539036
/content/papers/10.3997/2214-4609.202539036
Loading

Data & Media loading...

Most Cited This Month Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error