Evaluating Retrieval-Augmented Generation for Subsurface Applications: Lessons learned

This abstract discusses the implementation and evaluation of Retrieval-Augmented Generation (RAG) systems in the subsurface domain, where professionals face challenges with large volumes of unstructured data and the need for effective knowledge transfer. The RAG tool, developed by a cross-functional team, leverages text-based experiences from subsurface professionals, enriched with metadata, to enable efficient information retrieval and support faster decision-making.

Three key lessons emerged from the evaluation phase:

1. Metrics Alone Are Insufficient: Quantitative metrics like retrieval recall and faithfulness provide valuable diagnostics but do not fully capture system performance. Qualitative user feedback is essential to interpret results and guide improvements.
2. User Feedback Is Critical: The system incorporates a feedback loop where users evaluate queries, retrieved documents, and generated answers. This feedback refines both the content and structure of responses, ensuring the tool meets real-world needs.
3. Curated Evaluation Datasets Are Crucial: Domain-specific datasets, informed by user interactions, are necessary for robust evaluation. Building such datasets is challenging but vital for meaningful assessment and continuous improvement.

The abstract concludes that RAG systems, when combined with expert feedback and tailored evaluation data, can enhance knowledge retrieval and decision-making in safety-critical subsurface applications.