Unlocking Data Insights: A Data Lake Approach with Historical Drilling Data, Apache Spark and BI Tools

This study explores the integration of data Lakehouse platforms, Apache Spark, and BI(Business Intelligence) tools to enhance the efficiency of historical drilling data analysis for well planning. Advanced downhole equipment generates terabytes of data at high frequencies, necessitating robust storage and processing solutions. By leveraging a corporate data lake and cloud-based PySpark, the study organizes and interconnects hundreds of well datasets, integrating them into a dedicated database for seamless visualization via BI tools.

Key analyses included Dogleg Severity (DLS) computations, machine learning (ML)-based bit performance evaluation using Logging While Drilling (LWD) data. Challenges such as data preprocessing, outlier removal, and code validation were addressed through iterative development and condition enhancements. The study highlights the automation of offset wells analysis, significantly reducing time and effort compared to manual approaches.

Novel contributions enabling Drilling Engineers to adopt roles of Data Engineers and Scientists. The findings underscore the potential of ML in automating analytical workflows and extracting actionable insights from extensive datasets, driving efficiency and innovation in drilling operations.