Deep Reinforcement Learning Algorithm for Wellbore Cleaning Across Drilling Operation

We propose a novel framework for real-time drilling operation planning updates using deep reinforcement learning algorithm, enabling drilling process reactions to be automated. The framework includes a decision tree algorithm to represent the environment dynamic changes based on the imposed actions parallel to a Gaussian process algorithm to quantify the safe operating window in real-time. Combining these two algorithms leads to mounting a Markov Decision Process (MDP) environment for a decision-making system.

We demonstrate the effectiveness of our framework by implementing an off-policy deep reinforcement learning algorithm, using a deep Q-learning network to create experiences, and employing synchronous updates on the agent. Given the essence of reinforcement learning, the framework can be efficiently implemented for on-the-spot decision-making, allowing the driller to receive an effective sequence of actions considering company policies.

Our algorithm achieves state-of-the-art performance on weight-to-slip hole conditioning operation, a wellbore cleaning operation after drilling a stand before connecting to the next pipe. The performance evaluation exhibits its efficiency in real-time operation overhaul, eliminating non-value-added activities. Our framework thus opens the door for automating the process based on the operating parameters obtained in real-time.