Application of Machine Learning to GPU Optimization, Deep Q-Networks and Computational Fluid Dynamics

Throughout society today, machine learning has been catapulted to a transformative problem solving approach across various domains, ranging from natural language processing to computer vision to engineering optimization. The fundamental principle is the ability of algorithms to learn patterns and make decisions based on data, rather than relying on explicitly programmed instructions. This dissertation addresses the research question: “How can machine learning techniques be applied to improve computational efficiency and prediction accuracy in high-performance scientific computing tasks, including GPU kernel optimization, Deep Q-Networks, and computational fluid dynamics?” To answer the question, we devised three distinct problems, each of which is orthogonal to the next to represent a wide breadth of exploration. The problems focus on the two paradigms of supervised learning and reinforcement learning. In the first problem, we propose a framework that leverages GPU hardware performance counters and a suite of machine learning models, including a deep neural network, a random forest, and a naive Bayes classifier, to automatically characterize and classify GPU kernels using supervised learning. This framework identifies performance bottlenecks and recommends targeted optimization strategies, thereby reducing the reliance on manual tuning and improving resource utilization. Experimental evaluations on stencil computations and sparse matrix operations demonstrate the effectiveness of our approach in enhancing GPU performance. The second problem introduces an accelerated Deep Q-Network (DQN) algorithm that reformulates the experience replay mechanism in reinforcement learning. By storing (state, action, target) tuples instead of (state, action, reward) tuples, our method eliminates redundant recomputation of target values and minimizes the use of computationally expensive max() operations. This refinement results in faster convergence, as evidenced by a reduction of up to 35% in training time and a decrease in the ...