With the increasing military application of unmanned combat aircraft (UCAV), unmanned combat will become the main combat mode in the future air battlefield. In closerange air combat, the environment is complex and the combat situation changes rapidly. The method based on game theory cannot meet the realtime requirements due to the large amount of data iteration, and the datadriven method has the problems of long training time and low execution efficiency. To solve this problem, a UCAV maneuver decision method based on deep reinforcement learning algorithm is proposed in this paper. Firstly, the flight drive module is constructed on the basis of UCAV threedegreeoffreedom model to form the state transition updating mechanism. Then, on the basis of PPO algorithm, ornsteinuhlenbeck (OU) random noise was added to improve UCAV's ability to explore unknown state space, and LSTM was combined to enhance UCAV's ability to learn sequence sample data, so as to improve the training efficiency and effect of the algorithm. Finally, the effectiveness and superiority of the proposed method are verified by designing three groups of closerange air combat simulation experiments and comparing the performance with PPO algorithm.