Based on the complementarity of these two features, this paper presents an infrared target tracking algorithm in combination with the deep feature and the gradient feature. The deep feature and gradient feature are used to represent the semantic information and the local structure information in this paper, enhancing the ability to characterize any target. Next, the tracking models based on different features can further improve the tracking performance. Finally, the paper builds up a model mutual mechanism based on the combination of deep feature tracking model and gradient feature tracking model, implementing positioning precisely to the target. The article selects the latest infrared tracking database and uses the database to verify the effectiveness of the algorithm. The results show that the algorithm in this paper achieves a 3.8％ improvement in accuracy and obtains a 4.3％ improvement in success rate, enabling to effectively handle the effects of the background similarly and deformation.