In order to improve the feature expression ability of the target tracking algorithm based on Siamese network and obtain better tracking performance, a lightweight Siamese network target tracking algorithm based on secondorder pooling feature fusion is proposed. First, the Siamese network architecture is used to obtain the deep features of the target; then, the secondorder pooling network and the lightweight channel attention are added in parallel at the end of the Siamese network architecture to obtain the secondorder pooling features and channel attention features of the target, respectively. Finally, the depth feature of the target, the secondorder pooling feature and the channel attention feature are fused, and the fused feature is used for crosscorrelation operation, and the obtained response graph can distinguish the target and the background well, and improve the discriminative ability of the model, and improve the accuracy of target positioning, thereby improving target tracking performance. The proposed algorithm only uses the Got10k dataset for endtoend training and is validated on the OTB100 and VOT2018 datasets. The experimental results show that the proposed algorithm achieves a significant improvement in tracking performance compared with the benchmark algorithm SiamFC: on the OTB100 dataset, the accuracy and success rate are increased by 7.5% and 5.2%, respectively; on the VOT2018 dataset, the expected average overlap rate increases by 4.3%.