Aimed at the problems that the target image detection methods are low in accuracy and slow at detection speed of current UAV (Unmanned Aerial Vehicles, UAV), a target detection algorithm in combination with the lightweight network and the improved multi scale structure is proposed. Firstly, MobileNetV3 lightweight network is used to replace the backbone network of YOLOv4, reducing the model complexity and improving the detection speed. Secondly, the improved multi scale PANet network is introduced to enhance the flow superposition of high dimensional image features and low dimensional location features, and improve the classification and location accuracy of small targets. Finally, the K means method is introduced to optimize the parameters of the target anchor frame to improve the detection efficiency. Meanwhile, a new UAV target image dataset Drone dataset is constructed by combinating with the open dataset and the self shot images. The results show that the mAP and FPS of the proposed algorithm reach 91.58% and 55 f/s, and the parameter number of 44.39M is only 1/6 of the YOLOv4 algorithm and is superior to the mainstream SSD, the YOLO series and the Faster R CNN algorithms.