In order to improve the robustness of visual moving object tracking, a tracking algorithm is proposed based on local patches and weighted background. Firstly, the local patches are sampled respectively from the foreground area and background area of the object. Secondly, the probability map of each patch to the frame image is calculated through the integral histogram based local exhaustive search. Finally, different weights are assigned to the searching results of patches by maximizing each probability map, and the final localization of the object in current frame is obtained by the weighted sum of the searching results of local patches. The experimental results indicate that the proposed algorithm exceeds the background weighted tracking and local patch based tracking in both tracking precision and tracking successful rate. Besides, the proposed algorithm is not too complex.