Abstract: The ship detection in infrared video has wide application value in fishery administration, port monitoring and other fields. Traditional background modeling methods, such as GMM(Gaussian mixture model), Codebook, and ViBe, will make more false detection in the ship detection from the infrared ocean video because of the impact of the waves. The paper proposes a new algorithm to detect ships in the infrared ocean video. The algorithm framework adopts the Top-Hat operation to preprocess the infrared image to filter the clutter effectively, then improves Vibe algorithm to detect the moving ship target. Experimental results show that the method can effectively suppress the background noise and get better detection results.