The computer vision-based detection of construction workers in images or videos is necessary for the safety managements and productivity of construction workers. Researchers in previous studies, detecting construction workers on the construction scene via computer vision techniques, have considered various features such as motion, shape, and color. Due to the pose changes of the workers, construction worker detection using body shape as a feature in the construction scene remains a challenging task. This study proposes a safety vest detection method, as a preceding method of the construction worker detection, which uses the motion of workers and the color pixels of safety vests for distinguishing from others on the construction scene independent of the workers pose changes. The background subtraction method is performed by using an approximate median filter for the purpose of reducing the candidate regions that the color pixel classification will be performed as a sequential step. Then, the color pixel classification method is performed with a comparative analysis of two color spaces (Lab and HSV [hue, saturation, and value]) and three types of classifiers (a support vector machine [SVM], an artificial neural network [ANN], and a logistic regression [LR]) to classify the pixels of safety vests accurately. The proposed method has been tested on the actual construction site video streams. The proposed method of this study, thus far, is making progress toward the achievement of the robust and effective construction worker detection on the construction scene.