Joint Segmentation and Recognition of Worker Actions using Semi-Markov Models

Vision-based automated recognition of worker actions has gain lots of interest during the past few years. However, existing research all requires pre-segmented video clips, which is not applicable in the real situation. Furthermore, pre-segmented videos abandon the temporal information of action transition. A joint action segmentation and recognition method, which can segment continuous video stream while recognizing the action type for each segment, is an urgent need. In this paper, we model the worker actions with a discriminative semi-Markov model. In the model, a set of features is defined to capture both the local and global characteristics of each action cycle. Then the semi-Markov model is formulated as an optimization problem and solved by the cutting plane method for simultaneous action segmentation and recognition. Scale-Invariant Feature Transform (SIFT) is applied to detect feature points in the region of interest in every frame. Two descriptors (Histograms of Oriented Gradients ? HOG, Histograms of Optical Flow ? HOF), are computed in the feature points to encode the scenario and motion flow simultaneously. Finally, the Bag-of-Feature strategy is adopted for feature representation. Experimental results from real world construction videos show that the proposed method is able to segment and recognize continuous worker actions correctly, resulting in a prospecting application in automated productivity analysis.