Construction workers are commonly subjected to ergonomic risks due to manual material handling that requires high levels of energy input over long work hours. Fatigue in musculature is associated with decline in postural stability, motor performance, and altered normal motion patterns, leading to heightened risks of work-related musculoskeletal disorders. Physical fatigue has been previously demonstrated to be a good indicator of injury risks, thus, monitoring and detecting muscle fatigue during strenuous work may be advantageous in mitigating these risks. Currently, few researchers have investigated how physical fatigue and exertion can be continuously monitored for practical use outside laboratory settings. Exercise-induced fatigue has been shown to impact motor control; thus, it can be measured using jerk, the time derivative of acceleration. This paper investigates the application of a machine learning approach, Support Vector Machine (SVM), to automatically recognize jerk changes due to physical exertion. We hypothesized that physical exertion and fatigue will influence motions and thus, can be classified based on jerk values. The motion data of six expert masons were collected using IMU sensors during two bricklaying tasks. The pelvis, upper arms, and thighs jerk values were used to classify inter- and intra-subject rested and exerted states. Our results show that on average, intra-subject classification achieved an accuracy of 94% for a five-course wall building experiment and 80% for a first-course experiment, leading us to conclude that jerk changes due to physical exertion can be detected using wearable sensors and SVMs.