Evaluating Road Segmentation Performance in Participatory Sensing: An Investigation into Alternative Metrics

For road condition assessment, participatory sensing has been proposed in literature utilizing a normal vehicle equipped with a dashboard camera. In such environment, the main technical challenge is not only the recognition performance on target classes such as cracks, but also the preparation of training and test datasets with high quality annotations. This study found that the annotation quality presents a unique problem in the performance test of participatory sensing-based road condition assessment. To address the problem, this study explores the adequacy of most commonly-used evaluation metric, the Intersection over Union (IoU), and suggest alternative metrics for road segmentation models in the context of participatory sensing. Experiments were conducted on the AIM crack dataset collected from urban road environments using dashboard cameras on normal vehicles. This study provides new insights into the importance of considering proper evaluation metrics in participatory sensing-based infrastructure monitoring.