Semantic segmentation of closed-circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel in the image, from which defect types, locations and geometric information can be obtained. In this study, a deep convolutional neural network (CNN), namely DilaSeg, is developed based on dilated convolution for improving the segmentation of sewer pipe defects including cracks, tree root intrusion and deposit. Sewer pipe CCTV images are extracted from inspection videos and are annotated to be used as the ground truth labels for training the model. DilaSeg is constructed with dilated convolution for producing feature maps with high resolution. Both DilaSeg and the state-of-the-art model, fully convolutional network (FCN), are trained and evaluated on the annotated dataset using the same hyper-parameters. The results of the experiments indicate that the proposed DilaSeg improved the segmentation accuracy significantly compared with FCN, with 18% of increase in mean pixel accuracy (mPA) and 22% of increase in mean intersection over union (IoU) with a fast detection speed.