Rehabilitation of urban infrastructures has received considerable attention in North America, creating a need for automation. Automating the rehabilitation process of various infrastructure facilities is driven by the need for cost reduction, higher quality and improved safety. This paper describes an automated system, AUTO-DETECT, recently developed, for rehabilitation of sewer pipes. AUTO-DETECT automatically analyzes the CCTV videotapes that depict the conditions of the surveyed pipes and consequently detects and classifies defects. It introduces five sets of specialized neural networks, each is dedicated for one type of defect. The paper also presents the integration aspects of these five sets of neural networks to formulate a solution strategy that is utilized to improve the performance of the developed diagnostic system.