In recent years, deterioration of concrete structures such as floating and delamination has occurred. These damages can lead to the falling of concrete pieces. It may cause damage to third parties such as vehicles or pedestrians passing under the bridge. Therefore, the concrete damages need to be detected and repaired early. Japanese highway companies conduct hammering tests on concrete structures once every five years to detect damages such as floating and flaking. However, the inspector must be near the bridge to perform the hammering sound inspection, which requires work at a high place and a lot of time and cost. Also, when using aerial work vehicles, traffic regulation under the bridge is required which may cause traffic congestion and accidents. Infrared thermography is a non-destructive inspection technique that detects areas of floating or delamination by imaging a unique temperature distribution on the concrete surface using an infrared camera. The inspection cost of infrared thermography is significantly lower than that of hammering test. Since it is a long-distance nondestructive inspection, work at heights and traffic restrictions are not required. However, it is difficult to ensure the accuracy of damage detection because the peculiar temperature distribution acquired by infrared thermography includes an abnormal temperature distribution caused by something other than damages. In this study, by using deep learning, we improved the accuracy of the unique temperature distribution imaged by infrared thermography and ensured detection accuracy that can be used practically. We also report on the construction of an automatic discrimination system that implements the deep learning discrimination algorithm on a cloud server.