Installation work of large inner/outer wall panel glasses increases the labor load and stress of workers and causes the danger of such accidents as falling and crane overturning. To solve this problem, Gil designed the easy handling robot system which is composed of mobile system, 5-DOF manipulator system and HRI (Human-Robot Interface) device included HRC (Human-Robot Cooperation) algorithm. During glass installation work using this robot system, detecting the position of the HRI device has great influence on the work efficiency. While working there, the construction worker placed the HRI device on panel glass randomly. Also there are many factors that disturb the detectin the HRI device. In this paper describes a method for detecting the HRI device robustly. To robustly find the HRI device, in this paper, applied the RANSAC (RANdom SAmple Consensus) and LSF (Least Square Fit) algorithm. And sensor module for detect the HRI device is composed of IR (Infrared Ray) sensor and RC servo motor. The distance between the sensor module and the circle-shaped HRI device is utilized for detecting this device. In this paper, to verify the robust method, performed laboratory experiment: for detecting the HRI device randomly placed on, for finding the HRI device put with a square-shaped disturbance.