Concrete structures are heavily used in most modern societies and the population of structures in need of inspection is rapidly growing. On the other hand, the manpower for inspection is decreasing. This has brought into focus the need for automated inspection methods for concrete structures. The hammering test is a popular method for inspection that uses the sound resulting from a hammer impact on the surface of the structure for defect detection. Previous methods largely employed machine learning approaches for the automation of the hammering test. Weakly supervised methods used positive queries answers on sample pair similarity: a human user was questioned on the similarity of pairs of hammering samples and similar pairs were used to transform the feature space. However, it can be expected that dissimilar pairs would also be gathered in this process. Therefore, in the present paper is proposed a method for weakly supervised defect detection in concrete structures using hammering with both positive and negative answers to queries. After the initial feature space transformation based on positive query answers, another feature space transformation is introduced based on negative query answers. Experiments in laboratory conditions showed the effectiveness of the proposed method.