This study quantitatively evaluated the interpretability of deep learning–based concrete damage classification models using Grad-CAM and compared the results with performance metrics to establish fundamental criteria for practical applications. Three representative CNN models— GoogLeNet, ResNet-50, and EfficientNet-B0—were tested with varying dataset sizes (750, 1500, 3000 images) and Grad-CAM threshold values (0.3, 0.5, 0.7). Model performance was assessed using accuracy and F1-score, while interpretability was evaluated with the Grad-CAM–based Damage Ratio. The experimental results showed that both performance and interpretability improved as the dataset size increased; however, a trade-off between the two metrics was observed. EfficientNet-B0 achieved the highest accuracy, whereas GoogLeNet produced wider activation regions with a higher Damage Ratio. In addition, threshold 0.5 yielded the most balanced results in terms of interpretability and noise suppression. In conclusion, this study highlights the importance of balancing performance and interpretability in deep learning–based structural damage diagnosis and proposes baseline criteria for model and threshold selection. Future research should focus on enhancing interpretability by incorporating diverse damage types and real-world structural data.