Contained in:
Book Chapter

Semi-Automated Visual Quality Control Inspection During Construction or Renovation of Railways Using Deep Learning Techniques and Augmented Reality Visualization

  • Apostolia Gounaridou
  • Evangelia Pantraki
  • Vasileios Dimitriadis
  • Athanasios Tsakiris
  • Dimosthenis Ioannidis
  • Dimitrios Tzovaras

The construction industry stands to greatly benefit from the technological advancements in deep learning and computer vision, which can automate time-consuming tasks such as quality control. In this paper, we introduce a framework that incorporates two advanced tools - the Visual Quality Control (VQC) tool and the Digital Twin visualization with Augmented Reality (DigiTAR) tool - to perform semi-automated visual quality control in the construction site during the execution phase of the project. The VQC tool is a backend service that detects potential defects on images captured on-site using the Mask R-CNN algorithm trained on annotated images of concrete and railway defects. The surveyor, aided by the Augmented Reality (AR) technology through the DigiTAR tool, can in-situ confirm/reject the detected defects and propose remedial actions. All the quality control results are recorded in the relevant BIM model and can be viewed on-site overlaid on the physical construction elements. This solution offers a semi-automated visual inspection that can speed up and simplify the quality control process, especially in case of large linear infrastructures, illustrating the added value of AR-based applications in Digital Twins

  • Keywords:
  • BIM,
  • Augmented Reality,
  • AR in Construction,
  • Deep Learning,
  • Computer Vision,
  • Visual Inspection,
  • Digital Twins,
+ Show More

Apostolia Gounaridou

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-0381-7505

Evangelia Pantraki

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-0359-9667

Vasileios Dimitriadis

Centre for Research and Technology Hellas, Greece - ORCID: 0009-0001-8880-119X

Athanasios Tsakiris

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0003-2552-7346

Dimosthenis Ioannidis

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-5747-2186

Dimitrios Tzovaras

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0001-6915-6722

  1. Abdulla. (2017). Mask R-CNN for object detection and segmentation using TensorFlow 2.0. In GitHub repository.
  2. Atha, D. J., & Jahanshahi, M. R. (2018). Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Structural Health Monitoring, 17(5), 1110–1128. DOI: 10.1177/1475921717737051
  3. Attard, L., Debono, C. J., Valentino, G., Castro, M. Di, Masi, A., & Scibile, L. (2019). Automatic crack detection using Mask R-CNN. 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), 152–157. DOI: 10.1109/ISPA.2019.8868619
  4. Brien, D. O., Osborne, J. A., Perez-Duenas, E., Cunningham, R., & Li, Z. (2023). Automated crack classification for the CERN underground tunnel infrastructure using deep learning. Tunnelling and Underground Space Technology, 131. DOI: 10.1016/j.tust.2022.104668
  5. Cao, X., Xie, W., Ahmed, S. M., & Li, C. R. (2020). Defect detection method for rail surface based on line-structured light. Measurement: Journal of the International Measurement Confederation, 159. DOI: 10.1016/j.measurement.2020.107771
  6. Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. (2018). Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering, 33(9), 731–747. DOI: 10.1111/mice.12334
  7. Chi, H. L., Kim, M. K., Liu, K. Z., Thedja, J. P. P., Seo, J., & Lee, D. E. (2022). Rebar inspection integrating augmented reality and laser scanning. Automation in Construction, 136. DOI: 10.1016/j.autcon.2022.104183
  8. COGITO project. (n.d.). Retrieved September 29, 2023, from
  9. Concrete Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from
  10. Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from
  11. Gan, J., Li, Q., Wang, J., & Yu, H. (2017). A hierarchical extractor-based visual rail surface inspection system. IEEE Sensors Journal, 17(23), 7935–7944. DOI: 10.1109/JSEN.2017.2761858
  12. García-Pereira, I., Portalés, C., Gimeno, J., & Casas, S. (2020). A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimedia Tools and Applications, 79(9–10), 6483–6501. DOI: 10.1007/s11042-019-08419-x
  13. Guo, F., Qian, Y., Rizos, D., Suo, Z., & Chen, X. (2021). Automatic rail surface defects inspection based on mask r-cnn. In Transportation Research Record (Vol. 2675, Issue 11, pp. 655–668). SAGE Publications Ltd. DOI: 10.1177/03611981211019034
  14. He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN.
  15. Kumar, P., Sharma, A., & Kota, S. R. (2021). Automatic multiclass instance segmentation of concrete damage using deep learning model. IEEE Access, 9, 90330–90345. DOI: 10.1109/ACCESS.2021.3090961
  16. Kwon, O. S., Park, C. S., & Lim, C. R. (2014). A defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality. Automation in Construction, 46, 74–81. DOI: 10.1016/j.autcon.2014.05.005
  17. Laxman, K. C., Tabassum, N., Ai, L., Cole, C., & Ziehl, P. (2023). Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 370. DOI: 10.1016/j.conbuildmat.2023.130709
  18. Liang, Z., Zhang, H., Liu, L., He, Z., & Zheng, K. (2019). Defect detection of rail surface with deep convolutional neural networks. Proceedings of the World Congress on Intelligent Control and Automation (WCICA), 2018-July, 1317–1322. DOI: 10.1109/WCICA.2018.8630525
  19. Lockley, S., Benghi, C., & Černý, M. (2017). Xbim.Essentials: A library for interoperable building information applications. The Journal of Open Source Software, 2(20), 473. DOI: 10.21105/joss.00473
  20. Meng, S., Gao, Z., Zhou, Y., He, B., & Djerrad, A. (2023). Real-time automatic crack detection method based on drone. Computer-Aided Civil and Infrastructure Engineering, 38(7), 849–872. DOI: 10.1111/mice.12918
  21. Microsoft HoloLens2. (n.d.). Retrieved September 29, 2023, from
  22. Papamarkou, T., Guy, H., Kroencke, B., Miller, J., Robinette, P., Schultz, D., Hinkle, J., Pullum, L., Schuman, C., Renshaw, J., & Chatzidakis, S. (2021). Automated detection of corrosion in used nuclear fuel dry storage canisters using residual neural networks. Nuclear Engineering and Technology, 53(2), 657–665. DOI: 10.1016/
  23. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. DOI: 10.1007/s11263-007-0090-8
  24. Vuforia Engine. (n.d.). Retrieved September 29, 2023, from
  25. Wei, F., Yao, G., Yang, Y., & Sun, Y. (2019). Instance-level recognition and quantification for concrete surface bughole based on deep learning. Automation in Construction, 107. DOI: 10.1016/j.autcon.2019.102920
  26. Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019). Railway track fastener defect detection based on image processing and deep learning techniques: A comparative study. Engineering Applications of Artificial Intelligence, 80, 66–81. DOI: 10.1016/j.engappai.2019.01.008
  27. Xu, X., Zhao, M., Shi, P., Ren, R., He, X., Wei, X., & Yang, H. (2022). Crack detection and comparison study based on faster R-CNN and Mask R-CNN. Sensors, 22(3). DOI: 10.3390/s22031215
  28. Xu, Y., Li, D., Xie, Q., Wu, Q., & Wang, J. (2021). Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN. Measurement: Journal of the International Measurement Confederation, 178. DOI: 10.1016/j.measurement.2021.109316
  29. Xue, Y., Cai, X., Shadabfar, M., Shao, H., & Zhang, S. (2020). Deep learning-based automatic recognition of water leakage area in shield tunnel lining. DOI: 10.17632/xz2nykszbs.1
  30. Xue, Y., & Li, Y. (2018). A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering, 33(8), 638–654. DOI: 10.1111/mice.12367
  31. Zhang, Z., Liang, M., & Wang, Z. (2021). A deep extractor for visual rail surface inspection. IEEE Access, 9, 21798–21809. DOI: 10.1109/ACCESS.2021.3055512
  32. Zhang, Z., Yu, S., Yang, S., Zhou, Y., & Zhao, B. (2021). Rail-5k: A real-world dataset for rail surface defects detection.
  33. Zheng, D., Li, L., Zheng, S., Chai, X., Zhao, S., Tong, Q., Wang, J., & Guo, L. (2021). A defect detection method for rail surface and fasteners based on deep convolutional neural network. Computational Intelligence and Neuroscience, 2021. DOI: 10.1155/2021/2565500
  34. Zhou, Y., Luo, H., & Yang, Y. (2017). Implementation of augmented reality for segment displacement inspection during tunneling construction. Automation in Construction, 82, 112–121. DOI: 10.1016/j.autcon.2017.02.007
  • Publication Year: 2023
  • Pages: 865-876

  • Publication Year: 2023

Chapter Information

Chapter Title

Semi-Automated Visual Quality Control Inspection During Construction or Renovation of Railways Using Deep Learning Techniques and Augmented Reality Visualization


Apostolia Gounaridou, Evangelia Pantraki, Vasileios Dimitriadis, Athanasios Tsakiris, Dimosthenis Ioannidis, Dimitrios Tzovaras



Peer Reviewed

Publication Year


Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry


Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year


Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press



eISBN (pdf)


eISBN (xml)


Series Title

Proceedings e report

Series ISSN


Series E-ISSN






Export Citation


Open Access Books

in the Catalogue


Book Chapters





from 873 Research Institutions

of 64 Nations


scientific boards

from 340 Research Institutions

of 43 Nations



from 344 Research Institutions

of 37 Nations