Contained in:
Book Chapter

Semi-Automated Visual Quality Control Inspection During Construction or Renovation of Railways Using Deep Learning Techniques and Augmented Reality Visualization

  • Apostolia Gounaridou
  • Evangelia Pantraki
  • Vasileios Dimitriadis
  • Athanasios Tsakiris
  • Dimosthenis Ioannidis
  • Dimitrios Tzovaras

The construction industry stands to greatly benefit from the technological advancements in deep learning and computer vision, which can automate time-consuming tasks such as quality control. In this paper, we introduce a framework that incorporates two advanced tools - the Visual Quality Control (VQC) tool and the Digital Twin visualization with Augmented Reality (DigiTAR) tool - to perform semi-automated visual quality control in the construction site during the execution phase of the project. The VQC tool is a backend service that detects potential defects on images captured on-site using the Mask R-CNN algorithm trained on annotated images of concrete and railway defects. The surveyor, aided by the Augmented Reality (AR) technology through the DigiTAR tool, can in-situ confirm/reject the detected defects and propose remedial actions. All the quality control results are recorded in the relevant BIM model and can be viewed on-site overlaid on the physical construction elements. This solution offers a semi-automated visual inspection that can speed up and simplify the quality control process, especially in case of large linear infrastructures, illustrating the added value of AR-based applications in Digital Twins

  • Keywords:
  • BIM,
  • Augmented Reality,
  • AR in Construction,
  • Deep Learning,
  • Computer Vision,
  • Visual Inspection,
  • Digital Twins,
+ Show More

Apostolia Gounaridou

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-0381-7505

Evangelia Pantraki

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-0359-9667

Vasileios Dimitriadis

Centre for Research and Technology Hellas, Greece - ORCID: 0009-0001-8880-119X

Athanasios Tsakiris

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0003-2552-7346

Dimosthenis Ioannidis

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0002-5747-2186

Dimitrios Tzovaras

Centre for Research and Technology Hellas, Greece - ORCID: 0000-0001-6915-6722

  1. Abdulla. (2017). Mask R-CNN for object detection and segmentation using TensorFlow 2.0. In GitHub repository. https://github.com/ahmedfgad/Mask-RCNN-TF2
  2. Atha, D. J., & Jahanshahi, M. R. (2018). Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Structural Health Monitoring, 17(5), 1110–1128. DOI: 10.1177/1475921717737051
  3. Attard, L., Debono, C. J., Valentino, G., Castro, M. Di, Masi, A., & Scibile, L. (2019). Automatic crack detection using Mask R-CNN. 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), 152–157. DOI: 10.1109/ISPA.2019.8868619
  4. Brien, D. O., Osborne, J. A., Perez-Duenas, E., Cunningham, R., & Li, Z. (2023). Automated crack classification for the CERN underground tunnel infrastructure using deep learning. Tunnelling and Underground Space Technology, 131. DOI: 10.1016/j.tust.2022.104668
  5. Cao, X., Xie, W., Ahmed, S. M., & Li, C. R. (2020). Defect detection method for rail surface based on line-structured light. Measurement: Journal of the International Measurement Confederation, 159. DOI: 10.1016/j.measurement.2020.107771
  6. Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. (2018). Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering, 33(9), 731–747. DOI: 10.1111/mice.12334
  7. Chi, H. L., Kim, M. K., Liu, K. Z., Thedja, J. P. P., Seo, J., & Lee, D. E. (2022). Rebar inspection integrating augmented reality and laser scanning. Automation in Construction, 136. DOI: 10.1016/j.autcon.2022.104183
  8. COGITO project. (n.d.). Retrieved September 29, 2023, from https://cogito-project.eu/
  9. Concrete Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from https://www.kaggle.com/datasets/motono0223/concrete-crack-segmentation-dataset
  10. Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from https://www.kaggle.com/datasets/lakshaymiddha/crack-segmentation-dataset?select=crack_segmentation_dataset
  11. Gan, J., Li, Q., Wang, J., & Yu, H. (2017). A hierarchical extractor-based visual rail surface inspection system. IEEE Sensors Journal, 17(23), 7935–7944. DOI: 10.1109/JSEN.2017.2761858
  12. García-Pereira, I., Portalés, C., Gimeno, J., & Casas, S. (2020). A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimedia Tools and Applications, 79(9–10), 6483–6501. DOI: 10.1007/s11042-019-08419-x
  13. Guo, F., Qian, Y., Rizos, D., Suo, Z., & Chen, X. (2021). Automatic rail surface defects inspection based on mask r-cnn. In Transportation Research Record (Vol. 2675, Issue 11, pp. 655–668). SAGE Publications Ltd. DOI: 10.1177/03611981211019034
  14. He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. http://arxiv.org/abs/1703.06870
  15. Kumar, P., Sharma, A., & Kota, S. R. (2021). Automatic multiclass instance segmentation of concrete damage using deep learning model. IEEE Access, 9, 90330–90345. DOI: 10.1109/ACCESS.2021.3090961
  16. Kwon, O. S., Park, C. S., & Lim, C. R. (2014). A defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality. Automation in Construction, 46, 74–81. DOI: 10.1016/j.autcon.2014.05.005
  17. Laxman, K. C., Tabassum, N., Ai, L., Cole, C., & Ziehl, P. (2023). Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 370. DOI: 10.1016/j.conbuildmat.2023.130709
  18. Liang, Z., Zhang, H., Liu, L., He, Z., & Zheng, K. (2019). Defect detection of rail surface with deep convolutional neural networks. Proceedings of the World Congress on Intelligent Control and Automation (WCICA), 2018-July, 1317–1322. DOI: 10.1109/WCICA.2018.8630525
  19. Lockley, S., Benghi, C., & Černý, M. (2017). Xbim.Essentials: A library for interoperable building information applications. The Journal of Open Source Software, 2(20), 473. DOI: 10.21105/joss.00473
  20. Meng, S., Gao, Z., Zhou, Y., He, B., & Djerrad, A. (2023). Real-time automatic crack detection method based on drone. Computer-Aided Civil and Infrastructure Engineering, 38(7), 849–872. DOI: 10.1111/mice.12918
  21. Microsoft HoloLens2. (n.d.). Retrieved September 29, 2023, from https://www.microsoft.com/en-us/hololens/
  22. Papamarkou, T., Guy, H., Kroencke, B., Miller, J., Robinette, P., Schultz, D., Hinkle, J., Pullum, L., Schuman, C., Renshaw, J., & Chatzidakis, S. (2021). Automated detection of corrosion in used nuclear fuel dry storage canisters using residual neural networks. Nuclear Engineering and Technology, 53(2), 657–665. DOI: 10.1016/j.net.2020.07.020
  23. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. DOI: 10.1007/s11263-007-0090-8
  24. Vuforia Engine. (n.d.). Retrieved September 29, 2023, from https://developer.vuforia.com/downloads/SDK
  25. Wei, F., Yao, G., Yang, Y., & Sun, Y. (2019). Instance-level recognition and quantification for concrete surface bughole based on deep learning. Automation in Construction, 107. DOI: 10.1016/j.autcon.2019.102920
  26. Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019). Railway track fastener defect detection based on image processing and deep learning techniques: A comparative study. Engineering Applications of Artificial Intelligence, 80, 66–81. DOI: 10.1016/j.engappai.2019.01.008
  27. Xu, X., Zhao, M., Shi, P., Ren, R., He, X., Wei, X., & Yang, H. (2022). Crack detection and comparison study based on faster R-CNN and Mask R-CNN. Sensors, 22(3). DOI: 10.3390/s22031215
  28. Xu, Y., Li, D., Xie, Q., Wu, Q., & Wang, J. (2021). Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN. Measurement: Journal of the International Measurement Confederation, 178. DOI: 10.1016/j.measurement.2021.109316
  29. Xue, Y., Cai, X., Shadabfar, M., Shao, H., & Zhang, S. (2020). Deep learning-based automatic recognition of water leakage area in shield tunnel lining. DOI: 10.17632/xz2nykszbs.1
  30. Xue, Y., & Li, Y. (2018). A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering, 33(8), 638–654. DOI: 10.1111/mice.12367
  31. Zhang, Z., Liang, M., & Wang, Z. (2021). A deep extractor for visual rail surface inspection. IEEE Access, 9, 21798–21809. DOI: 10.1109/ACCESS.2021.3055512
  32. Zhang, Z., Yu, S., Yang, S., Zhou, Y., & Zhao, B. (2021). Rail-5k: A real-world dataset for rail surface defects detection. http://arxiv.org/abs/2106.14366
  33. Zheng, D., Li, L., Zheng, S., Chai, X., Zhao, S., Tong, Q., Wang, J., & Guo, L. (2021). A defect detection method for rail surface and fasteners based on deep convolutional neural network. Computational Intelligence and Neuroscience, 2021. DOI: 10.1155/2021/2565500
  34. Zhou, Y., Luo, H., & Yang, Y. (2017). Implementation of augmented reality for segment displacement inspection during tunneling construction. Automation in Construction, 82, 112–121. DOI: 10.1016/j.autcon.2017.02.007
PDF
  • Publication Year: 2023
  • Pages: 865-876

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

Semi-Automated Visual Quality Control Inspection During Construction or Renovation of Railways Using Deep Learning Techniques and Augmented Reality Visualization

Authors

Apostolia Gounaridou, Evangelia Pantraki, Vasileios Dimitriadis, Athanasios Tsakiris, Dimosthenis Ioannidis, Dimitrios Tzovaras

DOI

10.36253/979-12-215-0289-3.86

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

140

Fulltext
downloads

125

Views

Export Citation

1,354

Open Access Books

in the Catalogue

2,362

Book Chapters

3,870,371

Fulltext
downloads

4,516

Authors

from 936 Research Institutions

of 66 Nations

66

scientific boards

from 351 Research Institutions

of 43 Nations

1,249

Referees

from 381 Research Institutions

of 38 Nations