Contained in:
Book Chapter

Integrating Real-Time Object Detection into an AR-Driven Task Assistance Prototype: An Approach Towards Reducing Specific Motions in Therbligs Theory

  • Xiang Yuan
  • Qipei Mei
  • Xinming Li

Due to challenges in filling vacant positions and the heightened demands posed on existing staff, employers and project managers are progressively considering the recruitment of inexperienced individuals and seeking strategies to swiftly provide them with essential job-specific knowledge. The potential of industrial AR has been widely researched to support workers in overcoming skill-related knowledge and enhancing industrial processes. However, most studies focus on demonstrating technology usability across different processes and overcoming engineering hurdles on a case-by-case basis. There is no direct benefit analysis on how AR assists construction tasks at human motion level, and how to eliminate the ineffective motions and reduce the duration of effective motions. To fill this gap, this paper first establishes an AR-based near real-time object detection system of small tools and components involved in task processes for egocentric perception of workers in the construction industry. Later, the Standard Operating Procedure (SOP) for scaffolding assembly activities is deconstructed from a manual process into Therbligs-based elemental motions. Finally, this research conducted a comparative study of two prototypes across four dimensions of evaluation. As a step forward in this direction, this paper renews the connotations of Therbligs theory under industry 5.0 era, rethinks the AR-assisted construction task processes, and applies appropriate technologies enhancing the adaptability of AR technology for construction workers’ needs

  • Keywords:
  • Augmented Reality (AR); Microsoft HoloLens 2; Object Detection; Task Assistance; Therbligs,
+ Show More

Xiang Yuan

University of Alberta, Canada

Qipei Mei

University of Alberta, Canada - ORCID: 0000-0003-1409-3562

Xinming Li

University of Alberta, Canada - ORCID: 0000-0001-6802-033X

  1. Butaslac, I. M., Fujimoto, Y., Sawabe, T., Kanbara, M., & Kato, H. (2022). Systematic Review of Augmented Reality Training Systems. IEEE Transactions on Visualization and Computer Graphics, 1–20. DOI: 10.1109/TVCG.2022.3201120
  2. Büttner, S., Prilla, M., & Röcker, C. (2020). Augmented Reality Training for Industrial Assembly Work—Are Projection-based AR Assistive Systems an Appropriate Tool for Assembly Training? Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. Honolulu HI USA: ACM. DOI: 10.1145/3313831.3376720
  3. Casano, D. (2021). HoloHelp: HoloLens Detection for a Guided Interaction. University of Catania.
  4. David, D. (2000) Therbligs: The Keys to Simplifying Work, The Gilbreth Network: Therbligs. Available at: https://gilbrethnetwork.tripod.com/therbligs.html (Accessed: 26 July 2023).
  5. de Souza Cardoso, L. F., Mariano, F. C. M. Q., & Zorzal, E. R. (2020). A survey of industrial augmented reality. Computers & Industrial Engineering, 139, 106159. DOI: 10.1016/j.cie.2019.106159
  6. Eswaran, M., & Bahubalendruni, M. V. A. R. (2022). Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review. Journal of Manufacturing Systems, 65, 260–278. (36). DOI: 10.1016/j.jmsy.2022.09.016
  7. Farasin, A., Peciarolo, F., Grangetto, M., Gianaria, E., & Garza, P. (2020). Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens: Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 165–172. Valletta, Malta: SCITEPRESS - Science and Technology Publications. DOI: 10.5220/0008877901650172
  8. Fuglseth, S. S. (2022). Object Detection with HoloLens 2 using Mixed Reality and Unity a proof-of-concept (Bachelor thesis, Høgskolen i Molde - Vitenskapelig høgskole i logistikk). Høgskolen i Molde - Vitenskapelig høgskole i logistikk. Retrieved from https://himolde.brage.unit.no/himolde-xmlui/handle/11250/3023916
  9. George, R. (2021). Using Object Recognition on Hololens 2 for Assembly (M.S.). Retrieved from https://www.proquest.com/docview/2621280160/abstract/34D5D9DBA2ED4869PQ/1
  10. Ghasemi, Y., Jeong, H., Choi, S. H., Park, K.-B., & Lee, J. Y. (2022). Deep learning-based object detection in augmented reality: A systematic review. Computers in Industry, 139, 103661. DOI: 10.1016/j.compind.2022.103661
  11. Government of Canada, S. C. (2022). Retrieved from https://www.statcan.gc.ca/en/subjects-start/labour_/labour-shortage-trends-canada#shr-pg0
  12. Grubert, J., Hamacher, D., Mecke, R., Böckelmann, I., Schega, L., Huckauf, A., … Tümler, J. (2010). Extended investigations of user-related issues in mobile industrial AR. 2010 IEEE International Symposium on Mixed and Augmented Reality, 229–230. DOI: 10.1109/ISMAR.2010.5643581
  13. Ke, Y. (2018). Research on the Chinese Industrialized Construction Migrant Workers from the Perspective of Complex Adaptive System: Combining the Application of SWARM Computer Simulation Technology. Wireless Personal Communications, 102(4), 2469–2481. DOI: 10.1007/s11277-018-5266-8
  14. Khan, N., Saleem, M. R., Lee, D., Park, M.-W., & Park, C. (2021). Utilizing safety rule correlation for mobile scaffolds monitoring leveraging deep convolution neural networks. Computers in Industry, 129. Scopus. DOI: 10.1016/j.compind.2021.103448
  15. Kim, J., Olsen, D., & Renfroe, J. (2022). Construction Workforce Training Assisted with Augmented Reality. 2022 8th International Conference of the Immersive Learning Research Network (iLRN), 1–6. DOI: 10.23919/iLRN55037.2022.9815960
  16. Lee, K., Jeon, C., & Shin, D. H. (2023). Small Tool Image Database and Object Detection Approach for Indoor Construction Site Safety. KSCE Journal of Civil Engineering, 27(3), 930–939. DOI: 10.1007/s12205-023-1011-2
  17. Liao, W., Iseley, T., & Behbahani, S. (2022). Industry/University Cooperative Research Centers (IUCRC): A Critical Component for Addressing Underground Infrastructure Challenges. 56–66. DOI: 10.1061/9780784484289.007
  18. Łysakowski, M., Żywanowski, K., Banaszczyk, A., Nowicki, M. R., Skrzypczyński, P., & Tadeja, S. K. (2023, June 6). Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8. arXiv. Retrieved from http://arxiv.org/abs/2306.03537
  19. Niebel, B., & Freivalds, A. (2013). Niebel’s Methods, Standards, & Work Design. McGraw-Hill Education.
  20. Ninjatacoshell. (2012). English: The 18 therbligs. Own work. Retrieved from https://commons.wikimedia.org/wiki/File:Therblig_(English).svg
  21. Oyekan, J., Hutabarat, W., Turner, C., Arnoult, C., & Tiwari, A. (2020). Using Therbligs to embed intelligence in workpieces for digital assistive assembly. Journal of Ambient Intelligence and Humanized Computing, 11(6), 2489–2503. DOI: 10.1007/s12652-019-01294-2
  22. PatrickFarley. (2023, July 18). What is Custom Vision? - Azure AI services. Retrieved October 10, 2023, from https://learn.microsoft.com/en-us/azure/ai-services/custom-vision-service/overview
  23. Peñaloza, G. A., Saurin, T. A., & Formoso, C. T. (2020). Monitoring complexity and resilience in construction projects: The contribution of safety performance measurement systems. Applied Ergonomics, 82, 102978. DOI: 10.1016/j.apergo.2019.102978
  24. Qin, Y., Wang, S., Zhang, Q., Cheng, Y., Huang, J., & He, W. (2023). Assembly training system on HoloLens using embedded algorithm. Third International Symposium on Computer Engineering and Intelligent Communications (ISCEIC 2022), 12462, 121–128. SPIE. DOI: 10.1117/12.2660940
  25. Sung, R. C. W., Ritchie, J. M., Lim, T., & Medellin, H. (2009). Assembly planning and motion study using virtual reality. 31–38. Scopus. DOI: 10.1115/WINVR2009-713
  26. Tao, W., Lai, Z.-H., Leu, M. C., Yin, Z., & Qin, R. (2019). A self-aware and active-guiding training & assistant system for worker-centered intelligent manufacturing. Manufacturing Letters, 21, 45–49. DOI: 10.1016/j.mfglet.2019.08.003
  27. The Home Depot. (2022). Retrieved from https://www.homedepot.ca/product/metaltech-scaffold-bench-multipurpose-4-in-1-6-ft-baker-scaffold/1001160246
  28. Trinh, M. T., & Feng, Y. (2020). Impact of Project Complexity on Construction Safety Performance: Moderating Role of Resilient Safety Culture. Journal of Construction Engineering and Management, 146(2), 04019103. DOI: 10.1061/(ASCE)CO.1943-7862.0001758
  29. Ungureanu, D., Bogo, F., Galliani, S., Sama, P., Duan, X., Meekhof, C., … Pollefeys, M. (2020, August 25). HoloLens 2 Research Mode as a Tool for Computer Vision Research. arXiv. DOI: 10.48550/arXiv.2008.11239
  30. Wang, B., Zhang, Z., Jiang, C., Zhao, Y., Ding, S., Xu, F., & Niu, J. (2021). A Novel Approach Combined with Therbligs and VACP Model to Evaluate the Workload During Simulated Maintenance Task. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12771 LNCS, 164–173. Scopus. DOI: 10.1007/978-3-030-77074-7_13
  31. Wolf, J., Wolfer, V., Halbe, M., Maisano, F., Lohmeyer, Q., & Meboldt, M. (2021). Comparing the effectiveness of augmented reality-based and conventional instructions during single ECMO cannulation training. International Journal of Computer Assisted Radiology and Surgery, 16(7), 1171–1180. DOI: 10.1007/s11548-021-02408-y
  32. Wu, S., Hou, L., & Chen, H. (n.d.). Measuring the impact of Augmented Reality warning systems on onsite construction workers using object detection and eye-tracking.
  33. Wu, S., Hou, L., Zhang, G. (Kevin), & Chen, H. (2022). Real-time mixed reality-based visual warning for construction workforce safety. Automation in Construction, 139, 104252. DOI: 10.1016/j.autcon.2022.104252
  34. Wu, Z., Zhao, T., & Nguyen, C. (2020). 3D Reconstruction and Object Detection for HoloLens. 2020 Digital Image Computing: Techniques and Applications (DICTA), 1–2. DOI: 10.1109/DICTA51227.2020.9363378
  35. Zhang, Y., Xuan, Y., Yadav, R., Omrani, A., & Fjeld, M. (2023, February 3). Playing with Data: An Augmented Reality Approach to Interact with Visualizations of Industrial Process Tomography. arXiv. DOI: 10.48550/arXiv.2302.01686
PDF
  • Publication Year: 2023
  • Pages: 121-132

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

Integrating Real-Time Object Detection into an AR-Driven Task Assistance Prototype: An Approach Towards Reducing Specific Motions in Therbligs Theory

Authors

Xiang Yuan, Qipei Mei, Xinming Li

DOI

10.36253/979-12-215-0289-3.12

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

149

Fulltext
downloads

115

Views

Export Citation

1,347

Open Access Books

in the Catalogue

2,262

Book Chapters

3,790,127

Fulltext
downloads

4,421

Authors

from 923 Research Institutions

of 65 Nations

65

scientific boards

from 348 Research Institutions

of 43 Nations

1,248

Referees

from 380 Research Institutions

of 38 Nations