Contained in:
Book Chapter

Predicting Mental Workload of Using Exoskeletons for Construction Work: A Deep Learning Approach

  • Abiola Akanmu
  • Adedeji Afolabi
  • Akinwale Okunola

Exoskeletons are gaining attention as a potential solution for addressing low back injury in the construction industry. However, use of active back-support exoskeletons in construction can trigger unintended consequences which could increase mental workload of users while working with exoskeletons. Prolonged increase in mental workload could impact workers’ wellbeing and productivity. Prediction of mental workload during exoskeleton-use could inform strategies to mitigate the triggers. This study investigates a machine-learning framework for predicting mental workload of workers while using active back-support exoskeletons for construction work. Laboratory experiments were conducted wherein Electroencephalography (EEG) data were collected from participants wearing active back-support exoskeletons to perform flooring task. The EEG data underwent preprocessing, including band filtering, notch filtering, and independent component analysis, to remove artifacts and ensure data quality. A regression-based Long Short-Term Memory network was trained to forecast future time steps of the processed EEG data. The performance of the network was evaluated using root mean square error (RMSE) and r-squared (R2). A RMSE of 0.1527 and R2 of 0.9665 indicating good fit and strong correlation, respectively, were observed between the predicted and actual EEG data. Results of the comparison between the actual and predicted mental workload also show strong correction with an R2 of 0.8692. The findings motivate research directions into real-time monitoring of mental workload of workers during exoskeleton-use. The study has significant implications for stakeholders, enabling them to gain a deeper understanding of the impact of mental workload while using exoskeletons thereby providing opportunities for mitigation

  • Keywords:
  • Work-related musculoskeletal disorders,
  • Exoskeleton,
  • Mental workload,
  • Electroencephalogram,
  • Long Short-Term Memory,
  • Flooring task,
+ Show More

Abiola Akanmu

Virginia Tech/Myers Lawson School of Construction, United States - ORCID: 0000-0001-9145-4865

Adedeji Afolabi

Virginia Tech/Myers Lawson School of Construction, United States - ORCID: 0000-0002-9065-4766

Akinwale Okunola

Virginia Tech/Myers Lawson School of Construction, United States - ORCID: 0009-0002-5235-1307

  1. Alabdulkarim, S., Kim, S., & Nussbaum, M. A. (2019). Effects of exoskeleton design and precision requirements on physical demands and quality in a simulated overhead drilling task. Applied Ergonomics, 80, 136-145. DOI: 10.1016/j.apergo.2019.05.014
  2. Alemi, M. M., Madinei, S., Kim, S., Srinivasan, D., & Nussbaum, M. A. (2020). Effects of Two Passive Back-Support Exoskeletons on Muscle Activity, Energy Expenditure, and Subjective Assessments During Repetitive Lifting. Hum Factors, 62(3), 458-474. DOI: 10.1177/0018720819897669
  3. Baltrusch, S. J., Houdijk, H., van Dieen, J. H., & de Kruif, J. T. C. M. (2021). Passive Trunk Exoskeleton Acceptability and Effects on Self-efficacy in Employees with Low-Back Pain: A Mixed Method Approach. Journal of Occupational Rehabilitation, 31(1), 129-141. DOI: 10.1007/s10926-020-09891-1
  4. Bequette, B., Norton, A., Jones, E., & Stirling, L. (2020). Physical and Cognitive Load Effects Due to a Powered Lower-Body Exoskeleton. Human Factors, 62(3), 411-423. DOI: 10.1177/0018720820907450
  5. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58-75. DOI: 10.1016/j.neubiorev.2012.10.003
  6. Chen, D., Huang, H., Bao, X., Pan, J., & Li, Y. (2023). An EEG-based attention recognition method: fusion of time domain, frequency domain, and non-linear dynamics features. Frontiers in Neuroscience, 17. DOI: 10.3389/fnins.2023.1194554.
  7. Chen, J. Y., Song, X. Y., & Lin, Z. H. (2016). Revealing the "Invisible Gorilla" in construction: Estimating construction safety through mental workload assessment. Automation in Construction, 63, 173-183. DOI: 10.1016/j.autcon.2015.12.018
  8. Chen, J. Y., Taylor, J. E., & Comu, S. (2017). Assessing Task Mental Workload in Construction Projects: A Novel Electroencephalography Approach. Journal of Construction Engineering and Management, 143(8). DOI: 10.1061/(ASCE)CO.1943-7862.0001345
  9. Coulibaly, P., & Baldwin, C. K. (2005). Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology, 307(1-4), 164-174. DOI: 10.1016/j.jhydrol.2004.10.008
  10. Cumplido-Trasmonte, C., Barquin-Santos, E., Garces-Castellote, E., Gor-Garcia-Fogeda, M. D., Plaza-Flores, A., Hernandez-Melero, M., Gutierrez-Ayala, A., Cano-de-la-Cuerda, R., Lopez-Moron, A. L., & Garcia-Armada, E. (2023). Safety and usability of the MAK exoskeleton in patients with stroke. Physiotherapy Research International. DOI: 10.1002/pri.2038
  11. de Looze, M. P., Bosch, T., Krause, F., Stadler, K. S., & O'Sullivan, L. W. (2016). Exoskeletons for industrial application and their potential effects on physical work load. Ergonomics, 59(5), 671-681. DOI: 10.1080/00140139.2015.1081988
  12. Fan, J., & Smith, A. P. (2017). The Impact of Workload and Fatigue on Performance. Communications in Computer and Information Science, 726, 90-105. DOI: 10.1007/978-3-319-61061-0_6
  13. Fox, S., Aranko, O., Heilala, J., & Vahala, P. (2020). Exoskeletons Comprehensive, comparative and critical analyses of their potential to improve manufacturing performance. Journal of Manufacturing Technology Management, 31(6), 1261-1280. DOI: 10.1108/Jmtm-01-2019-0023
  14. Frølich, L., & Dowding, I. (2018). Removal of muscular artifacts in EEG signals: a comparison of linear decomposition methods. Brain informatics, 5(1), 13-22. DOI: 10.1007/s40708-017-0074-6.
  15. Gonsalves, N., Akanmu, A., Gao, X. H., Agee, P., & Shojaei, A. (2023). Industry Perception of the Suitability of Wearable Robot for Construction Work. Journal of Construction Engineering and Management, 149(5). DOI: 10.1061/JCEMD4.COENG-12762
  16. Gonsalves, N. J., Ogunseiju, O. R., Akanmu, A. A., & Nnaji, C. A. (2021). Assessment of a Passive Wearable Robot for Reducing Low Back Disorders during Rebar Work. Journal of Information Technology in Construction, 26, 936-952. DOI: 10.36680/j.itcon.2021.050
  17. Gorgey, A. S. (2018). Robotic exoskeletons: The current pros and cons. World Journal of Orthopedics, 9(9), 112-119. DOI: 10.5312/wjo.v9.i9.112
  18. Guo, H. L., Zhang, Z. T., Yu, R., Sun, Y. K., & Li, H. (2023). Action Recognition Based on 3D Skeleton and LSTM for the Monitoring of Construction Workers' Safety Harness Usage. Journal of Construction Engineering and Management, 149(4). DOI: 10.1061/JCEMD4.COENG-12542
  19. Hernandez, C., Slaton, T., Balali, V., & Akhavian, R. (2019). A Deep Learning Framework for Construction Equipment Activity Analysis. Computing in Civil Engineering 2019: Data, Sensing, and Analytics, 479-486. DOI: 10.1061/9780784482438.061.
  20. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. DOI: 10.1162/neco.1997.9.8.1735.
  21. Jaiswal, A., Ramesh Babu, A., Zaki Zadeh, M., Wylie, G., & Makedon, F. (2023). Detecting Cognitive Fatigue in Subjects with Traumatic Brain Injury from FMRI Scans Using Self-Supervised Learning. Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, 83-90. DOI: 10.1145/3594806.3594868.
  22. Jebelli, H., Choi, B., & Lee, S. (2019). Application of wearable biosensors to construction sites. I: Assessing workers’ stress. Journal of Construction Engineering and Management, 145(12), 04019079. DOI: 10.1061/(ASCE)CO.1943-7862.0001729.
  23. Jebelli, H., Hwang, S., & Lee, S. (2018a). EEG-based workers' stress recognition at construction sites. Automation in Construction, 93, 315-324. DOI: 10.1016/j.autcon.2018.05.027.
  24. Jebelli, H., Hwang, S., & Lee, S. (2018b). EEG signal-processing framework to obtain high-quality brain waves from an off-the-shelf wearable EEG device. Journal of Computing in Civil Engineering, 32(1), 04017070. DOI: 10.1061/(ASCE)CP.1943-5487.0000719.
  25. Ke, J., Zhang, M., Luo, X., & Chen, J. (2021). Monitoring distraction of construction workers caused by noise using a wearable Electroencephalography (EEG) device. Automation in Construction, 125, 103598. DOI: 10.1016/j.autcon.2021.103598
  26. Kim, S., Moore, A., Srinivasan, D., Akanmu, A., Barr, A., Harris-Adamson, C., Rempel, D. M., & Nussbaum, M. A. (2019). Potential of Exoskeleton Technologies to Enhance Safety, Health, and Performance in Construction: Industry Perspectives and Future Research Directions. Iise Transactions on Occupational Ergonomics & Human Factors, 7(3-4), 185-191. DOI: 10.1080/24725838.2018.1561557
  27. Liu, M. Z., Xu, X., Hu, J., & Jiang, Q. N. (2022). Real time detection of driver fatigue based on CNN‐LSTM. IET Image Processing, 16(2), 576-595. DOI: 10.1049/ipr2.12373.
  28. Liu, P. K., Chi, H. L., Li, X., & Li, D. S. (2020). Development of a Fatigue Detection and Early Warning System for Crane Operators: A Preliminary Study. Construction Research Congress 2020: Computer Applications, 106-115. DOI: 10.1061/9780784482865.012.
  29. Liu, Y., Li, X. L., Lai, J. R., Zhu, A. B., Zhang, X. D., Zheng, Z. M., Zhu, H. J., Shi, Y. Y., Wang, L., & Chen, Z. Y. (2021). The Effects of a Passive Exoskeleton on Human Thermal Responses in Temperate and Cold Environments. International Journal of Environmental Research and Public Health, 18(8). DOI: 10.3390/ijerph18083889
  30. Mantini, D., Franciotti, R., Romani, G. L., & Pizzella, V. (2008). Improving MEG source localizations: an automated method for complete artifact removal based on independent component analysis. NeuroImage, 40(1), 160-173. DOI: 10.1016/j.neuroimage.2007.11.022.
  31. Marchand, C., De Graaf, J. B., & Jarrasse, N. (2021). Measuring mental workload in assistive wearable devices: a review. Journal of Neuroengineering and Rehabilitation, 18(1). DOI: 10.1186/s12984-021-00953-w
  32. Massardi, S., Pinto-Fernandez, D., Babic, J., Dezman, M., Trost, A., Grosu, V., Lefeber, D., Rodriguez, C., Bessler, J., Schaake, L., Prange-Lasonder, G., Veneman, J. F., & Torricelli, D. (2023). Relevance of hazards in exoskeleton applications: a survey-based enquiry. J Neuroeng Rehabil, 20(1), 68. DOI: 10.1186/s12984-023-01191-y
  33. Mastropietro, A., Pirovano, I., Marciano, A., Porcelli, S., & Rizzo, G. (2023). Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines. Sensors (Basel), 23(3). DOI: 10.3390/s23031367
  34. Mehmood, I., Li, H., Qarout, Y., Umer, W., Anwer, S., Wu, H., Hussain, M., & Antwi-Afari, M. F. (2023). Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 56, 101978. DOI: 10.1016/j.aei.2023.101978.
  35. Missonnier, P., Deiber, M. P., Gold, G., Millet, P., Pun, M. G. F., Fazio-Costa, L., Giannakopoulos, P., & Ibanez, V. (2006). Frontal theta event-related synchronization: comparison of directed attention and working memory load effects. Journal of Neural Transmission, 113(10), 1477-1486. DOI: 10.1007/s00702-005-0443-9
  36. Miyamoto, K., Tanaka, H., & Nakamura, S. (2022). Online EEG-Based Emotion Prediction and Music Generation for Inducing Affective States. Ieice Transactions on Information and Systems, E105d(5), 1050-1063. DOI: 10.1587/transinf.2021EDP7171
  37. Nussbaum, M. A., Lowe, B. D., de Looze, M., Harris-Adamson, C., & Smets, M. (2019). An Introduction to the Special Issue on Occupational Exoskeletons. Iise Transactions on Occupational Ergonomics & Human Factors, 7(3-4), 153-162. DOI: 10.1080/24725838.2019.1709695
  38. Ogunseiju, O., Akinniyi, A., Gonsalves, N., Khalid, M., & Akanmu, A. (2023). Detecting Learning Stages within a Sensor-Based Mixed Reality Learning Environment Using Deep Learning. Journal of Computing in Civil Engineering, 37(4). DOI: 10.1061/JCCEE5.CPENG-5169
  39. Ogunseiju, O., Olayiwola, J., Akanmu, A., & Olatunji, O. A. (2022). Evaluation of postural-assist exoskeleton for manual material handling. Engineering Construction and Architectural Management, 29(3), 1358-1375. DOI: 10.1108/Ecam-07-2020-0491
  40. Poliero, T., Lazzaroni, M., Toxiri, S., Di Natali, C., Caldwell, D. G., & Ortiz, J. (2020). Applicability of an Active Back-Support Exoskeleton to Carrying Activities. Frontiers in Robotics and Ai, 7. DOI: 10.3389/frobt.2020.579963
  41. Qin, Y. M., & Bulbul, T. (2023). Electroencephalogram-based mental workload prediction for using Augmented Reality head mounted display in construction assembly: A deep learning approach. Automation in Construction, 152. DOI: 10.1016/j.autcon.2023.104892
  42. Renaud, O., & Victoria-Feser, M.-P. (2010). A robust coefficient of determination for regression. Journal of Statistical Planning and Inference, 140(7), 1852-1862. DOI: 10.1016/j.jspi.2010.01.008.
  43. Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991-1009. DOI: 10.1016/j.ergon.2005.04.005
  44. Simon, M., Schmidt, E. A., Kincses, W. E., Fritzsche, M., Bruns, A., Aufmuth, C., Bogdan, M., Rosenstiel, W., & Schrauf, M. (2011). EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions. Clinical Neurophysiology, 122(6), 1168-1178. DOI: 10.1016/j.clinph.2010.10.044
  45. Theurel, J., Desbrosses, K., Roux, T., & Savescu, A. (2018). Physiological consequences of using an upper limb exoskeleton during manual handling tasks. Applied Ergonomics, 67, 211-217. DOI: 10.1016/j.apergo.2017.10.008
  46. Wang, F., Xuan, Z., Zhen, Z., Li, K., Wang, T., & Shi, M. (2020). A day-ahead PV power forecasting method based on LSTM-RNN model and time correlation modification under partial daily pattern prediction framework. Energy Conversion and Management, 212, 112766. DOI: 10.1016/j.enconman.2020.112766.
  47. Wei, W., Zha, S., Xia, Y., Gu, J., & Lin, X. (2020). A hip active assisted exoskeleton that assists the semi-squat lifting. Applied Sciences, 10(7), 2424. DOI: 10.3390/app10072424.
  48. Welch, P. (1967). The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics, 15(2), 70-73. DOI: 10.1109/TAU.1967.1161901.
  49. Xing, X., Zhong, B., Luo, H., Rose, T., Li, J., & Antwi-Afari, M. F. (2020). Effects of physical fatigue on the induction of mental fatigue of construction workers: A pilot study based on a neurophysiological approach. Automation in Construction, 120, 103381. DOI: 10.1016/j.autcon.2020.103381.
  50. Yang, Y. Q., Ye, Z. H., Easa, S. M., Feng, Y., & Zheng, X. Y. (2023). Effect of driving distractions on driver mental workload in work zone's warning area. Transportation Research Part F-Traffic Psychology and Behaviour, 95, 112-128. DOI: 10.1016/j.trf.2023.03.018
  51. Young, M. S., Brookhuis, K. A., Wickens, C. D., & Hancock, P. A. (2015). State of Science: Mental Workload in Ergonomics. Ergonomics, 58, 1–17. DOI: 10.1080/00140139.2014.956151.
  52. Zhu, F., Kern, M., Fowkes, E., Afzal, T., Contreras-Vidal, J.-L., Francisco, G. E., & Chang, S.-H. (2021). Effects of an exoskeleton-assisted gait training on post-stroke lower-limb muscle coordination. Journal of Neural Engineering, 18(4), 046039. DOI: 10.1088/1741-2552/abf0d5
PDF
  • Publication Year: 2023
  • Pages: 691-700

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

Predicting Mental Workload of Using Exoskeletons for Construction Work: A Deep Learning Approach

Authors

Abiola Akanmu, Adedeji Afolabi, Akinwale Okunola

DOI

10.36253/979-12-215-0289-3.69

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

203

Fulltext
downloads

109

Views

Export Citation

1,354

Open Access Books

in the Catalogue

2,362

Book Chapters

3,870,371

Fulltext
downloads

4,516

Authors

from 936 Research Institutions

of 66 Nations

66

scientific boards

from 351 Research Institutions

of 43 Nations

1,249

Referees

from 381 Research Institutions

of 38 Nations