Explainable fatigue detection in assembly tasks through graph neural networks

Fatigue during assembly tasks can have a negative effect on subjective as well as objective quality of work. We recorded a novel dataset for the purpose of detecting fatigue in  assembly scenarios. Participants were instructed to assemble and disassemble model cars with the help of a robot arm. The  recordings consist of video, depth video, EEG and eye  racking data as well as questionnaires on the participants’ fatigue. The dataset can be provided to researchers on demand. In addition  to recording a dataset, we implemented a proof of  concept system to detect fatigue solely on image data. In our approach the eye tracking data was used to label the participants’ fatigue. Afterwards,  graph neural network was trained  on poses extracted from the video data and the generated labels. The classifications of the model are made transparent through the use of explainable AI using saliency maps and  GradCAM. This work can have a positive impact on human-machine interaction and assistance systems. Through explainability, we aim to increase the acceptance of such systems by  workers and industries.