Fujitsu Laboratories Ltd. and Fujitsu Research and Development Center Co., Ltd. (FRDC) have developed the "Actlyzer" hand wash movement recognition technology, which leverages AI and machine learning techniques to identify complex hand washing movements from video data captured by camera.

Amidst the ongoing global COVID-19 pandemic, the importance of hand washing as a measure to protect people's health from bacteria, influenza, and other infectious diseases is gaining renewed attention worldwide.

Fujitsu has expanded the recognition function of its existing "Actlyzer behavioral analysis technology", which can recognize a variety of subtle and complex human movements without relying on large amounts of training data. Specifically, Fujitsu has refined recognition capabilities for hand movements to create technology to automatically recognize complicated hand movements performed during hand washing.

This technology makes it possible to easily determine whether someone is following each of the 6 steps for handwashing recommended by the Japanese Ministry of Health, Labour and Welfare, reducing the number of man-hours required for intrusive visual checks by inspectors for on-site sanitation management.

At present, food service industry workers in Japan are required to perform these six steps when washing their hands and confirm that during each step the different parts of their hands have been rubbed more than a certain number of times. To this end, food service providers fill out a check sheet and make a self-report, and workers on-site undergo visual checks by supervisors.

Gesture recognition using deep learning is a common technique for identifying hand and finger movements. This conventional technique can detect multiple feature points, such as joints and fingertips, from an image of the hand, and determine the hand gesture based on the positional relationship of the feature points. However, one issue with the existing technology was that when people wash their hands correctly, both hands overlap and are lathered with soap, which obscures the detection points on the fingers and prevents accurate gesture recognition.

To resolve this challenge, Fujitsu Laboratories Ltd. and FRDC have developed a new AI technology that automatically and accurately recognizes hand movements under the conditions described above, expanding the recognition function of the original Actlyzer behavioral analysis technology.

With the new technology, the complex hand movements of handwashing are captured as a combination of hand shape and repetitive rubbing motions, detected by two deep learning engines: Hand Shape Recognition and Motion Recognition.

The two-hand shape recognition engine uses a learned model of a basic shape of two hands, which is a typical form of hand movement in which hands are placed on top of each other, to determine the handshape for each frame of the image. Focusing on the overall shape solves the problem of when fingertips and joint feature points cannot be correctly detected due to hand overlap or foam. In addition, Fujitsu's unique AI technology "High Durability Learning (High Durability Learning)", which can track data changes, is applied to ensure that the basic shape of both hands is recognized with high accuracy even when the camera position or lighting changes during operation.

The motion recognition engine uses a learned model that detects periodically changing motion from successive frames and counts the number of iterations as the number of rubs from the iteration pattern and its period. In addition, the results of these two recognition engines are fed back to each other to improve recognition accuracy. The motion recognition engine sets a threshold value for the magnitude of the motion to be judged in accordance with the steps recognized by the two-hand shape recognition engine to prevent detection of erroneous periods, such as hand tremors not related to foam motion or rubbing. The two-hand shape recognition engine improves detection accuracy by filtering the judgment result using the repetition pattern period detected by the motion recognition engine.

A hand-wash video data set with approximately 2000 variations; including people, camera positions, and soap types, was independently filmed and collected for learning and evaluation. It was confirmed that the accuracy of 6 steps of correct hand washing is average of 95% or more, and that the accuracy of the number of hands rubbing movements is 90% or more. When operating the system on-site, the omission of action could be prevented because the person washing their hands always knows how long to scrub their hands and which of the six stages they've completed because the system determines it's displayed on the screen until it's completed. In addition to this, the system automatically records data with the starting timestamp and length of action of completing each six steps, and the number of times of rubbing.

This technology automates on-site handwashing checks for workplaces that require strict hygiene control, eliminating the number of man-hours needed for visual confirmation and manual recording. Additionally, because the system doesn't recognize incorrect or incomplete hand washing, Fujitsu expects the solution will help educate users and ensure standardization for the right way of handwashing.