Abstract

PyAFAR is a Python-based, open-source facial action unit detection library for use with adults and infants. Convolutional Neural Networks were trained on BP4D+ for adults, and Histogram of Gradients (HoG) features from MIAMI and CLOCK databases for infants were used with Light Gradient Boosting Machines. In adults, Action Unit occurrence and intensity detection are enabled for 12 AUs. The 12 AUs were selected on the criterion that they occurred more than 5% of the time in the training data that included BP4D. Because 5% baseline was the minimum for which reliability could be measured with confidence, AU with prevalence lower than that were not included. Action unit intensity estimation is enabled for 5 of these AU. In infants, AU occurrence is enabled for 9 action units that are involved in expression of positive and negative affect. For both adults and infants, facial landmark and head pose tracking are enabled as well. For adults, multiple persons within a video may be tracked. The library is developed for ease of use. The models are available for fine-tuning and further training. PyAFAR may be easily incorporated into user Python code.

Pipeline of PyAFAR.

PyAFAR has separate modules for facial feature extraction and tracking, Face normalization and AU predictions:

It uses MediaPipe library for landmarks. Tracking is performed using the FaceNet. The Perspective-n-Point (PnP) method is used to predict Roll, Pitch and Yaw.

The landmark predictions are used to normalize faces using the dlib library.

Normalized faces are used for AU predictions (occurrence and intensity). Separate detection modules for occurrence are available for adults and infants. Intensity predictions are available for adults only.

PyAFAR can output frame-level predictions in CSV and JSON formats to enable easy reading with most platforms used by both computational as well as domain experts.