SECERN AI

FaceBuilder is a sophisticated 3D face modeling tool developed by Keentools that allows precise manipulation of 3D facial meshes using a pin-based deformation system. The tool supports interactive mesh editing across multiple views with synchronized 3D consistency.

• Python 3.10

First you need to clone the repo:

git clone https://github.com/CUBOX-Co-Ltd/facebuilder.git
cd facebuilder

Our default installation method is based on Conda package and environment management:

conda create -n facebuilder python==3.10
conda activate facebuilder
pip install -r requirements.txt
conda install -c conda-forge dlib # For dlib face landmark detector
pip uninstall numpy
pip install numpy==1.23.5 # Downgrade numpy to avoid error: cannot import name 'bool' from 'numpy'

You can run the application with:

python main.py

• Able to deform the mesh using multiview input images:
  • A single shared 3D mesh maintains consistency between all input images

• Detection: Uses both MediaPipe and dlib for robust 2D face detection in various poses
• Multi-stage Alignment:
  • Detects face landmarks automatically in the current image
  • Performs 3D pose estimation to calculate optimal rotation and translation
  • Uses PnP (Perspective-n-Point) algorithm to align the 3D mesh with 2D landmarks

FaceBuilder implements an intuitive pin-based interaction system with three specialized manipulation modes:

• Rotation Mechanism: Mouse movements (dx, dy) are converted to rotation elements
  • Horizontal movement (dx) → Y-axis rotation
  • Vertical movement (dy) → X-axis rotation
  • These rotation values are combined with the existing rotation matrix for natural rotation
• The entire mesh follows the mouse position precisely via calculated translation values
• Ray casting techniques ensure 2D mouse input accurately reflects on the 3D mesh
• Face structure maintains integrity while rotating naturally according to mouse movement

• Scaling Behavior:
  • Pulling pins outward → Enlarges the mesh
  • Pulling pins inward → Reduces the mesh
• Non-dragged pin remains fixed in both mesh and image space
• All rotation and scaling operations use the fixed pin as a reference point
• Allows precise control of mesh size while maintaining proportions

• 3D Rotation Control: Dragging the third pin while using two fixed pins as anchor points
• Uses PnP algorithm to calculate accurate 3D rotation and translation
• Maintains proportions while enabling precise mesh control
• Two non-dragged pins remain exactly fixed in their original positions