AUTOMATIC 3D FACE PRINTING By: David Gelbendorf, Hila Ben-Moshe Supervisor: Alon Zvirin 19.10.2015
Jan 19, 2016
AUTOMATIC 3D FACE PRINTING
By:David Gelbendorf,
Hila Ben-MosheSupervisor: Alon Zvirin
19.10.2015
Face Printing?
- GIP sensor: Today we are capable to capture 3D images
- We are able to print in 3D, with the right printer
- Everyone has a passport photo on he’s I.D card
- Why not printing a face in 3D?
Face Printing!
Face Printing - Goal
First Stage: Developing a tool for processing a facial video
and printing a 3D face automatically Input: a 3D record of a frontal face Output: a 3D printed model of the face
Second Stage: Wrap the tool in a easy-to-use GUI
Face Printing – The challenge
- Printing a 3D object requires:- A good 3D image, which contains all face features (eyes, nose and mouth) and with high density
Face Printing – The challenge
- Printing a 3D object requires:- Smoothness and fullness
- The 3D frames are not complete. Many pixels has NAN values and missing data (non-Continuity)
- Choosing bounding box
Face Printing – The challenge
- Printing a 3D object requires:- Transformation of the 3D GIP file to a 3D printer standard format (for instance obj, stl)
Face Printing – Block Diagram
Recoding a video
Choosing Best
Frame
Processing frame (creating model)
Converting to STL
Model ready to
Face Printing – A good Image
- How to choose a good frame? - Solution: Use Viola Jones face
detection algorithm:- Use VJ for searching facial features- Use VJ for detecting movement of facial
features
Choosing Best Frame
Face Printing – A good Image
Profile Face (no detections) True positives + False Positives (double detection of mouth and left eye)
Semi-Profile Face with low detection results
Choosing Best Frame
Detected face frame: 216Detected face frame: 84Detected face frame: 174
Face Printing – A good Image
Double detection of mouth,no detection of left/right eye
No detection of pair of eyes Perfect match (one match for each detector)
Choosing Best Frame
Detected face frame: 78Detected face frame: 47Detected face frame: 111
Face Printing – A good Image
Facial features detection: - Scanning each frame for a face, pair of
eyes, left eye, right eye, nose and mouth- Sorted by importance :
- 2: Face and pair of eyes:- 1: left eye, right eye, *nose- 0.5: Mouth- *0.75: if found more than one nose, there will be penalty
- *0.5: if found more than 2 pair of eyes- Grade: normalized to [0..1]
Choosing Best Frame
Face Printing – A good Image
Movement Detection: - Examining difference of VJ detections in adjacent frames (previous and next frames) :- Pair of eyes, left eye, mouth, nose
- Calculating the distance of adjacent detections frames (Euclidean metric)
- Calculating a grade according to the results. A perfect grade (1.00) will be achieved only if perfect match is found between previous and next frames in all examined features.
Choosing Best Frame
Face Printing – A good Image
Movement Detection: - Examining difference of VJ detections in adjacent
frames (previous and next frames) for 4 detectors: Pair of eyes, left eye, mouth, nose
- Calculating the distance of adjacent detections frames (Euclidean metric):- Let - Let - The delta distance between adjacent frames is
calculated as follows:
- The total movement grade is calculated according to d (bigger d results with lower grade)
Choosing Best Frame
Face Printing – A good Image
- The final grade combined from both methods is calculated as:
- *VJDetecionGrade + (1-)*movementGrade- Testing different values has proven to be
most effective- The grading m
- is a parameter and therefore can be changed- Other grading methods could be added or
changed easily
- The chosen frame of the video would be the one with best score.
Choosing Best Frame
Face Printing – A good Image
- Grades examples:
Choosing Best Frame
0.910 0 0.710.82
Face Printing – Image Processing
The chosen image is a raw 3d image, contains noise, holes and discontinuities.
Goal: automatic filtering of the image in order to create a 3D complete, continuous model
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Challenge: choosing a right bounding box for the desired model (for instance, without background)
Solution: choosing the bounding box automatically with optimization
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Bad Bounding Box (Noise is cut. Also, in the back of the head there is too much information)
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Good Bounding Box:
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Optimization of bounding box: Specially important for Z axis (depth) Iterative “exhaustive search” for optimal max
Z: Starting from a min size of max Z, and
progressing with max of Z axis Dividing the min – max range into levels,
increasing the max Z value in each iteration Calculating a grade for each max Z value Searching for minimum grade
Second option: run constrained optimization to find optimal min and max value for x, y, z (6 variables)Choosing Best Frame Processing Frame
Face Printing – Image Processing
Searching for minimum Weighted grade according to: Length of Contour 2D: 0.2 Length of Contour 3D: 0.2 Number of pixels: 0.3 Quadratic Shape of Contour: 0.7
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Choosing Best Frame Processing Frame
-0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
maxZ
grad
e
num pixels
quadratic rms
2D curve length3D curve length
wighted grade
Face Printing – Image Processing
Filtering: 3D filter:
A chosen filter (median, average, raw, beltrami) Clean Margins Remove pixels far from the median of their
neighbors Remove pixels with few neighbors Choose K largest connectivity components
Filling Holes: cubic interpolation of each 2D connected
component
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Problem: Dark eyebrows: risk for missing data for
building the model
Solution: Detect and fix holes in eyebrows:
Detect eyebrows area with Viola Jones algorithm (by detecting pair of eyes position).
If holes detected, try to fix by moderate filtering.
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows - Detection: Let (rectangle of pair of eyes position. Let (w,h) be the corresponding width and
height. The eyebrows position is a shifting and
enlargement transform as follows:
Missing pixels fraction of eyebrows area is calculated (on mask of valid pixels): If fraction > 8% than try to fix eyebrows.
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows – Examples (Detect): Example 1: eyebrows detection returns true negative
(0% missing pixels):
Example 2: eyebrows detection returns true positive (28% missing pixels):
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows –Repair: Trying to moderate image filtering
parameters: Changing threshold of minimal neighbors from 4
to 2. Iteratively compute the distance threshold
between a pixel and it’s median with is neighbors. Increasing threshold until fraction of missing data <8% or after reaching maximum threshold value.
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows – Examples (Detect and fix): Example 1: eyebrows detection returns true negative
(0% missing pixels):
Example 2: eyebrows detection returns true positive (28% missing pixels):
After fixing,missing pixelsfraction decreasesto 5.7%
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows – Examples: Example 3: Final Model with and without fixing
Choosing Best Frame Processing Frame
Face Printing – Image Processing
Fixing Eyebrows – drawbacks: If Viola Jones algorithm fails to detect
pair of eyes detection, eyebrows detection will fail as well.
Moderate filtering may result with a distorted model.
Choosing Best Frame Processing Frame
Face Printing – Image Processing
After bounding box optimization and filtering: The 3D model is prepared The Contour of the model is prepared
Choosing Best Frame Processing Frame
Face Printing – Writing to STL
Face model is ready, we can transform the model into 3D STL format which is a known standard for 3D models In the writing process, the 3D model’s points
are re-arranged in triangles and their corresponding normals.
Choosing Best Frame Processing Frame Writing STL file
Face Printing Application– Drawbacks
Automatic bounding box may result in a cut face
Automatic filtering process (which is not configured to each face individually) may result with a less accurate model, compared to a model which was filtered manually.
Run-time of the all automatic modules may take a couple of minutes
Face Printing – future improvements
Improving run-time of bottlenecks : Run-time for calculating grade for each frame Run-time for calculating bounding box Run-time for writing STL modules
Improving filtering by adjusting parameters for each frame (change some parameters to be dynamic)
Building a better model from several different frames
Face Printing - GUI
Face Printing - GUI
All option can be easily operated from the GUI: Choosing a video First Option:
Getting a 3D model to print automatically Second Option:
Edit parameters for each module:- Choose best frame- Choose bounding box (option for manual or
automatic bounding box)- Choose file format and save the model to print
Advanced menu to change several parameters
Bibliography
Viola Jones in Matlab: http://www.mathworks.com/help/vision/ref/
vision.cascadeobjectdetector-class.html