Skip to main content


Multimodal & Cross-Spectral Biometrics

Multimodal biometric systems have the ability to integrate the evidence presented by multiple sources of information.

Multimodal biometric systems are now the best suited solution for any industry where high accuracy and security is required because they require multiple biometric credentials for positive identification instead of one as in a unimodal system. Furthermore, in many military and law enforcement scenarios where a probe face image is only available in an alternate sensing modality.

This identification task, where the probe and gallery images are from different modalities, is called heterogeneous face recognition (HFR). Examples of HFR are cross-age, sketch-to-face, cross-makeup, look-alikes and cross-spectral face recognition and synthesis. 

Due to the different physical phenomenology of different spectral bands face images captured by spectral sensors will carry different information about the physical face. Multispectral face recognition is exploits these spectral bands to improve face recognition. BIIC researchers are developing within-spectral and cross-spectral matching techniques while using visible, SWIR and thermal face images. They are also working on developing eye detection techniques that can operate when using face images captured in any spectral band.  Six different projects that have been completed by BIIC researchers are described below:

Robust Multimodal Biometrics Recognition

In BIRC, novel multimodal techniques have  developed that exploit simultaneously and   collaboratively  the  coupling information among all biometrics modalities to make a decision. Sparse theory and dictionary learning techniques have been exploited in this research.

Cross-age Face Recognition Based on a Facial Age Estimation Scheme: 

Face recognition across age has not received adequate attention until recently. Facial aging can degrade the face recognition performance drastically. A typical approach to cross-age face recognition is to synthesize new faces at all possible ages, which is slow and inefficient especially when working on a large-scale face recognition database.

More recently, human age estimation has become an active research topic. However, the use of age estimation algorithms to enhance cross-age face recognition has not been explored. In this project, we bridged the gap by investigating how age estimation can help cross-age face recognition.

The basic idea is to estimate the age of a test face, and to synthesize a face image only at the estimated age for each individual in the database rather than generating face images at all possible ages. If this method is successful, it will make cross-age face recognition system more efficient and practical.  

This project involved four steps as shown in Fig. 2.  First, we refined and improved our previous method to obtain a high performance age estimator. We then used the Yamaha database for age estimation, and further improve and verified the method on the MORPH database.

Second, we used the estimated age to guide the face synthesis module to render only a limited number of face images.

Third, we performed face recognition using our methods for age estimation and face synthesis. The MORPH database will was used for evaluation, which contains more than 55,000 face images. 

Finally, we investigated the effect of image quality on age estimation and cross-age face recognition by assessing the system performance when encountering face images at different quality levels.

Cross-age framework
Figure 2. Cross-age Face Recognition Framework

A Pre-Processing Methodology for Handling Passport Facial Photos:

In many security applications, the problem of matching facial images that are severely degraded remains to be a challenge. Typical sources of image degradation include low illumination conditions, image compression, out-of-focus acquisition etc.

Another type of degradation that received very little attention in the face recognition literature is the presence of security watermarks on documents (e.g. passports). Although preliminary work in the area has mitigated certain challenges (removal of noise present on documents) the image restoration part requires more attention to further overcome missing information from face images due to image de-noising. This project examined the effects of a preprocessing methodology that mitigates the effects of security watermarks on passport facial photos in order to improve image quality and face recognition overall.

The types of images that were investigated are face images from passport photos. The proposed project focused on answering the following questions: (1) how do original passport face photos affect recognition performance? (2) Which pre-processing algorithms affect recognition performance the most? (3) What are the optimal conditions that FR is feasible under different levels of pre-processing using our novel algorithm?

Partial Face Matching Across the Infrared Band: 

Recent interest in heterogeneous biometric recognition is motivated by the ability of active infrared (IR) cameras to “see” at night, through fog, rain and under other challenging conditions. Matching partial heterogeneous face images to a gallery of visible images is a special case of heterogeneous biometric recognition. It is a challenging open research problem that is also well justified by many practical cases of face recognition, e.g., where face images of non-cooperative subjects are captured under difficult environments including variable distances and illumination conditions.

Our team has previously developed reliable solutions to heterogeneous matching of near frontal ( + or - 20 degrees) face images. In this proposal, we focused on further exploring the possibility of matching parts of faces captured at different bands and conditions.

Three main tasks were implemented in our experimental plan:

  1. Applied existing and developed new pre-processing techniques to extract partial facial information in multiple scenarios from existing NIR, SWIR and LWIR (variable ranges) image datasets, see Fig. 4. State-of- the-art low-rank based image restoration and super-resolution techniques previously developed by our team was adapted to pre-process IR imagery with poor qualities.
  2. Developed techniques to enhance face image quality and perform matching of partial heterogeneous face images by involving state-of-the-art algorithms for heterogeneous face recognition.
  3. Tested and presented performance evaluation results using different matching algorithms. Simulations included multiple scenarios and the following questions were answered, i.e. “which part of a face is most informative for partial heterogeneous face recognition?” and “what performance can be achieved with and without informative parts of face images?”
Figure 4. Shows examples of imagery at different spectrum.

Benefits of Cross-spectral (visible, NIR and LWIR) and Cross-Distance Face Recognition in Non-Ideal Environments

Objectives: Standard face recognition (FR) systems compare new facial images or probes with gallery pictures to establish identity. They typically perform well in the visible band, when lighting is good and cooperative subjects are close to the camera and without facial expression. However, many law enforcement and military applications deal with mixed FR scenarios that involve matching active (0.9 - 2.5 microns) and passive (3-5 or 7-14 microns) infrared (IR) probe images against all images (e.g. mug shots) in a visible gallery database. This is also known as the heterogeneous problem.

While there are studies reported in the literature where either NIR or thermal images are matched against visible ones, there is no study reported where all face datasets available are simultaneously collected (i) in three different bands, covering the visible, active and passive IR bands, and (ii) at different standoff distances. In this project, we have utilized an existing multi-spectral, multi-distance dataset to investigate the benefits of cross-spectral, cross-distance, and cross-expression face recognition.

This dataset is basically one sub-dataset collected as part of the Clarkson’s ‘Unconstrained Biometrics at a Distance’ project, namely the ‘Multi-spectral Face Dataset’. In particular, the dataset contains visible, NIR and LWIR face data captured under ideal 25ft and 35ft standoff distances. In this project as shown in Fig. 5, we investigated answering the following questions:

  1. Can we efficiently match long wave infrared (LWIR) and Near Infrared (NIR) face images to their visible counterparts?
  2. Can we repeat (1) when standoff distances varies (i.e. cross-spectral, cross-distance)?
  3. Can we repeat (1 and 2) when facial expression varies?

Towards Low Cost, Deployable Thermal Biometrics

In thermal-based face recognition investigations, the selection of infrared (IR) detectors is frequently critical in producing key trace evidence for the successful solution of human identification problems. The two detector technologies most commonly used for face examination currently are cooled (Photonic IR) and uncooled (e.g., microbolometers).

Acquisition of thermal face imprints is used as the first step in the analytical process of identification and comparison of thermal imaging characteristics of human faces (e.g., subcutaneous face patterns etc.). The main issues in the acquisition of face images regarding the usage of IR detectors include the following facts:

  1. Cost-Size-Deplorability: the usage of high sensitivity detectors (photonic IR) with low noise which results in high image quality; but this requires cooling the detectors (which makes direct photon detection) and, thus, the complexity as well as the size and inability for easy deployment of such detectors increases.
  2. Temperature Calculation: certain IR cameras (mainly the uncooled ones) do not have built-in software that allow the users to focus on specific areas of the Focal Plane Array (FPA) and calculate the temperature.
  3. Variable Field-of-View (FOV) Optics: the selection of camera components is critical in producing accurate readings and it is highly dependent on the experience of the camera user.

In this work, we extended the capabilities of an existing uncooled system, by determining which camera lenses can be used to improve the image quality of the uncooled detector, and build software to convert gray level values to temperature for each pixel within the FPA.

An evaluation study was done that determined the temperature readings are statistically similar to those acquired when using a high-end uncooled detector that was used as the ground truth data. To enhance the capabilities of current uncooled thermal imaging systems in terms of passively collecting subcutaneous features from human faces, the following tasks were conducted:

  1. designed a GUI to acquire imaging measurements from a subject's face using an uncooled thermal camera.
  2. Investigated which camera lenses can be used to improve the image quality of the uncooled thermal camera readings when operating in a controlled lab environment.
  3. Designed methods to study the effect of the camera's dynamic range (in terms of bit resolution) to the accuracy of gray level (or intensity) values to temperature conversion.
  4. Performed an evaluation study to determine that the converted temperature readings (when using the uncooled system) are statistically similar to those acquired when using a high-end uncooled thermal imaging system that delivers temperature per pixel values (ground truth data).