Skip to main content


Video Analytics

BIIC researchers are developing computer vision algorithms that automatically analyzes video clips to detect and determine temporal and spatial events. Video analytics algorithms are developed for wide range of applications such as entertainment, health-care, retail, automotive, transport, home automation, safety, law enforcement, surveillance and security.

At BIIC researchers are developing algorithms to detect pedestrians, activity recognition and predication. Three typical projects are described below:

Detecting and Tracking Facial and Sub-Facial Regions in Thermal Image Sequences

The first goal of this project is to explore a new tracking paradigm. We will utilize the face tracking capabilities of an existing visible-based system (MS Kinect) to accurately perform face tracking on data captured from a low cost thermal-based camera system, before recognition (using full or partial faces) is performed.

The second goal of this work is to test whether we can develop the necessary software platform to allow simultaneous face tracking when using two thermal sensors (placed at different angles to the face) assisted by two Kinect sensors.

The third and final goal is to investigate the feasibility of extracting and using full frontal face images captured by the low quality thermal sensors to perform FR. The results will be compared to those computed when applying the same FR algorithms to full frontal face images captured by high quality thermal sensors.

The focus of this project will be on the design and development of a software tool, which can interface with a Kinect and a low cost thermal sensor, to achieve accurate tracking of facial and sub-facial regions in the thermal band before FR is applied. In this regard, the following tasks will be conducted:

Video Analytics: Q-Fire: Waiting in line and passing objects.

Simultaneous Recognition of Humans and Their Actions

Human recognition is important in many law enforcement applications. Significant progress has been made in human identification and verification in the past two decades; however, human identification is still a challenging problem, especially when operating in an unconstrained environment with non-cooperative users. In many security and surveillance scenarios, individuals are observed to be performing various actions, rather than standing still. So identifying humans in action is a typical scenario in non-cooperative biometrics.

In this project, we will design algorithms that can not only identify individuals but also determine their corresponding actions in a video. An advantage of this research is that the biometric system can process a query involving both identity and action (e.g., retrieving all videos in a database containing the waving action of a particular individual), rather than the identity-only based query supported by traditional systems.

In order to develop a useful system to identify people performing an action, our approach will have several components:

  1. Extracting discriminative features and exploring efficient learning algorithms for action analysis. We will design and implement algorithms that can perform action recognition (e.g., running, picking up, moving box, opening door, telephoning, etc.) in diverse scenarios.

    To systematically study this problem, we assembled a video database consisting of about 30 actions of more than 30 individuals as observed in three spectra: visible, NIR and thermal IR. We will use this new database to develop new algorithms for action recognition;

  2. Human recognition in action videos. While video-based face recognition has been studied for a while, few of them target human recognition from action videos. We will explore the kinds of features that can be extracted for person identification from action videos. Our plan is to investigate the use of both biometric traits including face, and soft biometric traits, such as gender, age, height, etc.
  3. Human-action pair based search and retrieval. We will design a video retrieval system that locates relevant videos based on biometric-action queries. Thus, the query will include the image of a person (e.g., face image) as well as an action (e.g., “running”). We will systematically quantify the performance of the retrieval system.

Noncooperative Human Identification by Face and Clothing

Searching for a person in a large video archive, possibly made of videos acquired by a network of surveillance cameras, is a fundamental task because it allows tracing down when and where a person was present in the scene. For instance, it allows saving hours of manual video inspection to find and trace the presence of criminals, like those in the Boston Marathon bombings. In such a scenario, quite often the human identification task can only rely on biometric traits acquired in unconstrained conditions from noncooperative subjects. More specifically, this means that both gallery and probe images of someone’s trait may be heavily corrupted by noise or other nuisance factors, such as the pose of an individual, or the illumination and occlusions.

In such hostile conditions, human identification is typically attempted via face recognition (if enough image resolution is provided). Although it has improved significantly even with corrupted probe and gallery images, there are still plenty of unfriendly scenarios where face recognition lack robustness. Hence, there is a critical need to reduce those cases in order to increase the effectiveness of human identification for searching unconstrained video archives.

The main goal of this project is to improve human identification by fusing the face modality with a pseudo-modality such as the clothing appearance of individuals. Using multiple biometric modalities is an effective approach for increasing identification robustness. Besides face, other viable options for the considered scenario include gait, and clothing appearance.

The extraction and characterization of human gait is not practical, unless specific conditions are met, which are typically too restrictive. The clothing appearance is not a biometric trait. However, its effectiveness in matching the identity of people that between sightings have not changed their clothes has been demonstrated, which is why it was chosen.

The objectives of this research include: