Blog post

How to Apply Kalman Filter for Pedestrian Recognition

Learn what the Kalman filter is and how it helps in pedestrian recognition for safer autonomous driving

Updated on January 18, 2022

7 mins read

Intelligent, self-guided, and self-driving vehicle systems are closer than you think. They may not be roaming the streets just yet, but the pace of technological advancement is pushing the advent of self-driving cars closer. The automatic system utilizes computer vision algorithms to detect and predict activity for all other traffic participants, including other vehicles, pedestrian recognition or even wild animals.

For a self-driving car, pedestrian detection is a vital feature to ensure safety. So, the main goal of the machine learning approach in the automotive domain is the possibility to predict dangerous situations on the road ahead of time. A critical benefit the system provides to the cars is distinguishing pedestrians and vehicles in motion so that they make smarter and safer driving decisions.

In this article, we go through the machine learning development services and methodologies used in autonomous vehicles, such as object detection, object segmentation and prediction to track their surroundings properly, pedestrian detection, predict object locations and possible collisions. We do not only explain how real-time pedestrian detection works in theory but also display the practical implementation of the methodologies in self-driving cars:

Real-time pedestrian presence detection with neural networks

Object tracking and its velocity prediction is a well-researched area outside the automotive industry. These are mostly conducted using computer vision and machine learning approaches.

Pedestrian recognition components

Considering the effort to build pedestrian tracking software, the research focuses on several modules (see fig. 1.):

  • a pedestrian detector
  • a pedestrian tracker and location predictor
  • a road segmentation module
  • a pedestrian collision prediction module itself

Now, let’s discuss each pedestrian location predictor in details.

Pedestrian detector

Computer vision approaches based on neural network solutions allow detecting a variety of objects with different accuracy. Most popular neural network topologies for vehicle and pedestrian detection are GoogleNet, SSD, MobileNet and YoLo, with the last one being used to build media for this article.

All these are state-of-the-art solutions for pedestrian detection in cars. But they can detect pedestrians in a single image only. Therefore, using neural networks to establish tracking and build its trajectory is not enough: additional solutions must be implemented to link the same pedestrian in two consequence images from a video sequence.

Learn how Intellias built an innovative solution to recognize drivers’ handwritten text

Read more

Pedestrian tracker and location predictor

In the 1960s, a set of algorithms was developed to enable motion tracking. Kalman filter and its variations were used to model Apollo spacecraft trajectories to the Moon and back. Today, these algorithms can be applied in an autonomous vehicle, too, to predict the position and velocity of approaching pedestrians. Let’s discover the specifics of the Kalman filter for pedestrian detection.

Road segmentation module

A front camera view handles the road recognition part in a self-driving car. A neural network labels the pixels of a road in images, usually by Fully Convolutional Neural Network (FCN), particularly FCN-VGG16, trained with KITTI or CityScape dataset for road segmentation. This article is based on the mentioned neural network as well.

Pedestrian collision prediction module

This module analyzes the intersection of pedestrian location prediction and the road segment. If these two objects intersect, the car and pedestrian collision has a non-zero probability, and it is a zero one in the opposite case. The collision probability depends on the direction vectors of both pedestrians and the car.

General scheme of pedestrian recognition
How to Apply Kalman Filter for Pedestrian Recognition

Learn how Intellias helped to build and support a comprehensive monitoring system for car diagnostics and real-time notifications to drivers

Read more

Other approaches in pedestrian detection for autonomous vehicle

While the neural network approaches for object detection and road segmentation are well-described in many scientific articles, we would like to focus on two more components in this article: Kalman Filter for pedestrian detection and Pedestrian collision prediction module.

Kalman Filter

Kalman Filter (KF) is a core component of pedestrian detection. KF implements a series of  predictions and  state updates which are executed as the single dataflow displayed next (fig. 2.):

General scheme of the Kalman filter
How to Apply Kalman Filter for Pedestrian Recognition

Let’s dive deeper into more details of how the Kalman filter works with the linear motion model of the tracking object, using visual image data only:

1. Prediction step

The current state Xt+1 is estimated based on the previous state Xt using motion model F (transition state matrix) and process noise v:
How to Apply Kalman Filter for Pedestrian Recognition

The above equation for the new state contains information about one’s position Pt and velocity Vt and can be written in a matrix form:
How to Apply Kalman Filter for Pedestrian Recognition

This is equivalent to the following:
How to Apply Kalman Filter for Pedestrian Recognition

The state estimation can be described as a vector:
How to Apply Kalman Filter for Pedestrian Recognition

We neglect the process noise v for simplification, and we also should calculate covariance matrix Pt+1 – a measure of the estimated state accuracy – based on the covariance matrix Pt from the previous step of the Kalman filter:
How to Apply Kalman Filter for Pedestrian Recognition

2. The update step

We are to obtain the new measurements z of the bound boxes from the pedestrian detection module and calculating measurement update y:
How to Apply Kalman Filter for Pedestrian Recognition
H is a transformation matrix (or measurement matrix), intended to map object detection data space into our model space.

The next step is calculating the Kalman gain (K):
How to Apply Kalman Filter for Pedestrian Recognition
The Kalman gain basically shows what to rely on, either estimated state or detection data:
How to Apply Kalman Filter for Pedestrian Recognition
If the Kalman gain is a bigger number, sensor data will be more impactful on the calculation of the new estimated state. If it is a smaller number, we have to rely on the mathematical model of the estimated state.

The output of the Kalman filter is a fine-tuned trajectory of pedestrian direction. The prediction of pedestrian location for an extended period can be implemented by simply repeating step #1. Evidently, a long-lasting prediction will have less accuracy than a shorter one, with each new prediction step of the Kalman filter adding more uncertainty to the predicted data.

AI & Machine Learning in Automotive

Train your vehicle with advanced AI and ML algorithms for safe driving

Learn more

Pedestrian collision prediction module

The general approach to pedestrian detection in cars also includes pedestrian collision predictor. The module obtains raw data from the pedestrian detector, prediction data from the Kalman filter and data from the road segmentation module.

The purpose of the module is to combine all the information and analyze the interposition of a pedestrian, his/her predicted location and the location on the road. The output of the model is a probability of a car-pedestrian collision.

The following is a brief overview the module’s workflow:

1) Pedestrian step distance calculation

In the real-world conditions, information on the distance between the car and the pedestrian, along with the car speed, can be easily obtained from the vehicle’s ADAS computer. Pedestrian recognition using automotive RADAR sensor or LIDAR hardware handles distance measurement to the obstacles.

However, in this case, the information was absent, and distance d was withdrawn from geometrical calculations based on the pedestrian box’s bottom line and camera view angle in the following way (fig. 3):
Distance to the pedestrian
How to Apply Kalman Filter for Pedestrian Recognition
How to Apply Kalman Filter for Pedestrian Recognition
where Bh– pedestrian bound box height (predicted or detected) in pixels,
ih– image height in pixels,
C– constant calculated manually for a specific car camera (data obtained for a specific test image dataset).

For test experiment it is not important to define the absolute value of distance, it is more important to calculate the distance metric to understand how far pedestrian stands from the vehicle.

2) Collision prediction step

To calculate the collision probability, the information on the bottom coordinates of pedestrian bound boxes is required, which is predicted and obtained from the object detector and Kalman filter. The main idea of this step is to check the intersection of a road segment and box’s bottom line.

In case of an intersection, a pedestrian (or pedestrian predicted location) stands on the vehicle path, and the collision may happen.

To simplify, the collision probability was calculated as the ratio of distance to the predicted pedestrian location and general distance of detected road available for the car:
How to Apply Kalman Filter for Pedestrian Recognition

where IoU– a function showing the intersection over the union of road segment and the pedestrian predicted location bound box,
bh– pedestrian bound box height (predicted or detected) in pixels,
rh– road segment max height in pixels.

An example of the algorithm implementation is in pic.2, with the results of automatic road segmentation highlighted in green. There are confidence members for each bound box: blue bound boxes for pedestrian detection confidence and green/red bound boxes for collision probability. The box color also defines the collision probability: red – high probability, green – low probability.

The final output of pedestrian collision detector approach (blue boxes – detections, green boxes – location predictions)
How to Apply Kalman Filter for Pedestrian Recognition


Computer vision problem requires a solution for automotive driving, particularly pedestrian detection and tracking with moving camera. And such a solution is possible: convolutional neural network allows for robust pedestrian tracking. However, an additional challenge for advanced automotive driving is the pedestrian location prediction algorithm, which can be based on pure machine learning techniques.

In this article, we have described the main components of pedestrian detection for autonomous vehicle and reviewed the pedestrian location prediction algorithm in-depth, which is based on the Kalman filter. The result we have achieved show that the listed mathematical algorithms allow building a single approach to real-time pedestrian detection and avoid dangerous road situations in self-driving cars.

Contact our automotive experts at Intellias to learn more about the Kalman filter application for pedestrian recognition and the mathematical algorithms in autonomous vehicles.

Your subscription is confirmed.
Thank you for being with us.

4.9 Thank you for your vote. 19865 95d2a728fe

Tell us about your project

I give consent to the processing of my personal data given in the contact form above under the terms and conditions of Intellias Privacy Policy. I want to receive commercial communications and marketing information from Intellias by electronic means of communication (including telephone and e-mail).
* I give consent to the processing of my personal data given in the contact form above under the terms and conditions of Intellias Privacy Policy.

iso-27001 (1)
iso-2001-2015 (1)

Thank you for your message.
We will get back to you shortly.

Thank you for your message.
We will get back to you shortly.