Automatic License Plate Recognition (ALPR) system (Figure 1) is one of the most popular video analytics applications for smart cities. The system can be deployed on highways, toll booths, and parking lots to enable fast vehicle identification, congestion control, vehicle counting, law enforcement control, automatic fare collection, and more.

With a powerful edge artificial intelligence (AI) processor, ALPR can be deployed on edge devices and run in real-time, which is crucial for:

* Improving product miss-rates with better performing neural network (NN) that are more resilient to a wide range of scenarios.

* Improving overall detection latency.

* Lowering the overall total cost of ownership (TCO) compared to existing systems, including installation and maintenance costs.

* Increased data protection and improved privacy by eliminating the need to send raw video.

The high compute power introduced by Hailo-8™ also enables for processing of several vehicles concurrently, remotely, and with high accuracy, as opposed to traditional object detection networks, where accuracy tends to decrease by a factor of x5 for small objects (Figure 2). For instance, a vehicle from 100 meters will occupy only a few hundred pixels in full high-definition (FHD) frame, which behooves high-capacity NN.

Application Pipeline

The Hailo ALPR application pipeline – depicted in the following diagram – includes three NN that run on the Hailo-8™ device (Figure 3). It 1) detects the vehicles, 2) detects the license plates, and 3) recognizes the license plate characters (LPRNet). The entire pipeline runs with the GStreamer framework.

To optimize the application latency, Hailo used the Hailo GStreamer Tracker, which avoids running unnecessary calculations on vehicles that are already being recognized and utilizes quality estimation to avoid running the LPRNet on blurred license plates. The pipeline was designed to meet the challenging requirement of running in real-time for 1080p input resolution with several vehicles in each frame.

All the NN models were compiled using the Hailo Dataflow Compiler and all pre-trained weights and precompiled models were released in the Hailo Model Zoo. The Hailo Model Zoo also supports retraining of the models on custom datasets to ease transferability to other environments. Hailo notes that all models were trained on relatively generic use cases and can be optimized (in terms of size/accuracy/fps) for a specific scenario with dedicated datasets.

Vehicle Detection

For vehicle detection, Hailo used a network based on YOLOv5m with a single class that aggregates all types of vehicles. YOLOv5 is a strong single-stage object detector released in 2020 and trained with Pytorch. To train the vehicle detection network, Hailo took several different datasets and aligned them to the same annotation format. Note that different datasets may also have a different separation between vehicle types, and that this network was trained to detect all kinds of vehicles to the same class (Figure 4). Using high capacity NN, such as YOLOv5m, means vehicles can be detected with very high accuracy, even at a great distance, enabling the application to detect and track vehicles even on highways.

License Plate Detection

Hailo’s license plate detection network is based on Tiny-YOLOv4 with a single class. Tiny-YOLOv4 is a compact single-stage object detector released in 2020 and trained with the Darknet framework. Although the model has relatively modest accuracy on the COCO dataset (19mAP), it has been found to be sufficient for detecting a license plate in a single-vehicle image. Hailo had trained the network using different license plate detection datasets and many negative samples (images of vehicles that do not contain license plates) (See Figure 5).

LPRNet

LPRNet is a CNN with variable-length sequence decoding, driven by connectionist temporal classification (CTC) loss that was trained with Pytorch. This network was trained using mostly autogenerated synthetic datasets of Israeli license plates, which makes it suitable for recognizing license plates with numbers only. To adopt LPRNet in other regions, Hailo recommends using a mix of synthetic and real datasets that represent the license plates of that region and change the number of classes if needed (i.e. adding a different alphabet). In the Hailo Model Zoo, Hailo provides re-training instructions and a Jupyter notebook that shows how to generate the synthetic dataset that was used for training the LPRNet (Figure 6).

Deploying ALPR Using the Hailo TAPPAS

Hailo has released the ALPR application sample as part of the Hailo TAPPAS. The ALPR sample application builds the pipeline using GStreamer in C++ and allows developers to run the application either from a video file or a USB camera. Other arguments that allow users to control the application include: setting parameters for the detectors (for example, the detection threshold); the tracker (for example, keep/lost frame rate); and the quality estimation (minimum license plate size and quality thresholds).

The Hailo Model Zoo also allows users to re-train the NN with their own data and port them to the ALPR TAPPAS application for fast domain adaptation and customization. The goal of the Hailo ALPR application is to provide a solid baseline for building an ALPR product by implementing the full machine learning (ML) application pipeline on the Hailo-8™ and an embedded host processor to be deployed on the edge.

What is GStreamer?

GStreamer is an open-source media framework for developing powerful, complex media application pipelines (Figure 7). The GStreamer pipeline is constructed by connecting different GStreamer plugins together. Each plugin is responsible for certain functionality and the combination of all of them creates the full pipeline. For example, a simple GStreamer pipeline to display a video file would include a plugin to handle the file read, a second plugin to decode the format of the file, and a third plugin to display the decoded frame. Each plugin declares its inputs (sinks) and outputs (sources), and the framework generates the full pipeline utilizing its LEGO-like building blocks.

Hailo’s GStreamer Support

As part of the HailoRT or Hailo’s runtime library, it had released a GStreamer plugin for AI inferencing on the Hailo-8™ AI processor (libgsthailo). This plugin takes care of the entire configuration and inference process on the device, which makes the integration of the Hailo-8™ to GStreamer pipelines easy and straightforward. It also enables inference of a multi-network pipeline on a single Hailo-8™ to facilitate a full ML system.

Except for the standard HailoRT plugin, the ALPR application also uses additional GStreamer plugins that are released with the TAPPAS package – Hailo GStreamer tools (libgsthailotools).

* Tracking: this GStreamer plugin implements a Kalman Filter tracker and is responsible for tracking a general object in an image. It receives updates for each operation of the detection network and is able to associate objects from past frames to assign them a unique ID across frames. The tracker can also generate predictions for the location of the objects in unseen frames.

* Quality estimation: this plugin is able to estimate the quality of an image by calculating the variance of its edges. It receives an input image and calculates its blurriness (score).

* Crop & Resize: this plugin can generate different crops of an image by specific locations. It receives an image and a series of Regions of Interest or boxes (ROIs) and generates several images of fixed size.

* Hailo Filter: a general plugin that allows users to embed C++ code into the pipeline. For example, postprocessing functionality.

* Hailo overlay: to draw the final output of the application, Hailo used a specialized plugin that aggregates all the predictions, draws the bounding boxes and metadata, and generates the final output frame.

Performance

The following table (Figure 8) summarizes the performance of the ALPR application on Hailo-8™ and i.MX8 with USB Camera in FHD input resolution (1920×1080) as well as breakdown for the NN standalone performance.

The Hailo ALPR application presents an end-to-end application framework for deploying AI in intelligent transportation applications on the edge. It includes the entire application pipeline deployed in GStreamer with Hailo TAPPAS and re-training capabilities of each neural network to enable customization with the Hailo Model Zoo. This application provides a solid baseline for building an APLR product with Hailo-8™.