Video Enhanced RFID Tracking System

: Generally speaking, crimes in which vehicles are used are generally highly organized criminal activities affecting all regions of the world and with clear links to illegal activities and terrorism [3]. Therefore, catching illegal vehicles on the roads is an important preventive measure in security. In this paper, we propose a Video Enhanced RFID Tracking System to monitor the activities of vehicles through the use of readers, tags, and video together. The proposed system retrieves the registered visual properties of vehicles in the environment by querying their RFID tags on the database in the Command Control Center. It processes the video frames simultaneously, and extracts the visual features of detected vehicles. The information collected both from video and RFID database query is then combined in such a way that they validate each other. If an inconsistency is detected in this process an existence of illegal vehicle is found in the environment.


Introduction
Safety should be a primary concern in major metropolitan areas. However, due to the attributes involved, it is a very complex and difficult task. In order to minimize the impact on citizens, in addition to relying on security forces, some technology-based measures are necessary not only nationally but also internationally. One of these measures involves vehicle tracking. Vehicles are generally used in organized criminal activities affecting all regions of the world. They are not only stolen for their own inherent value, but also trafficked to finance other crimes. According to the Interpol report [3] published in 2013, there are 7.2 million stolen motor vehicles reported. In addition to the Interpol's data repository, which accommodated more than 125 million searches within the year of 2013, a network of stolen vehicle databases of 130 countries was created to share information. As a result, around 117,000 motor vehicles were identified worldwide during the same year. FBI's Uniform Crime Reporting (UCR) Program defines motor vehicle theft as the theft or attempted theft of a motor vehicle. Crimes involving motor vehicle theft represent an important portion of the crimes committed worldwide. According to the victimization surveys, thefts involving motor vehicles get reported to the police very often. Over 80% of the thefts involving motor vehicles in developed countries are reported to the police (Alvazzi del Frate 2005, van Dijk et la. 2007). The United Nations Office on Drugs and Crime (UNODC) reports that for every 100,000 people, the number of crimes involving vehicular theft can reach between 300 and 350 worldwide (International Statistics on Crime and Justice, 2010). Prevention of such of an illegal activity that happens worldwide with a high occurrence that causes billions in losses cannot be achieved by simply increasing security measures deployed in motor vehicles or by doing searches in worldwide databases hoping to find results. There is an utmost requirement for a system that is capable of: Tracking motor vehicles in an automated manner so that a vehicle can be located without being bound to a database where only the owner of the motor vehicle can find the vehicle with a reasonable chance of success. In this study we propose a Video Enhanced RFID Tracking System to monitor the activities of vehicles through the use of readers, tags, and video together. The proposed system validates information acquired from RFID tags installed in the vehicles with the visual properties extracted from video data. It first retrieves the registered visual properties of vehicles in the environment by querying their RFID tags on the database in the Command Control Center. It also detects the vehicles in the video frames simultaneously, and extracts their current visual features. The information collected both from video and RFID database query is then combined to validate each other. If the registered visual properties of a vehicle can't be found in the video frames, then that vehicle is labeled as an illegal vehicle.

Literature Review
Tracking systems have been deployed in many different areas. Some areas include vehicle tracking in transportation, item tracking in warehouses, and underground exploration. Regardless of the area of usage, tracking systems have been classified under two major groups: 1) manual tracking systems and 2) technology based tracking systems. Manual tracking systems are time consuming, impractical, and labor intensive. Therefore, they have a number of limitations such as pinpointing the exact location, communication failure, and the size of the area used. On the other hand, technology based tracking systems can further be classified as Reader-based Tracking System (RTS) and Node-based Tracking Systems (NTS). RTS systems are composed of two components: readers and tags. The reader detects the presence of a tag. The tag represents an object to be tracked. NTS, on the other hand, tracks objects by using the RF distance between nodes and radio. In this tracking technology, there is no role for tags. Intelligent Transportation Systems (ITS) can be defined as systems in which information and communication technologies are applied in the field of road transport (including infrastructure, vehicles and users) and in traffic and mobility management, as well as for interfaces with other modes of transport, according to the EU Directive 2010/40/EU (7 July 2010) various methods for vehicle detection and tracking in the context of ITS have been proposed in the computer vision community. Algorithms based on background subtraction with a blob detection are widely used to detect moving vehicles in the scene. Bhaskar and Young [1] used Gaussian Mixture Models (GMM) for the background subtraction and applied morphological operations to further improve the results. Huang [2] also developed a very similar system, but they also included shadow removal operation and sub-feature extraction in post-processing stage. Another approach uses object-based detection such as shape and color of vehicles. Tsai et al. [3] developed a color based Bayesian Classifier to select possible vehicle candidates. After locating vehicle candidates, three important features, including corners, edge maps, and coefficients of wavelet transformations, are used for constructing a cascade multichannel classifier to detect vehicles. Yang et al. [13] proposed a method where they track the vehicle based on detecting only its windshield instead of tracking vehicles as a whole. The reason behind this is that the windshield is the least affected part of vehicles in collision. The probability of each pixel being the center of the windshield is estimated by the fusion of shape matching and edge matching likelihoods. Vehicle hypotheses are then inferred from the confidence map consisting of posterior probabilities by the Mean-Shift algorithm. Lastly, tracking is done by using the Kalman filter over the windshield detection results. Under night conditions, the tail and head lights of vehicles are discriminative enough to be considered interest points. Chen et al. [5] uses a fast bright-object segmentation process to extract pixels of lighting objects. Those lighting objects are then grouped by a spatial clustering process in order to obtain groups of lights of potential moving vehicles. Next, a feature-based vehicle tracking process based on location, dimension, and pixel distribution information is applied to analyze the spatial and temporal information. Li and Yao [6] also utilized vehicle lights in night conditions. Maximally Stable Extremal Regions (MSER)-based segmentation is used to extract light candidates in HSV color space. Then, all lamp candidates are tracked by using the Kalman filter. Wang [7] used Joint Random Fields as a vehicle model to improve the vehicle segmentation process. Data-dependent contextual constraints among both detection labels and latent variables are integrated during the detection process. The algorithm handles both moving cast shadows/lights and various weather conditions. In recent years, fusion techniques have become a valuable asset in vehicle detection and tracking. Robert [8] has implemented a new framework to detect vehicles based on a hierarchy of features, detection, and feature fusion. The system first fuses vehicle features including headlights or windshields. Then, a constant acceleration tracking model augmented with traffic-domain rules is implemented to deal with the problems related with collision. Yang et al. [13] also used feature fusion in their work, though with different features such as vehicle color, edges, and interest points. Sheng et al. [10] proposed an improved background estimation process by using fast Constrained Delanuay Triangulation (CDT). The new method estimates the background model self-adaptively.
When one of them is seen as the candidate, the main features are extracted by using Principal Component Analysis (PCA). And then, the eigenvector which consists of the main features are put into a trained SVM classifier that verifies whether the area is a vehicle area or otherwise. Lastly, a parallelogram is used to represent vehicle contours. Liu et al. [11] used a multi-view approach to handle occlusions and issues generated from multiple points of view. The proposed method works in two stages: the two-step view selection process and the dual-layer occlusion handling. For the two-step view selection, a Multi-Modal Particle Filter (MMPF) is proposed to track vehicle. Spatial-temporal analysis is employed to further decide to maintain the consistence of view transition. For the duallayer collision handling, a cluster-based dedicated vehicle model for partial collision and a backward re-tracking procedure for full collision are integrated complementarily to deal with collision problems.

Video Enhanced RFID Based Tracking System(VETS)
Generally speaking, monitoring and tracking are known to be the two fundamental attributes necessary for reliability in certain industries such as transportation, healthcare, and defense and security. Although each one has a specific focus in providing a degree of reliability, they are known to be closely coupled metrics used in effective information mining, regardless of the application area.
If you live in a metropolitan city, you may soon realize that there should be some sort of counter measures taken in order to make it safe and enjoyable. In a complex environment where multiple attributes play a significant role, safety is not easy to provide unless everything is closely monitored. Monitoring works well if the right technology is in place. For instance, an opportunity to install and/or carry a very high tech item: a remotely identifiable vehicle and readable license plate and a driver license, which are designed to identify drivers as they drive in and around the city. Therefore, the proposed idea is being promoted as a way to save time and simplify traffic monitoring, including those which involve suspicious activities, traffic jams, traffic accidents, auto thefts, etc. Hence, the system in Figure 1 containing RFID and smart tags has promising potential.

A. Preliminary Work
The proof of concept framework, using three sets of monitoring devices (each set contains a camera and a RFID tag reader) was established at an extremely busy street in the Besiktas district of Istanbul, Turkey. The flow of the traffic was monitored in order to isolate predefined object or suspicious objects, in terms of color, type, and license plate (if possible). With the preliminary use of object identification and tracking algorithm, the current success ratio of the VETS system using the data recorded during the above average day is 90%. However, refinements for bad weather conditions and nighttime are in progress.

B. Environment Setting
VETS aims to identify, follow, and match the vehicles in real-time.
The system needs to read the object's RFID tag numbers while the object comes into any camera's view angle. Then, the system tries to match RFID tag number with the vehicle which is identified. If the system matches the RFID tag number with the object, the system saves the date, time, and location for this object and starts tracking the object between locations. At the last step, the system will calculate and show the vehicle path to the user. However, if the system cannot detect or match the RFID tag number with the vehicle, the system will again save the date, time and location and warn the end user.

D. Environmental Setup for Data Recordings
The system, which supports the paired devices, is setup in the BAU building facing Ciragan Street. The data collection is conducted by two sources: three video cameras and two types of RFID readers (two passive and one active). The organization of the devices was sequential and arranged in a triangle shape layout to complete the initial experimentation of data recording.

1) Sequential camera setup:
In camera monitoring setup, the cameras located in sequential order, as seen in Figure 2. According to the first environmental setup, Camera A (CA) and active RFID reader A (RA) are located at location A on the street, if an object (vehicle or human) enters view angle of the camera, the camera and RFID tag reader will detect and read the object. Because the camera monitoring setup is sequential, all cameras should be placed in an order, maybe on the same straight line, so Camera B (CB) and RFID reader B (RA) is located at location B and Camera C (Cc), RFID reader C (Rc) located at location C. Again if an object is recognized by the camera or read by tag reader, both the camera and RFID tag reader tries to identify the object. The last system, Camera C and passive RFID reader (for camera C) is located at location C of the hall. Every camera's view angle is different from the others. For instance, Camera B's view range does not cover Camera A and Camera C's view ranges. After the environment setup, the system prototype will try to detect and identify the object and its path. If the system cannot succeed in matching the RFID information with the camera view, the system will start tracking mode for this object. Moreover, if the system cannot read the RFID tag while the object is in the view angle, the system will again start the tracking mode for this object. However, in this environment setup, there are just two possible paths for using three cameras, such as A-B-C and C-B-A. While testing the system, the objects can only travel between two cameras (e.g., A-B, B-A, B-C, C-B. We can test the system with a total of six possible paths.

3) System Activity Diagram: The RFID and Camera
Following system contains four subsystems, including a RFID reader, camera recorder, mapping process, and special case process. The RFID reader program aims to read and store RFID tags for each system location in real-time. The purpose of the Video Recording Process is to detect vehicles in each frame taken by the cameras. The mapping process aims to combine other two processes and tries to map the tag number with the video frame information. The last process is special case process and is triggered when a problem occurs while the mapping process between the camera database with RFID database. When the mapping process cannot map a vehicle, the special case process will be triggered to start tracking for the unidentified vehicle in close camera locations.

E. RFID Reader
As shown in Figure 4, the information processing module of the RFID Reader always tries to read the RFID tags in range. If the process cannot is unsuccessful, the process again tries to read RFID tag again. However, if the RFID tag reader succeeds, the software tries to match the TFID tag number with the general database. If the process matches the RFID tag number, the process stores the matched information to database. Although the process cannot match the tag number with database, the system stores 'unknown' tag number, adding a tracking flag and warning the end user.

F. Video Recording
The Video Recording process receives and processes the video frames to detect the vehicles, as shown in Figure 5. Vehicle detection is performed in two steps. In the first step, GMM based background subtraction is applied to the current frame to detect moving objects [1]. Foreground regions are further processed by morphological operations. In the second step, each foreground region is analyzed with respect to their visual properties, to decide whether it is a vehicle or not [13]. If the process cannot detect any vehicle in the frame, the process finishes and starts from the beginning. However, if the process detects vehicle(s) in the frame, the process enumerates these detected vehicles. Then, the process stores the vehicle, time, location, and starting frame information.
The process gets the next frame and again enumerates the vehicle(s) while the previous vehicle is still within the shot frame, the next frame will be captured and processed. If the process cannot detect any vehicle in the next frame, the process stores the information from the last frame into the database too. The process stores the first vehicle view frame and the last vehicle view frame. Using this method, the system can understand how many frames in the car was located in the current camera view.

MAPPING PROCESS
The mapping process, as seen in Figure 6, aims to combine the information collected from the two sources to identify the vehicle. To achieve this goal, the process gets the specific location information of the Video & RFID from the header files. Upon establishing the link, each information source is separated in order to complete the mapping with the information already available in the databases. More specifically, each process tries to match this tag information from the general database to reach the vehicle and vehicle owner information. At the same time, the process again gets the same vehicle location information from the video. After getting this information, the process tries to match RFID tag information with the vehicle visual qualifications. If the system succeeds to match this information, the system stores the location and time information to database. However, if the system cannot succeed in matching, the system executes the tracking mode and warns the end user. At this point, the mapping process controls all of the missing RFID tags and counts how many unidentified vehicles are in the frame. If there is only one missing RFID tag, the process adds a flag and stores the location and vehicle information to the database. If there are more than one missing RFID tags, the process creates a special case for these vehicles. While creating a special case, the process stores the location, date, and all vehicles' information. To track these suspicious vehicles, the process triggers the Special Case process.

SPECIAL CASE PROCESS
The Special Case process, as shown in Figure 7, is triggered by the mapping process. While mapping vehicle camera frame information with RFID information, the mapping process triggers the special case process. This process tries to solve the unidentification problem with tracking the vehicle with sequential camera locations. If the process solves the problem, the process removes the flag for this case.

DATABASE SIDE
The Vehicle tracking system contains four processes, and these four processes use same database. Therefore, the database structure has to cover all of the requirements related to these processes. All processes can communicate over this database. The database architecture makes data transfer possible between all four processes when the data transfer is needed. When a vehicle is read by a RFID reader at any location, the reader sends the RFID tag number, location ID, read date, first reading time for that location, and last reading time for that location to the RFID reader which saves all of the information to the database. While the RFID reader process is being processed, the Video recording process processes all videoswhich are taken from security cameras -to detect vehicles in video frames. If the process detects a vehicle in the frame, the process tries to identify and classify the vehicle. If the process identifies the vehicle location information, the date, identified class number, first read time, and last read time is saved to database. The mapping process tries to map both RFID and Video process results in order to identify a vehicle at a specific time and specific location. The last process is the Special Case Process, which is triggered by mapping process. If vehicle(s) cannot be mapped with video frame and RFID tags, the Special Case Process is triggered. When this process is being called, this process checks the vehicle from database records. In recent years, performance is an important criterion to evaluate the system is successful or not. Database architecture is one of the most important factors that affects system performance. If the database architecture uses too much traffic to complete basic or common operations, the system needs to spend more time in regards to database. However, VETS database architecture is designed to make the system as fast as possible. The main idea of this architecture is based on using less data traffic between processes and the database.

A. VehicleOwner
When a vehicle is registered, the vehicle owner information is stored in the system. VehicleOwner stores owner information. The table can be extended with other attributes such as owner infotmation (PK: OwnerId).

B. OwnedVehicles
Registered vehicles are stored in this

D. WatchingVehicles
This

E. SpecialCase
This table decides the special case is solved or not. (PK: SpecialCaseId)

F. SpecialCaseDetected
The un-identified vehicles are stored in this table (PK: DetectedVehicleId)

G. DetectedRFID
When a vehicle's RFID tag is read, the system automatically stores this information in this

H. DetectedCameraFrame
When a vehicle is detected on a camera frame, the system automatically stores this information in this

I. LocationList
This table stores all detailed location information and enumerates the locations.

EXPERIMENTAL RESULTS
The experimentation conducted in the Besiktas district involved