Performance Analysis in Multi-KPI Optimizations

Importance of resource planning at airports, ports, logistic centers and similar operation points is increasing significantly each day due to competitions, intensities and irregularities in operations. Multi-objective optimization algorithms try to reach the user defined objectives of the related operations as much as possible but the performance of these algorithms starts to differ while the number of defined Key Performance Indicators (KPI’s) are increasing. In multi-KPI optimization algorithms, there are many issues and parameters to consider which affect the optimizer performances such as; relationship between KPI’s, the number of KPI’s, number of resources, tasks. In addition, due to some specific business rules in the operation, not every resource can be assigned to every task and the optimization algorithm needs to consider these rules when generating allocation plan. Within the scope of this study, an optimization algorithm which is developed by TAV Technologies is used to analysis optimizer performance changes according to the number of defined KPI’s. For the same resource and task group, the optimization algorithm configured with different KPI combinations and run repeatedly. Except for the KPI definitions, all other optimizer inputs were kept constant in all tests and the results were compared with each other. Specific business rules were ignored in this study to analysis test results clearly.


Introduction
Efficient resource planning contains complex problems in industries with intensive operations. According to Eurocontrol Report, total flight delay cost for aviation stakeholders and consumers is estimated at EUR 11.2 billion in 2012 for European Airspace [1]. Optimizing resources with considering schedules, costs, resource counts and other KPI's is extremely difficult because in such industries there are always many stakeholders involved in the operations with different KPI focuses [2]. For example, one of operation unit called former in shipping industry usually does not care transit inventory costs but this is a major logistic cost for another operation unit called later [2]. Also, stochastic nature of these operation networks can complicate the resource planning process [3]. It was published in an article that only %50 of vessels are arriving on scheduled time because of uncertainties in the shipping operations at sea and ports [4]. The importance of KPI's can vary from stakeholders to stakeholders as well as industry to industry. In air and sea transport, costs could be one of the most important KPI but in military operations timing is above all else [5].
Complexity in resource optimization occurs because of resource and task counts, resource qualifications, task requirements, business rules, multi-stakeholder and other external factors. There are more than 3 million combinations that need to be evaluated in a system that will assign only 10 tasks to 10 resources. Resource and task matching combination selection is based on optimizers KPI's definitions and business rules. It is not possible to achieve maximum success for each KPI in an optimizer which is configured with several KPI's [6]. Also, performance of each KPI depending on its weight and the effect of other KPIs according to their weight may differ [7]. In different business cases of different industries, several research has been done to solve the Multi-KPI optimization problems and the difficulty of the problem has been mentioned in all these researches [8] - [12]. In addition, publications examining the advantages and disadvantages of the algorithms obtained in these studies have been published [13]. Two main approaches have been followed by the scientist in the solution of the problem: first one is named as classical method in which optimal result is searched in each run and the second one is named as evolutionary method like genetic algorithm in which not a single result but possible candidates could be determined [14]. The weakness of these methods are widely researched in the literature but major deficiency with most of the classical models is on handling more than one KPI [15]. Two stage Monte Carlo solutions are also tested by scientists in large stochastic networks and promising results gathered [16] - [19].
In the scope of this study, as TAV Technologies, after examining the methods mentioned above we developed a unique, multi-KPI definable, mobile resource planning algorithm for airport and examined the effects of multi-KPI configurations of the optimizer on KPI performances. Due to commercial concerns, technical details related to the algorithm developed within the scope of this study were not shared. It should be noted that, several normalization methods used in iterations to compare KPI's with each other.

Simulation Definitions
Tests were designed as airport ground handling mobile resource optimization simulation. In each test, same 250 ground handling service tasks and same 50 ground handling service resources are used. These tasks represents cleaning, passenger boarding, docking, etc. on the airport apron for each flight and resources are the people who will accomplish these tasks. To simulate the physical difficulty of these tasks, different workload values are defined for all tasks. The total workload value of all tasks is determined as 9000 which means that average workload for a single task is 36. Each task has different starting/ending point and in order to compare each test result with each other, the starting/ending points for all tasks are kept constant in each test. "Workload Balancing" and "Total Distance Minimizing" KPIs were used in the tests. With Workload Balancing KPI (WLB), it is desired to distribute an equal amount of workload to mobile resources. With the Total Distance Minimizing KPI (TDM), it is aimed to reduce the distance travelled by the resources in the apron while performing the duties of the resources. To reduce the distance, it was necessary to assign the task to the resource closest to the starting point of the task. Of course, when the resource finishes a task, the current position of the resource becomes the ending position of the task.
As shown in Table-1, in order to make the results more accurate, the tests with the same optimizer configuration were repeated 5 times. In the optimizer configuration 1, both KPIs are deactivated and in this case, the resource allocations of tasks must be executed randomly without any restrictions. In optimizer configuration 2, optimizer will allocate resources with only considering Workload Balancing KPI and this case represent single KPI conditions. Another single KPI conditions is tested with optimizer configuration 8 for Total Distance Minimizing KPI. For all other optimizer configurations, the multi KPI condition was tested by changing the KPI weights.  As it can be seen in Figure-1, optimizer tries to distribute tasks considering their workloads between resources. In the Test-1 (No-KPI Configuration), the total workload of the tasks that each resources get as a result of optimizer allocation vary between 0 and 400. However, when the WLB KPI is activated as a single KPI in Test-2, the total workload of tasks received by the resources as a result of allocation vary between 155 and 210. In other words, when the WLB KPI is activated, the optimizer tries to distribute the tasks to the resources with considering workload balance. Thus, people as resources working in the same shift will get tired equally and employee's dissatisfaction will not be experienced. In the tests performed with the No KPI configuration, the average of the coefficient variant is 0.42, and in the tests with 100% WLB configuration, the CV average is 0.07. As a result of these test Workload Balancing KPI function's average performance is calculated as 83%. Another KPI that is as important as workload balancing in the efficient and balanced use of resources is to reduce the distance travelled by the resources during the shifts. The way to achieve this is to assign the tasks to the nearest resources. However, using CV in the performance measurement of this KPI will not be meaningful because the aim of the optimizer is to reduce the total distance travelled by all resources, not balancing the distances travelled by resources. After Workload Balancing KPI as a single KPI performance tests, Test-1 and Test-8 results were examined together as shown in Figure-3 to measure the performance of Total Distance Minimization KPI. After TDM KPI is activated, a significant decrease has been observed in the distance travelled by each resource during the operations. In fact, some resources have not been assigned to any task. As seen in Figure-4, the tests mentioned in Figure-3 were repeated 5 times and sum of total distances of resources were analysed for each test result. In the tests performed with the No KPI configuration, the average of the sum of total distances travelled by resources is 89 km, and in the tests with 100% TDM configuration, the average of the sum of total distances travelled by resources is 5.8 km. If the perfect optimization could be achieved, this value would be 2 km because the test was constructed in this way. As a result of these tests Total Distance Minimizing KPI functions maximum performance is calculated as 95.7%.

Multi KPI Performances of Optimizer
As seen in Figure-  When the results of all the tests are analysed together, the equal distribution of the workload is provided to a certain extent due to randomness in the tests performed with No KPI configuration. The same condition is not valid for the Total Distance Minimizing KPI because it can approach the worst possible performances with the No KPI configuration.
The WLD can have very close values to the best performance until its weight drops to 50%. Under 50% weight performance deteriorates very quickly.
TDM achieves its best performance with a logarithmic relationship with its weight.

Conclusions
As a result of the tests carried out, it is seen that, KPI's can affect each other in a different way. The best result for a KPI is achieved when that KPI is configured in the optimizer as a single KPI. However, for some KPI's like workload balancing, the best results can still be achieved up to a certain weight. In addition, for KPIs that are not conflicting with randomness, good results can be obtained even optimizer is not configured with a KPI.
For some KPIs like TDM, the weight of KPI can directly affect its performance. While no KPI definition has been made on optimizer, the results can be very close to the worst results for these KPI's.
When the number of KPI's reaches to higher numbers, combinations of different weights of KPIs should be examined in detail in order to achieve optimum performances. In some conditions, it possible to see that some KPIs do not affect each other.
In addition, in the tests carried out in this study, more resources were used than the required. When the number of resources has been reduced slightly, it is seen that in single KPI configurations the performance of the TDM decreases and the performance of the WLB remains almost same. However, at the point where the number of resources is more reduced, different results can be observed. Due to the number of different resources and tasks, KPI performances may change at different times of the year or even day. It will be useful to analyze the effect of the number of resources and tasks on these KPI performances in future studies.
When all the results and researches are evaluated together, it can be seen that operation-specific analysis will be useful in multi-KPI optimizations. It is not easy at this stage to achieve the best results in any business environment with a generic model.