A Novel Modified Lévy Flight Distribution Algorithm based on Nelder-Mead Method for Function Optimization

A Novel Modified Lévy Flight Distribution Algorithm based on Nelder-Mead Method for Function Optimization Ahmet Dündar , Davut İzci , Serdar Ekinci , Erdal Eker 4,* 1 Vocational School of Social Sciences, Muş Alparslan University, Muş, https://orcid.org/0000-0002-2123-0564 2 Department of Electronics and Automation, Batman University, Batman, https://orcid.org/0000-0001-8359-0875 3 Department of Computer Engineering, Batman University, Batman, https://orcid.org/0000-0002-7673-2553 4 Accounting and Tax Department, Muş Alparslan University, Muş, https://orcid.org/0000-0002-5470-8384


Introduction
The optimization techniques have been gaining a greater demand in recent decades due to the need for reliable and effective methods to deal with real-life problems that present increased complexity [1]. In addition to this fact, the increasing number of optimization problems of different fields has led the optimization techniques to be one of the major research areas [2].
Due to inherent disadvantages of deterministic techniques such as being derivative dependent and stagnating in local optimum, this field of research tends to develop different alternative As stated by this theorem, there is not any single algorithm that is convenient to solve all existing optimization problems. Therefore, there is a growing appetite for developing newer algorithms that may be quite effective for some of the problems [9]. Lévy flight distribution (LFD) algorithm [10] has been developed as part of the latter effort. The wireless sensor nodes that have connections related to Lévy flight motions have inspired the development of this algorithm.
Having a balance between global and local search stages is an of important feature that make metaheuristic algorithms desired tools to tackle with various problems [3]. However, due to their stochastic nature, it may not always be feasible to offer such integrity in those algorithms. One way of dealing with this issue is to benefit from the feature of existing algorithms instead of developing new ones. Hybridization is a great choice to do so since it allows the combination of existing and complementary algorithms [11]. In this way, a new structure with balanced feature in terms of exploration and exploitation can be achieved.
In terms of LFD algorithm, it has good explorative behavior due to Lévy Flight motions, however, lacks from exploitative structure. Bearing the above discussion in mind, the LFD algorithm can be improved by adopting another complementary approach so that a novel and more capable hybrid algorithm can be obtained. To achieve such a structure, the Nelder-Mead (NM) simplex search method [12] can be used.
The latter is a well-known simplex search method which is quite capable for local search. In terms of hybridization, several examples are available in the literature which adopts NM algorithm. Some of them are firefly algorithm for optimal reactive power dispatch [13], particle swarm optimization for modeling Li-ion batteries for electric vehicles [14], Harris hawks optimization for solving design and manufacturing problems [15], ant lion optimizer algorithm for structural damage detection [16], moth flame optimization for parameter identification of photovoltaic modules [17] and artificial electric field algorithm for optimization problems [18].
Considering the discussion so far, this paper aims to develop different hybrid versions by modifying LFD algorithm using NM method and investigate the promise of those variations for optimization problems. Therefore, three novel strategies were used to achieve NM modified LFD algorithms. All these strategies are discussed in related section of this paper.
In order to evaluate the performances of the original and the modified LFD versions, wellknown benchmark functions of unimodal and multimodal features were adopted. Then, statistical analysis was performed using metrics of average, standard deviation, best and worst. Besides, all approaches, including the original LFD algorithm were ranked. In this way, the ability of the algorithms was assessed for global and local searches. Further assessment took place by performing a nonparametric statistical test known as Wilcoxon's signed-rank test to confirm the capability does not occur by chance. The obtained results have shown that hybridizing LFD algorithm with NM method provides a significant performance improvement in general as was expected. Besides, it has also been found that the efficiency is increased if the NM method is applied after each iteration of LFD algorithm and run each time for the total number of LFD algorithm's current iteration.

Lévy Flight Distribution Algorithm
Wireless sensor networks having a Lévy flight ( ) motions related connection is the main inspiration for LFD algorithm [10]. Mathematically, the LFD algorithm is initialized by calculating the Euclidean distance ( ) between adjacent nodes which determines the position replacement of sensor nodes. To locate a sensor node, LF is performed. In such a case, the sensor node is placed close to another one with lower number of neighbors or in a position that has no sensor node. The latter behavior increases the effectiveness of the exploration. Two important parameters for generating random walks are the step length and the direction of the walk. To determine the step length ( ), the following equation can be used where is the Lévy distribution index having limits of 0 < ≤ 2.
The parameters of and , given in the above equation, can be determined using (2).
and represent standard deviation and calculated as in (3): where ζ is a function having the following definition for an integer .
The value is calculated as in (5) for the locations of adjacent agents ( and ): where and positions are represented by ( , ) and ( , ), respectively. A pre-defined threshold value is compared with the value of through iterations. The positions of search agents are re-adjusted using (6) for smaller values than the threshold.
In the above equation, is used for the number of iterations whereas and denote the lowest and the highest limits of the search space.
function represents the value and the LF direction.
is used as LF direction since it represents the position of the agent with the lowest number of neighbors. In order to increase the exploration capability, the agent of is moved towards the agent with the lowest number of neighbors using (7).
In the above equation, () is used to generate uniformly distributed random numbers in a range of [0, 1]. The following identification helps discovering the search space with more opportunities: where is the comparative scalar value with in each position update of . The value of is checked and compared with . In case of smaller values than values (6) is executed whereas (7) is used otherwise. The position of is updated using (9) and (10).
position is calculated using (9) whereas (10) provides the final position of . The solution with the best fitness value (target position) is denoted by . The parameters of 1 , 2 and 3 are used to represent random numbers of 0 < 1 , 2 , 3 ≤ 10. The following gives the total target fitness of neighbors ( ) around ( ) where the neighbor index and neighbor position of ( ) are denoted by and , respectively.
The total number of ( ) neighbors is represented by . ( ) denotes the fitness degree for each neighbor and given by (12).
A detailed flowchart of LFD algorithm is demonstrated in Figure 1.

Nelder-Mead Method
This algorithm is a simplex search method and developed to solve nonlinear functions using gradient-free computations [19]. An optimal point of 1 is determined by generating + 1 points of 1 , 2 , … X +1 . Then, the respective fitness function values of ( 1 ), ( 2 ),… ( +1 ) are evaluated and sorted in ascending order. Four scalar coefficients of reflection ( ), expansion ( ) contraction ( ) and shrinkage ( ) are used to replace the worst point of +1 . The computed fitness values allow determination of the best ( 1 ), the worst ( +1 ) and the centroid ( ̅ ) points. To identify the reflection, , (14) is used: The reflection point is expanded using (15): where denotes the expansion point and replaces the worst value for ( ) < ( ). Otherwise, this point is replaced by . The contraction step is performed for ( ) ≤ ( ). An outer contraction ( ) is generated using (16) to obtain the fitness value of ( ) in case of ( ) < ( +1 ).
The point of +1 is replaced by , then the iterations are terminated for ( ) ≤ ( ). Otherwise, the shrinkage occurs in the next action. An inner contraction ( ), provided in (17), may also be constructed in the contraction step to obtain fitness of ( ) for ( +1 ) ≤ ( ).
The point of +1 is replaced by , then the iterations are terminated for ( ) < ( +1 ). Otherwise, the shrinkage occurs. The shrinkage step is the final operation which constructs new points using (18).
The flowchart of NM simplex method is provided in Figure 2.

Proposed Hybrid Strategies
This section provides information about different approaches to hybridize LFD algorithm with NM simplex search method. In order to improve the performance of the original LFD algorithm, three different strategies were employed to adopt NM method for modifying LFD algorithm. To have a fair assessment, a dimension of 30, maximum iterations of 500 and a population size of 50 were adopted for all approaches.
In the first proposed strategy, LFD algorithm is performed. Then, the NM method is applied after LFD algorithm completes its task entirely.
The NM is run twice as much the number of iterations in this strategy which means the NM is performed for 1000 iterations since the chosen number of iterations was 500. This strategy was named as LFDNM-S1.
In the second proposed strategy, unlike the first one, the NM method is applied after each iterations of LFD algorithm instead of waiting for the completion of the latter algorithm.
However, the NM method is performed for + 1 iterations where is the dimension of the problem. This strategy was named as LFDNM-S2.
The third proposed strategy is the last approach that was adopted for modifying LDF algorithm using NM method. Similar to the second strategy, the NM method is applied after each iterations of LFD algorithm in this strategy, as well. However, the implementation of NM method lasts for iterations after each iteration of the LFD algorithm where is the current iteration of the latter algorithm. That means, for example, the NM method would run for 10 iterations if it is implemented after the 10 th iteration of LFD algorithm, and run for 11 iterations after 11 th iteration of LFD algorithm and so on. The last strategy was named as LFDNM-S3.

Experiments and Discussions
The performance validation of the original and NM modified versions of LFD algorithms together with the parameter settings are presented in this section. The performances of the respective algorithms were tested using wellknown four unimodal and four multimodal test functions provided in the following subsection. The algorithms were tested against each other using a set of fixed parameters for the sake of fair comparison. Therefore, a swarm size (search agents) of 50 and maximum iterations of 500 along with dimension ( ) of 30 were adopted for all algorithms. Besides, each algorithm was performed on each test function for 30 independent runs.
The parameter values for LFD were chosen to be 2 for threshold, 0.5 for , 1.5 for , 10 for 1 , 0.00005 for 2 , and 0.005 for 3 along with 0.9 for 1 and 0.1 for 2 [10]. In terms of NM method, the parameter values were chosen to be 1 for , 2 for , 0.5 for and 0.5 for [19].
In terms performance evaluation of the algorithms for global and local search abilities, the statistical values of average, standard deviation (Sdev), best and worst were used. Besides, the algorithms were ranked. In addition to those statistical metrics, a nonparametric statistical test known as Wilcoxon's signed-rank test [20] was also performed for further assessment of the algorithms. The adopted statistical metrics of average, Sdev, best and worst can mathematically be defined as given in (19), (20), (21) and (22), respectively.
where is the number of runs and is the function fitness value.

Benchmark Functions
The following benchmark functions listed in Table 1 have been adopted for this study. The related table contains unimodal test functions of Sphere, Schwefel 2.22, Rosenbrock and Step together with multimodal benchmark functions of Schwefel, Rastrigin, Ackley and Griewank. Those are all well-known test functions with different properties and present a good environment for performance evaluation of the algorithm such that the exploitation and the exploration capabilities of the algorithm can be assessed using unimodal and multimodal functions, respectively [21].
For example, the unimodal benchmark functions have one global optimum with no local optima and are good for assessment of exploitation ability of the algorithms. On the other hand, the multimodal benchmark functions have considerable number of local optima which make them good for assessing the exploration capability of the algorithms. The properties of both unimodal (Sphere, Schwefel 2.22, Rosenbrock, Step) and multimodal (Schwefel, Rastrigin, Ackley, Griewank) benchmark functions can also be seen visually as demonstrated in Figure 3. The performance of the proposed NM modified LFD algorithms together with the original LFD algorithm was tested against each other using those benchmark functions.
In terms of implementation of the algorithms on these benchmark functions, the related search domains listed in Table 1 for the respective test functions along with a dimension of 30 for each function were adopted. Then the population size, and number of iterations were defined, and the related algorithms were tested against those test functions in terms of statistical performance. The results were then compared with each other. Figure 4 shows the implementation steps in brief.

Exploitation Capability
As mentioned in the previous subsection, the unimodal functions ( 1 ( ), 2 ( ), 3 ( ), 4 ( )) provided in Table 2 can help assessing the local search capability of the algorithms under consideration [22]. It can easily be spotted that the average values for all test functions obtained by the LFDNM-S3 algorithm (shown in bold) are well below the other values achieved by the other algorithms. In addition, the LFDNM-S3 algorithm has achieved better values in terms of other statistical metrics. Besides, the constructed LFDNM-S3 algorithm has also been ranked the first, as well. The obtained results clearly show the third proposed strategy for NM modified LFD algorithm has a strong competitiveness in terms of exploitation.

Exploration Capability
In terms of assessment of global search capability, the multimodal functions ( 5 ( ), 494 6 ( ), 7 ( ), 8 ( )) provided in Table 3 can be used [22]. Similar to local search ability, the LFDNM-S3 algorithm has also achieved better results than the other algorithms in terms of all statistical metrics. Besides, the constructed LFDNM-S3 algorithm has also been ranked the first for multimodal functions, as well. The obtained results clearly show the LFDNM-S3 algorithm's strong competitiveness in terms of exploration, as well. Considering both global and local search capabilities, it can be concluded that the third proposed strategy for improving LFD algorithm through NM method has a good balance in terms of exploration and exploitation.  The comparison between LFDNM-S3 and the LFDNM-S1 algorithms shows no significant difference only for 5 ( ), however, for the rest of the functions LFDNM-S3 has clear superiority. In addition, the efficiency of the third approach was found to be better for NM modified LFD algorithm since it has demonstrated a greater balance between exploration and exploitation phases. Therefore, the latter can be used as an effective tool for optimization problems. Bearing the obtained results in mind, the constructed algorithms have the potential to be used for several different reallife optimization problems for future works. Some of them can be listed as controlling an automatic voltage regulator system, regulating the speed of a direct current motor, and operating a magnetic levitation system.