LEAKAGE ANALYSIS OF THE RSOR ALGORITHM

Abstract: The RSOR algorithm is a recursive algorithm that has been proposed as an alternative to the RLS algorithm for updating adaptive filter parameters. As with other algorithms, the forgetting factor, filter length and relaxation parameter significantly affects the performance of the RSOR algorithm. In this study, using an adaptive FIR filter in system identification mode, the effect of forgetting factor, filter length and relaxation parameter on the leakage phenomenon of the RSOR algorithm was analyzed. For this purpose, firstly, the effect of measurement noise on the adaptive filter output, namely the leakage phenomenon, was explained analytically, and then the influence of the forgetting factor and other filter parameters on this leakage phenomenon was examined. The results obtained from the simulation studies are compared with similar algorithms.


INTRODUCTION
The RLS (Recursive Least Squares) algorithm is preferred in many applications due to its high convergence speed despite its high computational load (Haykin, 2002, Diniz, 2013. On the other hand, some alternative algorithms have been proposed: These are the RSOR (Recursive SOR) algorithm based on the single-step SOR (Successive Over-Relaxation) iteration (Hatun and Koçal, 2012), and the RI (Recursive Inverse) algorithm based on the one-step gradient iteration (Ahmad et al., 2011a). In these type algorithms, the forgetting factor, which is a positive number and taken as close to 1, is used for tracking parameter changes. If the forgetting factor is chosen less than 1, the parameter tracking capability of the algorithm increases, but the stability and misadjustment value of the algorithm are affected negatively. If the forgetting factor approaches 1, the stability and misadjustment value of the algorithm changes positively, but in this case, the parameter tracking capability decreases.
Adaptive filters are used in system identification mode in many applications. In the system identification mode, the estimated values of the adaptive filter parameters converge to their correct values, and the measurement noise can be recovered from the signal estimation error. This result indicates that the system identification process is working correctly. However, in some cases, the measurement noise corrupts the output signal of the filter. In this case, which was called as "the leakage phenomenon", the system identification process does not work properly (Paleologu et al., 2008). The mathematical expression of the leakage signal can simply be derived, but the results obtained between different algorithms may not be the same. The leakage term depends on the forgetting factor, filter length, and other filter parameters, if any.
The leakage phenomenon is a fundamental problem which affects the adaptive filtering applications negatively by increasing the misadjustment. For example, the leakage signal causes a residual error and inefficient cancellation of noise signal in adaptive noise cancellation applications, or imperfect rejection of echo signal in echo cancellation applications (Haykin, 2002, Paleologu et al., 2008, Ciochină et al., 2009).
The effects of forgetting factor and filter length on the leakage phenomenon of RLS and RI algorithms have been studied in detail (Ciochină et al., 2009, Ahmad et al., 2011b. In this article, it is aimed to perform a similar leakage analysis for the RSOR algorithm.
In this paper: The leakage signal for the RSOR algorithm was formulated and analyzed depending on the forgetting factor and the other filter parameters in section 2. The simulation results obtained for the RSOR algorithm were compared with the RLS and RI algorithms in section 3, and some conclusions were given in section 4.

System Identification Using The RSOR Algorithm
In system identification process given in Figure 1, an algorithm adjusts the parameters of the adaptive filter. The system output and the adaptive filter output are given as The parameter vectors of the system and adaptive filter, and the data input vector are defined as follows, respectively: where is the filter length and ( ) is the input signal. The desired signal ( ) is tracked by the output of the filter, and it is written as where ( ) is the measurement noise with a zero-mean and variance 2 , and independent with the input signal ( ). The error signal ( ) is defined as The estimates for ( ) and ( ) can be computed as but they can be updated using the following recursive equations in practical: The RSOR algorithm solves the normal equation (9) using one SOR iteration during a sampling interval as follows Koçal, 2012, 2017).
Equations (11), (12) and (13) are used for implementation of the RSOR algorithm. The parameter is known as the relaxation parameter and should be taken between 0 < < 2 for the stability of the SOR iteration. If > 1 is chosen, the SOR iteration faster converges than the Gauss-Seidel iteration (Golub and Van Loan, 1996).

Quantitative Expression of The Leakage Phenomenon for The RSOR Algorithm
The RSOR algorithm in scalar updating form (13) can be represented in vector updating form as follows (Hatun and Koçal, 2012).
which is based on the following splitting as given in the classical SOR method (Golub and Van Loan, 1996).
A convergence analysis for the RSOR algorithm can be performed in a more tractable manner by decomposing the symmetric correlation matrix ( ) to its lower triangle, diagonal and upper triangle parts as follows Defining the parameter estimation errors as a vector: and subtracting the correct parameter vector, from the both sides of (14), the following iteration is obtained for the parameter error vector after some arrangements (Hatun and Koçal, 2012).
By rewriting the vector quantity in the second term as and substituting in (18) the following result is obtained for the parameter error vector.
A similar analysis gives the following iteration for the RI algorithm (Ahmad et al., 2011b, Salman et al., 2017).
The following equation is also given for the RLS algorithm (Ciochină et al., 2009).
Considering (18), the error is represented as The usage of (20) in (24) gives an estimate of the leakage term for the RSOR algorithm based on the data used and the filter parameters, and this term ( ), which is caused by ( ), leaks to the output signal of the filter On the other hand, using (6) and (10), the vector ( ) is represented as and thus, the normal equation (9) is rewritten as This equation also gives the parameter error vector in (22) for the RLS algorithm. Assuming is close to 1 and is high enough, the following assumption can be written.
Using this assumption, when time index goes to infinity or a large value, as → ∞, the normal equation (27) becomes and consequently, the parameter estimations vector ̂( ) is close to correct parameters , namely, ̂( ) ≅ . Thus, the filter output gives the system output as ̂( ) ≅ ( ), and the measurement noise can be close to the error signal as ( ) ≅ ( ). Thus, according to the leakage definition in (24), the measurement noise does not produce a leakage. Under the assumption (28), the parameter error iteration (20) is reduced to If < 1 or is not high enough, the second term in (27) is not equal to zero vector, and therefore, the time-averaged normal equation (27) cannot be reduced to (29) and the parameter error iteration cannot be reduced to (30). According to (17), the parameter estimations have fluctuations around the correct parameter values. Thus, according to the leakage definition in (24), the measurement noise ( ) causes to a leakage term ( ). If there is a leakage term ( ) ≠ 0, which leaks to the adaptive filter output as given in (25), the measurement noise cannot be close to the error signal, i.e., ( ) ≠ ( ), and thus, an accurate result cannot be obtained by solving the normal equation. The leakage signal is affected from low values, and higher values of (Ciochină et al., 2009, Ahmad et al., 2011b. If no leakage occurs, then the system identification process works properly, thereby letting ( ) = 0.

Analysis of The Leakage Signal for The RSOR Algorithm
Taking statistical expectation of the normal equation (27), the following is written and using the assumption in (28), this equation can be reduced to Based on the independence assumption, which considers adaptive parameters are statistically independent from the input signal (Haykin, 2002), the following approximation can be written By performing a statistical expectation analysis, it can also be shown that the RSOR algorithm produces {̂( )} = as → ∞, asymptotically. Taking statistical expectation of (20), the parameter error iteration for the RSOR algorithm can be written as Assuming the expected value of the correlation matrix estimation converges to its deterministic value at steady-state: the following approximations can be written Using these approximations in (34), the following result is obtained.
Under the assumption in (28), and considering the input signal ( ) is uncorrelated with the measurement noise ( ), the iteration (33) is reduced to If all eigenvalues of the RSOR algorithm are smaller than 1, this iteration converges to zero vector asymptotically as → ∞ Koçal, 2012, 2017). Thus, the expected value of the parameter estimates for the RSOR algorithm converges to the correct parameter vector (3), namely, {̂( )} = as → ∞, asymptotically.
A stochastic convergence analysis of the leakage signal can be performed by taking statistical expectation of the leakage term in (24). Considering the independence assumption, i.e., the adaptive parameters are statistically independent form the input signal, the following approximation can be written Because of the input signal is zero mean, the leakage term converges to zero if the parameter error vector convergences to zero vector or not zero vector. A more complicated noise sequence can be also considered as the measurement noise ( ), for example a contaminated Gaussian noise with its mean as where ( ) is known as Bernoulli-Gaussian impulsive noise sequence, the ( ) and ( ) are independent Gaussian noise sequences with zero means and variances 2 and 2 , respectively. The ( ) is a switching sequence which has zeros and ones, and modelled by Bernoulli random process with probability

SIMULATION RESULTS
In this section, adaptive filtering algorithms were used to identify the impulse response of A low frequency sine wave, ( ) = 0.5 sin(0.01 ) with a sampling interval = 1 sec., was added to the output signal measurements in the first three simulations to see if the measurement noise can be recovered from the estimation error and to observe the leakage signal. Thus, it can be observed whether the error is close to the noise. In the simulations, the RSOR algorithm was compared with the RLS and RI algorithms. The following initial values were used for initial phase of the algorithms: RLS: In the first simulation, the effect of the leakage phenomenon were analyzed and compared for the RLS, RSOR and RI algorithms with different forgetting factors and constant filter length ( = 8). The obtained results were given in Figure 2 for the RLS algorithm, in Figure 3 for the RSOR algorithm, and in Figure 4 for the RI algorithm. The leakage values of the used algorithms for different values (with = 8) were compared in Table 1 In the second simulation, the effect of the leakage phenomenon were analyzed and compared for the RLS, RSOR and RI algorithms with different filter lengths and constant forgetting factor ( = 0.96). The obtained results were given in Figure 5 for the RLS algorithm, in Figure 6 for the RSOR algorithm, and in Figure 7 for the RI algorithm. The leakage values of the used algorithms for different values (with = 0.96) were given in Table 2.
In the third simulation, the effect of the leakage phenomenon were analyzed and compared for the RSOR algorithm with different values of relaxation parameter with constant filter length ( = 8) and constant forgetting factor ( = 0.9). The obtained results were given in Figure 8. The leakage values of the used algorithms for different values (with = 0.9 and = 8) were presented in Table 3.     Figure 9 for the RLS algorithm, in Figure 10 for the RSOR algorithm, and in Figure 11 for the RI algorithm. The leakage values of the algorithms for different  values were compared in Table 4.  In the fifth simulation, a Gaussian noise ( ) = + ( ) with mean = 1 and variance 2 = 0.25 was added to the output signal measurements. The same initial values were used with the previous simulation. The error signal ( ) and the leakage signal ( ) = ( ) − ( ) were given in Figure 12 for the RLS algorithm, in Figure 13 for the RSOR algorithm, and in Figure 14 for the RI algorithm. The obtained leakage values of the algorithms for different values were compared in Table 5.   Figure 15 for the RLS algorithm, in Figure 16 for the RSOR algorithm, and in Figure  17 for the RI algorithm. The leakage values of the algorithms for different values were given in Table 6.  The following findings were obtained from the simulation results for the RSOR algorithm: -A sine wave is used as the measurement noise in the first three simulations, and it was observed that the measurement noise can be recovered from the estimation error when the variance of the leakage signal is low. -The value of leakage increases if the forgetting factor and the filter length are increase.
-The value of leakage decreases if the relaxation parameter is increases.
-The leakage value of the RSOR algorithm is lower than the RLS algorithm and higher than the RI algorithm. The RI algorithm has a decreasing step size parameter and therefore produces a lower leakage value in steady state. -A zero mean Gaussian noise sequence was used as the measurement noise in the fourth simulation and similar results were obtained. -In the fifth simulation, a non-zero mean Gaussian noise was used as the measurement noise and similar results were obtained. However, higher leakage values were obtained in all algorithms used compared to the fourth simulation. -A zero mean impulsive noise sequence was used as the measurement noise in the sixth simulation and similar results were obtained. But, although a measurement noise with a higher variance was used, lower leakage values were obtained in all algorithms compared to the fifth simulation.

CONCLUSIONS
In this paper, a quantitative representation for the estimation of the leakage phenomenon in the RSOR algorithm was presented. The leakage analysis was performed in the system identification setup. It was shown that the quantity of the leakage is proportional to the forgetting factor, the filter length, and the relaxation parameter. The theoretical results were verified by computer simulations. The obtained results have shown that the forgetting factor