A NEW DESIGN OF TANGENT HYPERBOLIC FUNCTION GENERATOR WITH APPLICATION TO THE NEURAL NETWORK IMPLEMENTATIONS

A CMOS hyperbolic tangent function generator circuit suitable for the implementation of analog neural networks is presented. In order to obtain an accurate yet simple circuit realization, a judiciously chosen symmetrical Pade approximation of the hyperbolic tangent function is proposed. As an illustrative application, we set up an application in which the proposed circuit is used as the nonlinear block of a two-layer neural network. Simulation results using Spectre Simulation tool in Cadence design environment with 0.18µm CMOS process verify proper operation of the proposed circuit as well as the neural network built around. These results demonstrate the validity of the theoretical analysis and the feasibility of the proposed circuit.


Introduction
This An accurate and high-speed nonlinear function generation is highly demanded in many areas of electronics that require computation, instrumentation and control system design [1][2][3]. Conventionally function generation is realized by using look-up table (LUT) [4], polynomial expansion and approximation methods. [5][6]. One of the major disadvantage of the LUT method is that it requires many hardware resources, thus this is solely used for digital implementations. On the other hand, methods relying on approximations such as Taylor and Padé approximants are mainly intended to the analog circuit realizations, which can be classified into two main groups; circuits realized with transistors operating in sub-threshold mode, usually destined for low frequency applications [7], and circuits using transistors operating in strong inversion region [8]. Besides the well-known limitations of the circuits designed with transistors operating in sub-threshold, the most important limitation of all these solutions is the limited accuracy. In order to increase the accuracy, the approximation order should be increased at the cost of increasing circuit complexity. Obviously, this fact leads to the increased chip area, which makes the circuits unattractive for VLSI design. One of the commonly used nonlinear function is the hyperbolic tangent function, which can be expressed depending on exponential function. In this paper, it is aimed to design hyperbolic tangent function generator with high accuracy and minimal circuit complexity using an appropriate approach. 1 Faculty of Electrical and Electronics Engineering, Istanbul Technical University, 34469 Maslak, Istanbul, Turkey, For this purpose, circuit realization is provided by using Padé approximation considering the following design criteria: i) Since the hyperbolic tangent function generator is realized in strong inversion region, it will operate in a much wider frequency range than the sub-threshold circuits proposed in the literature [9]. In neural network applications, signals are usually applied to neuron in the form of pulses, or spikes, which requires wide bandwidth operation of the neurons. These requirements may be difficult to achieve with sub-threshold circuits.
ii) The degree of approximation should be very high to increase accuracy in analog circuits. This causes the design complexity for the implementation of circuit. For this reason, it is necessary to recommend a special approximation for hyperbolic tangent function, rather than a general approximation (Taylor, Padé etc.) for the desired function.
One of the application area of the hyperbolic tangent function is the analog neural network (ANN) implementations. In ANN circuits, one of the most common function used to perform the activation process is a S-shaped function and produces outputs in the scale of -1 to 1, which is perfectly realized with the hyperbolic tangent function. There are many circuit examples designed to perform the hyperbolic tangent function [10][11]. In this work, an ANN that performs character recognition is also used to demonstrate the usefulness of the proposed circuit. For this purpose, the proposed tangent hyperbolic circuit is used to realize the activation process. All simulation results are obtained in UMC 0.18µm CMOS technology. The paper is planned as follows: The realization of the hyperbolic tangent function and its circuit implementation is given in Section 2 and Section 3 respectively. Simulation results of the proposed circuits and comparison with the previous works are presented in Section 4. The application of the proposed hyperbolic tangent function generator is described in Section 5. Finally, conclusion part is presented in Section 6.

The Design of Hyperbolic Tangent Function Generator
As it is well known from the literature, hyperbolic tangent function can be expressed in terms of the exponential function as follows: Besides this, exponential function has very effective and well-known approximations [12][13]. Among these, Taylor and Padé approximations constitute two mainstreams in the design of exponential function generators, as the former is a polynomial type approximation, where, m and n are used to define the nominator and denominator degrees of the Padé approximant, respectively [14]. It should be noted that the choice of the polynomial or rational type approximation affects the performance of the resulting function generator drastically. In general, the use of Padé approximant leads to more complicated but more accurate systems, as the rational function approximates the fast variation of the exponential function for finite x values, thanks to the existing denominator term. The design of hyperbolic tangent function generator circuit is considered based on the following two criteria: i) The circuit complexity of the resulting function generator will depend on the mathematical complexity of the approximating function, that is the orders of the numerator and denominator polynomials in Eqn. (3). ii) In addition to the simplicity of the circuit design, the accuracy of the function generator's response over wide input and output range is also very important. In this paper, the proposed realization of the hyperbolic tangent function is based on the expression in Eqn. (1) and exponential Padé approximants defined in Eqn. (3). By substituting the approximate expression in (3) into (1) and rearranging of the terms, the following simple approximation of the hyperbolic tangent function is obtained: Note that the proposed approximation is obtained by considering the following realization of the involved exponential terms in Eqn. (1) as follows: where the first and the second product terms are symmetrical exponential Padé approximants of order [m/n] and [n/m], respectively. It should be noted that owing to the use of the symmetrical Padé approximants, the individual errors of each approximant cancel out each other and leads to more accurate approximation. On the other hand, this choice leads to mathematically simpler expression, which in turn leads to simpler circuit realization. These advantages will be discussed further in Section 2.2.

Fig.1 Comparison of Padé approximation with symmetrical pairs, arbitrary Padé approximation and Taylor series approximation
The Matlab simulation results obtained using Taylor and Padé approximants are shown in Fig.1. In order to obtain a meaningful comparison, the degrees of nominator and denominator polynomials of the different approximations are taken as equal. As it can be easily seen from these results, the symmetrical Padé approximation gives a much more accurate result than Taylor approximation in a wide range.

Advantages of the Proposed Approach
Due to the nature characteristics of the hyperbolic tangent function, symmetrical rational Padé pairs are selected for approximating to the hyperbolic tangent function, and obtained both a much more accurate result and a simpler circuit implementation. Symmetrical Padé approximants can be expressed as follows: as a result of which, hyperbolic tangent function assumes the following form: Assuming that even and odd parts of the polynomial A(x) are respectively E(x) and O(x), that is , the approximation of the hyperbolic tangent function is obtained as follows: Therefore, all even order terms in the numerator and all odd order terms in the denominator are missing. Therefore, the use of symmetrical Padé pair may lead to the simplest algebraic approximation compared to approximations composed using all other Padé combinations. On the other hand, in order to demonstrate the usefulness of the proposed method, the error function of the proposed approximation can be obtained as follows: where Ɛ1 and Ɛ2 are the error functions of the Padé approximants. Similarly, the error function of the hyperbolic tangent function can be calculated as follows: It should be also noted that, the error functions 1 and 2 are symmetrical for x =0. Therefore, 1 and 2 cancel each other for small x and tanh remains small. Although for large values of x, the error functions lose their symmetric properties, the 1 +2 still remains smaller than either 1 or 2. The corresponding errors are given in Fig.2  Therefore, although the error is small in the case of symmetrical Padé approximants, still the values of the parameters m and n is important and affects the approximation error. Obviously, there is a trade-off between the complexity of the approximating function and the circuit complexity. The value n+m can be considered to estimate the complexity of the Padé approximation. For more complicated function, i.e for larger values of m+n, the approximation error is smaller, but the resulting circuit is more complicated. However, as m and n are increased, the rate of error reduction may not be as high as expected. To be specific, let us consider two different cases, in the first one of which is used Padé approximants with m=2, n=1 and in the second one of which with m=3, n=1. Therefore, it is compared the errors of the different symmetrical Padé approximants in Fig.3. For a limited input range, the symmetrical Padé pair for m=2, n=1 approximates hyperbolic tangent function with smaller error compared to the more complicated approximation, m=3, n=1. This provides an important advantage as the resulting circuit would be considerably simpler. Obviously, for larger input range, more complicated approximation yields better results, but having more complicated circuit realization. Nevertheless, the degrees of the symmetrical Padé approximants in Eqn. (4) are set to m=2 and n=1. With these parameters, the proposed approximation for the hyperbolic tangent function is obtained as follows: 15 6 15 ) tanh(

Implementation of the Hyperbolic Tangent Function Generator
Many squarer circuits proposed in the literature realizes the following function [15][16][17][18] (14) where IB is a scaling biasing current. The architecture of the proposed hyperbolic tangent function generator is depicted in Figure 4. The structure consists of two current mode squarers, each realizing the function in Eqn. (14) and a current mode analog multiplier, which realizes the following function: Therefore, by comparing Eqns. (16) and (17), it is seen that the circuit is indeed realizes approximate hyperbolic tangent function for x=Iin/IB and y=Iout/IB. As it can be seen from Fig.4, the circuit adopts three input currents. These currents are realized using a current steering/replicator block implemented using simple current mirrors. Concurrently, the scaling coefficient of 1/6 can be realized by adjusting the transistor sizes of the involved current mirrors, without having to use any additional circuit.  generator for m=2, n=1 (β1=15 and β2=15/6).
The hyperbolic tangent function generator has been redesigned using an alternative symmetrical Padé pair with the following parameters: m=3 and n=1 for comparative purposes. The hyperbolic tangent function obtained using the approach employed in Section 2. is given as follows: Copyright

Fig.5 Architecture which realizes hyperbolic tangent function generator with the symmetrical Padé pair for m=3, n=1
For x=Iin/IB and y=Iout/IB, the hyperbolic tangent function can be obtained in terms of currents as follows: If the hyperbolic tangent function is designed using squarer and multiplier subcircuits, the architecture of the circuit is obtained as in Fig.5. As can be seen from this figure, this circuit consists of 5 squarer and 2 multiplier subcircuits and therefore this realization in this way requires more complicated circuit.

Current-Mode Squarer Subcircuit
As it can be seen from the Fig.6, it is considered the squarer subcircuit-1 which is well-known circuit from the literature and operates in current mode [17]. This circuit is based on trans-linear loop (M1-M4), and all transistors used in the circuits are assumed to operate in saturation mode. According to this, the current equation for saturation region can be given as follows: where the transistor parameters have the usual meanings. By applying Kirchhoff's voltage law around the loop consisting of transistors M1-M4, using the square relationship given in Eqn. (20) and assuming that transistors are perfectly matched, we obtain the following relation for the transistors' drain currents: It is assumed that, the transistors have the same transconductance value. By substituting (20) into (21) and considering IDS1=IDS2=IB1, the output current is obtained as follows: where α1=1 and β1=15. Note that since the IB is the biasing current, the value of β1xIB is provided easily by scaling the corresponding current. The output of the second squarer circuit can be also obtained as follows: where α2=1and β2 = 15/6, this value can be obtained by appropriate scaling of the biasing current.

Multiplier Subcircuit
This subcircuit which performs the multiplication operation, can easily be realized with the following equation [19]: As can be easily seen also from the Fig.6, this circuit is based on two trans-linear loops and achieves squaring operations by using these loops. The first loop formed by the M21-M24 transistors provides the squarer function (X+Y) 2 and the second loop formed by the M25-M28 transistors provides the squarer function (X-Y) 2 . The equations performed by the first and second loops can be given as follows, depending on the current values:

The Comparison and Simulation Results
In order to demonstrate the usefulness of the proposed circuit, the hyperbolic tangent function generator is designed for implementing in 0.18µm UMC technology, under supply voltage of ±1V. For the proper operation, the biasing current is chosen for IB=5µA. In order to evaluate simulation results, it is compared the nonlinear transfer characteristic of the circuit with the ideal hyperbolic tangent function, defined as I0ut=α. tanh(βIin). As it can be seen from Fig.7a, the ideal characteristic obtained for α=1.088 and β=7800 is quite compatible with the simulation results. The average error is found to be as 3%. According to these results, it can easily be said that the proposed hyperbolic tangent function generator gives a much more accurate result compared to the other analog circuit generator in reference [9,22], at the cost of having more complicated circuitry. Besides this, in order to provide a fair comparison with other circuits in the literature in terms of the area, accuracy and power consumption, it is obtained the circuit layout as shown in Fig.7b. Crucial performance figures of the proposed circuit and the recently presented works are tabulated in Table 1.

180-nm CMOS
As it is known from the literature, in such circuits, digital only solutions (LUT) suffers from large chip area and from large power consumption [20]. As a remedy, hybrid realization has been proposed, which achieves better accuracy while providing acceptable chip area and power consumption [21]. Accordingly, as shown in Table 1, the proposed circuit offers a more advantageous solution in terms of the area and the accuracy compared to the hybrid circuit. Beside this, it is clear that the proposed circuit provides much better error performance compared to the other analog circuits. In addition, it is also found the power consumption of the proposed circuit as 244µW.

Application
In today state-of-the art technologies, the realization of artificial neural networks (ANN) featuring large number of neurons and huge number of interconnection between them poses significant challenges to the integrated circuit designers.  Therefore, many studies on the application and implementation of the ANNs are presented in the literature [22][23]. The general scheme of ANN is given in Fig.8a. In order to illustrate the usefulness of this work, the proposed hyperbolic tangent function generator is used as an activation function in ANN circuit in Fig.8a. The required weighted summing operation are realized using current mirrors in which the weights are set by adjusting transistor aspect ratios. The resulting circuit is shown in. Fig.8b. This circuit with one hidden layer operates in current mode and is used for the character recognition from the inputs set up to represent the letters of the alphabet from the letter "A" to the letter "E". These characters are expressed in a 5X3 matrix and represented by black and white patterns. The representation of these letters are shown in Fig.9. The training of the network and derivation of the proper weights are derived using MATLAB. These values are inserted to the proposed CMOS circuit using Cadence simulation program. Thus, by applying the input set that provides each character, the characters corresponding to the output values are obtained. It has also been tested whether the ANN is capable of generalization and fault tolerant. For this purpose, some of the incorrect input values are deliberately applied to the network and it is determined that the network generates correct results despite this error. Therefore, it can be said that for this ANN, where the proposed CMOS circuit performs the activation function, it is a fault tolerant circuit.

Conclusion
In this study, a new circuit that provides tangent hyperbolic function characteristics is proposed using Padé approximation. With the proposed approach, it is aimed to realize the tangent hyperbolic function algebraically simple. Thus, the circuit implementation of the tanh function is also reduced to a much simple form. In order to demonstrate the usefulness of the proposed circuit, this circuit is used as an activation function in an ANN with 15 inputs, 5 outputs and performing character recognition. The proposed circuit is based on the translinear principle and consists of sub circuits such as squarer and multiplier. By comparing the proposed circuit with those given in the literature, it can easily be said that the circuit still covers a small area while providing an accuracy of 3%. The proposed circuit has been applied to a simple character recognition problem, which is a well-known problem in the literature, and its usefulness and applicability have been proven.