Journal of Applied Mathematics

Volume 2012 (2012), Article ID 636078, 8 pages

doi:10.1155/2012/636078

Research Article

Estimation of Approximating Rate for Neural Network in Spaces

Jian-Jun Wang,¹ Chan-Yun Yang,² and Jia Jing¹

¹School of Mathematics & Statistics, Southwest University, Chongqing 400715, China

²Department of Mechanical Engineering, Taipei Chengshih University of Science and Technology, No.2 Xue-Yuan Rd., Beitou, Taipei 112, Taiwan

Received 13 February 2012; Accepted 27 March 2012

Academic Editor: Juan Manuel Peña

Copyright © 2012 Jian-Jun Wang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

1. Introduction

Artificial neural networks have been extensively applied in various fields of science and engineering. Why is so mainly because the feedforward neural networks (FNNs) have the universal approximation capability [1–13]. A typical example of such universal approximation assertions states that, for any given continuous function defined on a compact set of , there exists a three-layer of FNN so that it can approximate the function arbitrarily well. A three-layer of FNN with one hidden layer, inputs and one output can be mathematically expressed as

where ,   are the thresholds, are connection weights of neuron in the hidden layer with the input neurons, are the connection strength of neuron with the output neuron, and is the activation function used in the network. The activation function is normally taken as sigmoid type; that is, it satisfies as and as . Equation (1.1) can be further expressed in vector form as

Universal approximation capabilities for a broad range of neural network topologies have been established by researchers like Cybenko [1], Ito [5], and T. P. Chen and H. Chen [6]. Their work concentrated on the question of denseness. But from the point of application, we are concerned about the degree of approximation by neural networks.

For any approximation problem, the establishment of performance bounds is an inevitable but very difficult issues. As we know, feedforward neural networks (FNNS) have been shown to be capable of approximating general class of functions, including continuous and integrable ones. Recently, several researchers have been derived approximation error bounds for various functional classes (see, e.g., [7–13]) approximated by neural networks. While many open issues remain concerning approximation degree, we stress in this paper on the issue of approximation of functions defined over by FNNS. In [10], the researcher took some basics tools from the theory of weighted polynomial of functions (The weight function is ), under certain assumptions on the smoothness of functions being approximated and on the activation functions in the neural network, the authors present upper bounds on the degree of approximation achieved over the domain .

In this paper, using the Chebyshev Orthogonal series from the approximation theory and moduli of continuity, we obtain upper bounds on the degree of approximation in . We take advantage of the properties of the Chebyshev polynomial and the methods of paper [10], we yield the desired results, which can be easily extended to the space .

2. Multivariate Chebyshev Polynomial Approximation

Before introducing the main results, we firstly introduce some basic results on Chebyshev polynomials from the approximation theory. For convenience, we introduce a weighted norm of a function [14] given by

where , is multivariate weighted function, , , , . We denote the class of functions for which is finite by .

For function , the class of functions we wish to approximate in this work is defined as follows:

where , , , is a natural number, and .

2.1. A Chebyshev Polynomial Approximation of Multivariate Functions

As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory. Using the above notation, we introduce multivariate Chebyshev polynomials: , , . Evidently, for any , , we have

For , , let , then we have the orthogonal expansion .

For one-dimension degree of approximation of a function by polynomials of degree , one has the following:

where stands for the class of degree-m algebraic polynomials. From [15], we have a simple relationship which we will be used in the following. Let be differentiable, then we have

Let , and the de la Valle Poussin Operators is defined, that is,

Furthermore, we can simplify as follows:

where

A basic result concerning Valle Poussin Operators is

Now we consider a class of multivariate polynomials defined as follows:

Hence, we have the following theorem.

Theorem 2.1. For , let . Then for any , , we have

Proof. We consider the Chebyshev orthogonal polynomials , and obtain the following equality from (2.7):

where . Hence, we define the following operators:

where . Then we have

where is the identity operator. Let , then , . We view as a one-dimensional function . Using (2.4), (2.5), and (2.6), we have

Letting ,  ,  , if , we get from (2.15), (2.13), (2.14) and the inequality ,

In order to obtain a bound valid for all , for , we always have the trivial bound since . Letting , we conclude an inequality of the desired type for every .

This theorem reveals two things: (i) for any multivariate functions , there is a polynomial that approximates arbitrarily well in , (ii) quantitatively, the approximation accuracy of a polynomial can attain the order of , where is the dimension of multivariate polynomial, and is the smoothness of the function to be approximated.

3. Approximation by Feedforward Neural Networks

We consider the approximation of functions by feedforward neural networks with a ridge functions. We define the approximating function class composed of a single hidden layer feedforward neural network with hidden units. The class of function is

where satisfy the following assumptions.

(1)There is a constant such that ,  (2)For each finite , there is a finite constant such that  .

We define the distance from to as

where , are two sets in . We have the following results.

Theorem 3.1. Let condition (1) and (2) hold for the activation function . Then for every , , , and , we have

where

Proof. Firstly, we consider the partial derivative

where , and . Thus .

For any fixed and (here ), we consider a finite difference of orders

where ,   with , So

where we derive (3.7) by using (3.6), the mean value theorem of integral, (i.e., there is a , such that ) and the moduli of continuity .

From the definition of   and (3.7), we have

The last step can be made arbitrarily small by letting .

Using the Theorems 2.1 and 3.1, we can easily establish our final result.

Theorem 3.2. For , we have

This theorem reveals two things: (i) for any multivariate functions , there is a single hidden layer feedforward neural network with hidden units that approximates arbitrarily well in . That is, the feedforward neural networks can be used as the universal approximator of functions in ; (ii) quantitatively, the approximation accuracy of a mixture network of the form (3.1) can attain the order of , where is the dimension of input space, and is the smoothness of the function to be approximated.

4. Conclusion

In this work, the approximation order of feedforward neural networks with the form (3.1) has been studied. In terms of smoothness of a function, an upper bound estimation on approximation precision and speed of the neural networks is developed. Our research reveals that the approximation precision and speed of the neural networks depend not only on the number of hidden neurons used, but also on the smoothness of the functions to be approximated. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

Acknowledgments

This research is supported by Natural Science Foundation of China (no. 11001227), Natural Science Foundation Project of CQ CSTC (no. CSTC,2009BB2306), and the Fundamental Research Funds for the Central Universities (no. XDJK2010B005).

References

G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals, and Systems, vol. 2, no. 4, pp. 303–314, 1989.

J. J. Wang, Z. B. Xu, and W. J. Xu, Approximation Bounds by Neural Networks in L(w, p), vol. 3173 of Lecture Notes in Computer Science, Springer, Berlin, Germany, 2004.

J. J. Wang, B. Chen, and C. Yang, “Approximation of algebraic and trigonometric polynomials by feedforward neural networks,” Neural Computing and Applications, vol. 21, no. 1, pp. 73–80, 2011.

J. J. Wang and Z. B. Xu, “New study of neural networks: the essential order of approximation,” Neural Networks, vol. 23, pp. 618–624, 2010.

Y. Ito, “Approximation of continuous functions on R d by linear combination of shifted rotations of sigmoid function with and without scaling,” Neural Networks, vol. 5, no. 1, pp. 105–115, 1992.

T. P. Chen and H. Chen, “Approximation capability to functions of several variables, nonlinear functions, and operators by radial function neural networks,” IEEE Transactions on Neural Networks, vol. 6, pp. 904–910, 1995.

A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information Theory, vol. 39, no. 3, pp. 930–945, 1993.

M. Leshno, V. Ya. Lin, A. Pinkus, and S. Schocken, “Multilayer feedforward networks with a non-polynomial activation function and approximate any function,” Neural Networks, vol. 6, no. 6, pp. 861–867, 1993.

H. N. Mhaskar, “Neural networks for optimal approximation for smooth and analytic functions,” Neural Computation, vol. 8, pp. 164–177, 1996.

V. Maiorov and R. S. Meir, “Approximation bounds for smooth functions in C(R d)by neural and mixture networks,” IEEE Transactions on Neural Networks, vol. 3, pp. 969–978, 1998.

M. Burger and A. Neubauer, “Error bounds for approximation with neural networks,” Journal of Approximation Theory, vol. 112, no. 2, pp. 235–250, 2001.

V. Kurkova and M. Sanguineti, “Comparison of worst case errors in linear and neural network approximation,” IEEE Transactions on Information Theory, vol. 48, no. 1, pp. 264–275, 2002.

J. L. Wang, B. H. Sheng, and S. P. Zhou, “On approximation by non-periodic neural and translation networks in spaces,” ACTA Mathematica Sinica, vol. 46, pp. 65–74, 2003 (Chinese).

J. J. Wang, C. Yang, and J. Jing, “Approximation order for multivariate Durrmeyer operators with Jacobi weights,” Abstract and Applied Analysis, vol. 2011, Article ID 970659, 12 pages, 2011.

A. F. Timan, Theory of Approximation of Functions of a Real Variable, Macmillan, New York, NY, USA, 1963.