Estimation of Approximating Rate for Neural Network in Lpw Spaces

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.


Introduction
Artificial neural networks have been extensively applied in various fields of science and engineering.Why is so mainly because the feedforward neural networks FNNs have the universal approximation capability 1-13 .A typical example of such universal approximation assertions states that, for any given continuous function defined on a compact set K of R d , there exists a three-layer of FNN so that it can approximate the function arbitrarily well.A three-layer of FNN with one hidden layer, d inputs and one output can be mathematically expressed as where 1 ≤ i ≤ m, θ i ∈ R are the thresholds, w i w i1 , w i2 , . . ., w id T ∈ R d are connection weights of neuron i in the hidden layer with the input neurons, c i ∈ R are the connection strength of neuron i with the output neuron, and σ is the activation function used in the network.The activation function is normally taken as sigmoid type; that is, it satisfies σ t → 1 as t → ∞ and σ t → 0 as t → −∞.Equation 1.1 can be further expressed in vector form as Universal approximation capabilities for a broad range of neural network topologies have been established by researchers like Cybenko 1 , Ito 5 , and T. P. Chen and H. Chen 6 .
Their work concentrated on the question of denseness.But from the point of application, we are concerned about the degree of approximation by neural networks.
For any approximation problem, the establishment of performance bounds is an inevitable but very difficult issues.As we know, feedforward neural networks FNNS have been shown to be capable of approximating general class of functions, including continuous and integrable ones.Recently, several researchers have been derived approximation error bounds for various functional classes see, e.g., 7-13 approximated by neural networks.While many open issues remain concerning approximation degree, we stress in this paper on the issue of approximation of functions defined over −1, 1 d by FNNS.In 10 , the researcher took some basics tools from the theory of weighted polynomial of functions The weight function is ω x exp −Q x , under certain assumptions on the smoothness of functions being approximated and on the activation functions in the neural network, the authors present upper bounds on the degree of approximation achieved over the domain R d .
In this paper, using the Chebyshev Orthogonal series from the approximation theory and moduli of continuity, we obtain upper bounds on the degree of approximation in −1, 1 d .We take advantage of the properties of the Chebyshev polynomial and the methods of paper 10 , we yield the desired results, which can be easily extended to the space R d .

Multivariate Chebyshev Polynomial Approximation
Before introducing the main results, we firstly introduce some basic results on Chebyshev polynomials from the approximation theory.For convenience, we introduce a weighted norm of a function f 14 given by We denote the class of functions for which f p,ω is finite by L p,ω .
For function f : R d → R, the class of functions we wish to approximate in this work is defined as follows: , r is a natural number, and M < ∞.

A Chebyshev Polynomial Approximation of Multivariate Functions
As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory.Using the above notation, we introduce multivariate Chebyshev polynomials: For one-dimension degree of approximation of a function g by polynomials of degree m, one has the following: where P m stands for the class of degree-m algebraic polynomials.From 15 , we have a simple relationship which we will be used in the following.Let g be differentiable, then we have , and the de la Valle Poussin Operators is defined, that is, Furthermore, we can simplify V n f, t as follows: where

2.8
A basic result concerning Valle Poussin Operators V m f, t is 2.9 Now we consider a class of multivariate polynomials defined as follows: Hence, we have the following theorem.
Proof.We consider the Chebyshev orthogonal polynomials T m x , and obtain the following equality from 2.7 : where f s,i −1,1 d f x T s x i ω x i dx i .Hence, we define the following operators: where

2.14
where V 0 is the identity operator.Let We view V i,m i g as a one-dimensional function x i .Using 2.4 , 2.5 , and 2.6 , we have

2.15
Letting r i r, m i m, i 1, . . ., d, if m > r r − 1 , we get from 2.15 , 2.13 , 2.14 and the

2.16
In order to obtain a bound valid for all m, for m ≤ r r − 1 , we always have the trivial bound we conclude an inequality of the desired type for every m.This theorem reveals two things: i for any multivariate functions f ∈ Ψ r,d p,ω , there is a polynomial P ∈ P m that approximates f arbitrarily well in L p ω , ii quantitatively, the approximation accuracy of a polynomial P ∈ P m can attain the order of m −r , where m is the dimension of multivariate polynomial, and r is the smoothness of the function to be approximated.

Approximation by Feedforward Neural Networks
We consider the approximation of functions by feedforward neural networks with a ridge functions.We define the approximating function class composed of a single hidden layer feedforward neural network with n hidden units.The class of function is where φ x satisfy the following assumptions.
1 There is a constant We define the distance from where F, G are two sets in L p ω .We have the following results.Proof.Firstly, we consider the partial derivative where we derive 3.7 by using 3.6 , the mean value theorem of integral, i.e., there is a η ∈ 0, h|s • x| , such that A s h φ |s| x h |s| φ |s| b η and the moduli of continuity ω g, h sup |t|≤h |f x t − f x |.
From the definition of dist F, G, L p,ω and 3.7 , we have

3.8
The last step ω φ |s| , h can be made arbitrarily small by letting h → 0.
Using the Theorems 2.1 and 3.1, we can easily establish our final result.
This theorem reveals two things: i for any multivariate functions f ∈ Ψ r,d p,ω , there is a single hidden layer feedforward neural network N ∈ F n with n hidden units that approximates f arbitrarily well in L p ω .That is, the feedforward neural networks can be used as the universal approximator of functions in Ψ r,d p,ω ; ii quantitatively, the approximation accuracy of a mixture network of the form 3.1 can attain the order of n −r/d , where d is the dimension of input space, and r is the smoothness of the function to be approximated.

Conclusion
In this work, the approximation order of feedforward neural networks with the form 3.1 has been studied.In terms of smoothness of a function, an upper bound estimation on approximation precision and speed of the neural networks is developed.Our research reveals that the approximation precision and speed of the neural networks depend not only on the number of hidden neurons used, but also on the smoothness of the functions to be approximated.The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

5 where
w d φ w • x b x s φ |s| w • x b , 3.|s| s 1 • • • s d ,and x s d i 1 x s i i .Thus φ s b x s φ |s| b .For any fixed b and |x| < ∞ here |x| d i 1 x i , we consider a finite difference of orders s h,x φ b 0≤l≤s −1 |l| C l s φ hl • x b