RBF Network as Kernel Estimator

RBF neural networks belong to the class of kernel estimation methods. These methods use a weighted sum of a finite set of nonlinear functions (x-c_i) to approximate an unknown function f(x). The approximation is constructed from the data samples presented to the network using the following equation:

,

where h is the number of kernel functions, () is the kernel function, x is the input vector, c is a vector which represents the center of the kernel function in the n-dimensional space, and w_i are the coefficients to adapt the approximating function f(x). If these kernel functions are mapped to a neural-network architecture, a three-layered network can be constructed where each-hidden node is represented by a single kernel function and the coefficients w_i represent the weights of the output-layer.

The type of each kernel function can be chosen out of a large class of functions and in fact, it has been shown more recently that an arbitrary non-linearity is sufficient for representing any functional relationship by a neural network. Gaussian kernel functions are widely used throughout the literature. It has been shown that a small modification to the Gaussian kernel function improves the performance of RBF-networks for classification tasks:

When R equals 0, the kernel function is the classical Gaussian function  (see figure below). A large R creates a flat top of the kernel which more and more approaches the form of a cylinder with increasing R.

The output layer of an RBF network combines the kernel function of all hidden neurons with a linear-weighted sum of these functions. Depending on various parameters, the response of the network can assume virtually all thinkable shapes. Several possible response functions obtained from a network with five hidden neurons by varying the S and the R parameters are displayed below.