Can neural networks approximate any function given enough hidden neurons?

Question

I understand neural networks with any number of hidden layers can approximate nonlinear functions, however, can it approximate:

f(x) = x^2

Answer 1

There are theoretical limitations of Neural Networks. No neural network can ever learn the function f(x) = x*x Nor can it learn an infinite number of other functions, unless you assume the impractical:

1- an infinite number of training examples 2- an infinite number of units 3- an infinite amount of time to converge

NNs are good in learning low-level pattern recognition problems (signals that in the end have some statistical pattern that can be represented by some "continuous" function!), but that's it! No more!

Here's a hint:
Try to build a NN that takes n+1 data inputs (x0, x1, x2, ... xn) and it will return true (or 1) if (2 * x0) is in the rest of the sequence. And, good luck. Infinite functions especially those that are recursive cannot be learned. They just are!