Using genetic algorithms for neural networks

Question

Currently I am working on a project that would use genetic algorithms to optimize neural networks. I do realize this is probably not the best way to optimize them, but I'm new to both so I just wanted to try using them.

My plans are as follows (subject to change, a lot). My input neurons would be using a data set that could have just about any positive number (including decimals up to two places, so really they would be floating point numbers), but most likely between 0 to 20000. Because the importance is in how the numbers compare to each other in value rather than how big they are, they would first be divided by the highest number of all the values that would be inputted. They would them be multiplied by weights (any positive or negative float) before going to their hidden layer. Each neuron in the hidden layer would be summing all of it's inputs until they were done being calculated. They would then be run through a logistics function and be outputted.

My environment is Visual studio C++ 2010 Express and I'm using the clr.

My problem lies in the genetic algorithm and how it would work. It would be adjusting the weights. My problem is that when it randomly changes a bit in one of the weights (mutation rate), it could make the weights extraordinarily high or low, causing overflow or some other error when multiplied by the input and added with the others. I have no idea how I would organize my chromosomes either. So, would it just be better to perform randomization by selection weights rather than bits at random and changing them to a random number within a defined range? Basically I am looking for suggestions on how to organize this without causing errors with making values end up too big or too small while maintaining performance.

Thanks, (and sorry if this should be in theoretical computer science, but I thought it wouldn't fit in there)

Answer 1

(Artificial) Neural Networks (ANNs) are notoriously difficult to optimize, and genetic algorithms (GAs) are a reasonably good approach to doing so (mainly because everything else tends to be very limited in how well it can work). Of course there are alternatives that work well too, but they are more complicated and subtle to program and tune correctly (back-propagation with simulated annealing and learning momentum). And I understand your are doing this project mostly to play around with those things.

You might want to look at Evolutionary Neuro-controllers (ENC), this is a field where genetic (or evolutionary) algorithms are employed to train ANNs for complex navigation tasks (e.g. inter-planetary space missions is one of the applications of that I have personally done research on).

For the ANN part, I would suggest that you don't limit yourself to logistics functions (I know that the sigmoid is inspired from biological neurons, but that doesn't mean they are the best all the time). Many other functions exist as well, logistics functions are used in part because they make back-propagation much faster and simpler to do. But, Radial-Basis functions work wonders as well (IMO and from what I have seen, most successful applications of ANNs used Radial-Basis functions, i.e. RBF-NN). Typically, people use either Gaussian functions, hyper-spherical functions and very often, triangular functions (called Fuzzy Networks, another huge class of ANNs).

As for GAs, I would not recommend that type of mutation you are describing (i.e. to flip bits) for the reasons that you mentioned. People don't use this kind of mutation when dealing with real-valued genes. One very easy mutation method is just to decide (with some probability) to mutate an individual, then pick one element of its gene to be mutated and then simply generate a new gene element to replace it using a random number generator (rand()). With this, you can limit the scale of the generated gene elements to avoid problems of turning your individual degenerate (i.e. one completely wrong gene element can make the entire individual useless). What are the genes? Well, for a ANN, typically a large vector containing all the weights of all the neurons in your net. You can guess that people rarely apply GAs if the number of neurons is too large. I would also recommend that you use Tournament Selection for selecting individuals for reproduction. As for cross-over (i.e. mixing two parents to make a child), just keep the order of the weights and pick for each element of the child a weight from either parent at random with equal probability.

I have personally done what I described above and it works very well for certain problems (reduced size and high complexity, i.e. no obvious optimal solution).

Finally, don't expect it to work that easily. Typically, it will require a population size and a number of generation that is much much higher than what you would expect (after all, evolution is a very slow process!). So, don't try a population of 10 individuals and run for 50 generations, and sadly say "OH I guess it doesn't work...". Try more in the order of thousands of individuals in the population and several thousands to a hundred thousand generations, depending on the scale of the problem your are applying it to, of course.

Answer 2

Your problem lies in chromosome representation. It's known as Hamming Cliff Problem. You can use Gray Code for chromosome representation which doesn't have Hamming Cliff Problem