aalib (last release in 2001) is an open source ASCII art library that's used in applications like mplayer. You may want to check out its source code to see how it does it. Other than that, this page describes in more detail about how such algorithms work.
