The problem here is mostly parsing, which would be covered in a compiler course probably in second or third year. Once you can parse expressions to build up a recursive data structure representing the input (called a syntax tree) it's pretty trivial to evaluate such expressions. A recursive decent parser can also evaluate the expression as it goes without actually building a syntax tree.
For a full treatment you'd want a book on compilers, such as the dragon book. Also IIRC the book Programming: Principals and Practice using C++ covers an example like this.
You could also wait for chapter ten of The Art of Computer Programming to be published, which will cover parsing. It's scheduled to be out around 2020.
