AI Applications in C++: How costly are virtual functions? What are the possible optimizations?

Question

In an AI application I am writing in C++,

1. there is not much numerical computation
2. there are lot of structures for which run-time polymorphism is ne

Answer 1

You rarely have to worry about cache in regards to such commonly used items, since they're fetched once and kept there.

Cache is only generally an issue when dealing with large data structures that either:

1. Are large enough and used for a very long time by a single function so that function can push everything else you need out of the cache, or
2. Are randomly accessed enough that the data structures themselves aren't necessarily in cache when you load from them.

Things like Vtables are generally not going to be a performance/cache/memory issue; usually there's only one Vtable per object type, and the object contains a pointer to the Vtable instead of the Vtable itself. So unless you have a few thousand types of objects, I don't think Vtables are going to thrash your cache.

1), by the way, is why functions like memcpy use cache-bypassing streaming instructions like movnt(dq|q) for extremely large (multi-megabyte) data inputs.