I created a DLL containing a function named \"koduj\". Calling this function by using it inside an Excel worksheet cell returns the desired result. Calling \"koduj\" from VB
I'm guessing from the magnitude of the error that it's the numeric parameter that's going wrong - I would try more explicitly declaring the parameter type in your test VBA routine (probably Integer) and accepting it as that specific type on the C++ side (signed short, in that case).
There's a great Microsoft article about all this at http://msdn.microsoft.com/en-us/library/office/bb687915(v=office.15).aspx.