An index basically sorts your data on the given columns and then stores that order, so when you want to find an item, the database can optimize by using binary search (or some other optimized way of searching), rather than looking at each individual row.
Thus, if the amount of data you are searching through is large, you will absolutely want to add some indexes.
Most databases have a tool to explain how your query will work (for db2, it's db2expln, something similar probably for sqlserver), and a tool to suggest indexes and other optimizations (db2advis for db2, again probably something similar for sqlserver).
