As Knuth said,
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
First, get the code working. Second, verify that the code is correct. Third, make it fast.
Any code change that is done before stage #3 is definitely premature. I am not entirely sure how to classify design choices made before that (like using well-suited data structures), but I prefer to veer towards using abstractions taht are easy to program with rather than those who are well-performing, until I am at a stage where I can start using profiling and having a correct (though frequently slow) reference implementation to compare results with.