I can't tell what it is that you are trying to do. Perhaps it'd help if you can post some of your relevant source code...
But the fact that it gets crazy when you start scrolling makes me thing you aren't handling the convertView correctly in your adapter getView() method.