First a short overview of my code, i hope its understandable:
ToDoElement with a few variables.
You need to add a DeleteMethod of some sort to the Adapter and remove the item from the Adapter manually, not only remove it from the List.
Each time you refresh the the views using notifyDataSetChanged() the Adapter will call the loop around the List. Your List in the Adapter gets a null value due to the changes you made.