Fixed it with the following code in my map fragment:
public void onDestroyView() {
super.onDestroyView();
FragmentManager fm = getActivity().getSupportFragmentManager();
Fragment fragment = (fm.findFragmentById(R.id.map));
FragmentTransaction ft = fm.beginTransaction();
ft.remove(fragment);
ft.commit();
}