You can override onBackPressed and check to see if there is anything on the backstack.
@Override
public void onBackPressed() {
int fragments = getFragmentManager().getBackStackEntryCount();
if (fragments == 1) {
// make layout invisible since last fragment will be removed
}
super.onBackPressed();
}