You are right about your concern - you should not save the refresh token. By doing so, you jeopardize your client's data (and you know the reason; you wrote it in the question).
oAuth is not supposed to work this way.
You should keep the refresh token in-memory.