We have several empty abstract class in our codebase. I find that ugly. But besides this very stupid reason (ugliness), should I refactor it (into empty interface e.g.) ?
Interfaces are preferable in this case because it makes your code more robust for maintenance. That is, you can only extend a single class but you may implement many interfaces.
If there is absolutely no direct effect right now I would not touch it. If the maintenance event turns up that requires you to make a change then I would refactor it since I am already in the code.
In other words if it ain't broke don't fix it.