I don't have to worry about log4j's
performance even if I heavily log
Exactly. Do not optimize until your profiling results tell you to. There are cases, when logging performance is a bottleneck, but you need first to hit that case, then optimize for it.