Say I have a table or 2 that contains data that will never or rarely change, is there any point of trying to cache those data? Or will the EF context cache that data for me
You can check EF caching provider but be aware that caching in this way is performed strictly on query basis - so you must use the same query all the time to get cached data. If you use another query it will first be executed to be considered as cached and then you use it again to hit the cache. If you want to avoid this and cache data with ability to run any query on cached collection you must roll on your own solution (simply load data to list and keep it somewhere). When you load entities to cached list make sure that you turn off proxy creation (lazy loading and change tracking).
Caching per context instance really works but using context itself as a cache is pretty bad choice - in most scenarios I would call it EF anti-pattern. Use context as unit of work = do not reuse context for multiple logical operations.