Say I have a table or 2 that contains data that will never or rarely change, is there any point of trying to cache those data? Or will the EF context cache that data for me
The EF context will cache "per instance". That is, each instance of the DbContext keeps it's own independent cache of objects. You can store the resulting list of objects in a static list and query it all you like without returning to the database. To be safe, make sure you abandon the DbContext after you execute the query.
var dbContext = new YourDbContext();
StaticData.CachedListOfThings = dbContext.ListOfThings.ToList();
You can later use LINQ to query the static list.
var widgets = StaticData.CachedListOfThing.Where(thing => thing.Widget == "Foo");
The query executes the in-memory collection, not the database.