I have several database tables that just contain a single column and very few rows, often just an ID of something defined in another system. These tables are then reference
I would say the following criteria must be met:
your natural key must be absolutely, positively, no-exceptions-allowed, unique (things like names, social security numbers etc. usually seem to be unique - but really aren't)
your natural key should be as small as an INT, e.g. not significantly more than 4 bytes in size (don't use a VARCHAR(50) for your PK, and especially not for your clustering key in SQL Server !)
your natural key ought to be stable, e.g. never change (OK, with ISO country codes, this is almost a given - except when countries like Yugoslavia or the USSR collapse, or other like the two Germanies unite - but that's rare enough)
If those conditions are met, you can consider a natural key as your PK - but that should be the 2% exception in all your tables - not the norm.