In this post on ServerFault, they say you can't do it all through the ssh config file. But you can do it with the ssh config and your .bash_profile or whatever the terminal nerds call it.
in the ssh config file add
Host dev
Hostname server.com
User joe
then in your .bash_profile add an alias
alias domain1="ssh dev -t 'cd domains/domain1; bash'"
Here the dev refers to what you set up in the config file.
In the Terminal, just type domain1, you will be asked to put in your password and will go straight to the directory. Make a new alias for all your domains and it will make logging in to each one super easy.