I can\'t seem to set a new $PATH such that it is used when executing commands via ssh user@host command. I have tried adding export PATH=$PATH:$HOME/new_
As grawity said, ~/.bashrc is what you want, since it is sourced by non-interactive non-login shells.
I expect the problem you're having has to do with the default Ubuntu ~/.bashrc file. It usually starts with something like this:
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
You want to put anything for non-interactive shells before this line.