Using knife ec2 plugin to create VM in VPC private subnet

Question

Although I've written a fair amount of chef, I'm fairly new to both AWS/VPC and administrating network traffic (especially a bastion host).

Using the knife ec2 plugin, I would like the capability to dynamically create and bootstrap a VM from my developer workstation. The VM should be able to exist in either a public or private subnet of my VPC. I would like to do all of this without use of an elastic IP. I would also like for my bastion host to be hands off (i.e. I would like to avoid having to create explicit per-VM listening tunnels on my bastion host)

I have successfully used the knife ec2 plugin to create a VM in the legacy EC2 model (e.g. outside of my VPC). I am now trying to create an instance in my VPC. On the knife command line, I'm specifying a gateway, security groups, subnet, etc. The VM gets created, but knife fails to ssh to it afterward.

Here's my knife command line:

knife ec2 server create \
    --flavor t1.micro \
    --identity-file <ssh_private_key> \
    --image ami-3fec7956 \
    --security-group-ids sg-9721e1f8 \
    --subnet subnet-e4764d88 \
    --ssh-user ubuntu \
    --server-connect-attribute private_ip_address \
    --ssh-port 22 \
    --ssh-gateway <gateway_public_dns_hostname (route 53)> \
    --tags isVPC=true,os=ubuntu-12.04,subnet_type=public-build-1c \
    --node-name <VM_NAME>

I suspect that my problem has to do with the configuration of my bastion host. After a day of googling, I wasn't able to find a configuration that works. I'm able to ssh to the bastion host, and from there I can ssh to the newly created VM. I cannot get knife to successfully duplicate this using the gateway argument.

I've played around with /etc/ssh/ssh_config. Here is how it exists today:

    ForwardAgent yes
#ForwardX11 no
#ForwardX11Trusted yes
#RhostsRSAAuthentication no
#RSAAuthentication yes
#PasswordAuthentication no
#HostbasedAuthentication yes
#GSSAPIAuthentication no
#GSSAPIDelegateCredentials no
#GSSAPIKeyExchange no
#GSSAPITrustDNS no
#BatchMode no
   CheckHostIP no
#AddressFamily any
#ConnectTimeout 0
    StrictHostKeyChecking no
    IdentityFile ~/.ssh/identity
#IdentityFile ~/.ssh/id_rsa
#IdentityFile ~/.ssh/id_dsa
#Port 22
#Protocol 2,1
#Cipher 3des
#Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
#EscapeChar ~
    Tunnel yes
#TunnelDevice any:any
#PermitLocalCommand no
#VisualHostKey no
#ProxyCommand ssh -q -W %h:%p gateway.example.com
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
    GatewayPorts yes

I have also set /home/ubuntu/.ssh/identity to the matching private key of my new instance.

UPDATE:

I notice the following in the bastion host's /var/log/auth.log:

May  9 12:15:47 ip-10-0-224-93 sshd[8455]: Invalid user  from <WORKSTATION_IP>
May  9 12:15:47 ip-10-0-224-93 sshd[8455]: input_userauth_request: invalid user  [preauth]

Answer 1

I finally resolved this. I was missing the username when specifying my gateway. I originally thought that the --ssh-user argument would be used for both the gateway AND the VM I'm attempting to bootstrap. This was incorrect, username must be specified for both.

knife ec2 server create \
    --flavor t1.micro \
    --identity-file <ssh_private_key> \
    --image ami-3fec7956 \
    --security-group-ids sg-9721e1f8 \
    --subnet subnet-e4764d88 \
    --ssh-user ubuntu \
    --server-connect-attribute private_ip_address \
    --ssh-port 22 \
    --ssh-gateway ubuntu@<gateway_public_dns_hostname (route 53)> \
    --tags isVPC=true,os=ubuntu-12.04,subnet_type=public-build-1c \
    --node-name <VM_NAME>

Just the line containing the update (notice the ubuntu@ in front):

    --ssh-gateway ubuntu@<gateway_public_dns_hostname (route 53)>

I have now gone through and locked my bastion host back down, including removal of /home/ubuntu/.ssh/identity, as storing the private key on the bastion host was really bugging me.

FYI: When setting up a bastion host, the "out of the box" configuration of sshd will work when using the Amazon Linux AMI image. Also, some of the arguments above are optional, such as --ssh-port.