I have a shared /home setup using Perceus Cluster Software (http://perceus.org) for our Cluster. Nodes are using CentOS 6.1 x86_64. /home is shared from the head to the nodes by nfs (NFSv4).
root@head~]$ cat /etc/exports /var/lib/perceus/ 10.10.10.0/255.255.255.0(ro,no_root_squash,async) /home/ 10.10.10.0/255.255.255.0(rw,no_root_squash,no_all_squash,async)
Here is the /etc/fstab on each node (all the same).
... 10.10.10.2:/var/lib/perceus/ /var/lib/perceus/ nfs ro,soft,bg 0 0 10.10.10.2:/home/ /home nfs rw,soft,bg 0 0
/etc/fstab on nodes is a copy of the head/master with identical UID:GID.
I have created key pairs using the following method:
$ cd ~ $ rm -rf .ssh $ mkdir .ssh $ chmod 700 .ssh $ ssh-keygen -t dsa -P "" Generating public/private dsa key pair. Enter file in which to save the key (/home/user/.ssh/id_dsa): Your identification has been saved in /home/user/.ssh/id_dsa. Your public key has been saved in /home/user/.ssh/id_dsa.pub. The key fingerprint is: [SNIPPED] user@head The key's randomart image is: +--[ DSA 1024]----+ [SNIPPED] $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys $ chmod 400 ~/.ssh/authorized_keys
Here is the problem. When I try to ssh into each node, I am getting a "Connection Closed" error. Here is the debugging output.
$ ssh node01 Connection closed by 10.10.10.101 $ ssh node01 -vvv OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to node01 [10.10.10.101] port 22. debug1: Connection established. debug1: identity file /home/user/.ssh/identity type -1 debug1: identity file /home/user/.ssh/id_rsa type -1 debug3: Not a RSA1 key file /home/user/.ssh/id_dsa. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /home/user/.ssh/id_dsa type 2 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.3 .... SNIPPED ... debug2: dh_gen_key: priv key bits set: 139/256 debug2: bits set: 482/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: Wrote 144 bytes for a total of 981 debug3: check_host_in_hostfile: filename /home/user/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug3: check_host_in_hostfile: filename /home/user/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug1: Host 'node01' is known and matches the RSA host key. debug1: Found key in /home/user/.ssh/known_hosts:1 debug2: bits set: 501/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: Wrote 16 bytes for a total of 997 debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug3: Wrote 48 bytes for a total of 1045 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /home/user/.ssh/identity ((nil)) debug2: key: /home/user/.ssh/id_rsa ((nil)) debug2: key: /home/user/.ssh/id_dsa (0x7f79b940f650) debug3: Wrote 64 bytes for a total of 1109 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password ... [SNIPPED]... debug1: Next authentication method: publickey debug1: Trying private key: /home/user/.ssh/identity debug3: no such identity: /home/user/.ssh/identity debug1: Trying private key: /home/user/.ssh/id_rsa debug3: no such identity: /home/user/.ssh/id_rsa debug1: Offering public key: /home/user/.ssh/id_dsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug3: Wrote 528 bytes for a total of 1637 debug1: Server accepts key: pkalg ssh-dss blen 434 debug2: input_userauth_pk_ok: SHA1 fp 46:a2:c3:86........... debug3: sign_and_send_pubkey debug1: read PEM private key done: type DSA debug3: Wrote 592 bytes for a total of 2229 Connection closed by 10.10.10.101
I have making sure that /etc/ssh/sshd_config allows key based authentication (PubkeyAuthentication yes). I have made sure the the permissions on /home (once mounted on the nodes) is correct. Users are properly authenticated. I have tried nfs mounting with and without "no_all_squash" restarting nfs, rpcidmap, rpcbind and nfslock.
I have had this working with CentOS5 installed on the nodes with a different master/head node. CentOS6 just seems to be giving me extra problems with this.
If I don't create the key, of course I'm prompted for a password.
My hosts.allow/deny are empty on both clients and server.
The root user is able to connect. Perceus handles the key generation for the root user since it is part of the virtual file system. I am guessing that something is wrong with the generation of my key but I can't figure out what the problem is.