bitbucket ssh clone on aix 7.1 fails

2
votes

I'm facing issue while trying to clone a test repo(public) from bitbucket as shown below. Strangely the ssh auth is not going through for bitbucket and works fine for github when same public key is added to both accounts. I'm assuming that there is some sort of ssh negotiation variation between github and bitbucket which doesn't work with version of opnessh on my aix 7.1 server. Any clues what can be done here to find the root cause? Please note that the https clone on bitbucket works just fine.

ssh and git version

max-08$ ssh -V
OpenSSH_7.5p1, OpenSSL 1.0.2p  14 Aug 2018

max-08$ git --version
git version 2.18.0

permissions on .ssh directory

max-08$ ls -lrt .ssh
total 40
-rw-r--r--    1 max staff            57 Jan 14 16:36 config
-rwx------    1 max staff          1989 Jan 14 16:38 authorized_keys
-rw-------    1 max staff          3243 Jan 14 16:40 id_rsa
-rw-r--r--    1 max staff           755 Jan 14 16:40 id_rsa.pub
-rwx------    1 max staff          2040 Jan 14 17:00 known_hosts

ssh/git clone trace for bitbucket

max-08$ GIT_SSH_COMMAND="ssh -vvv -i /home/max/.ssh/id_rsa" git clone [email protected]:max/testing.git
Cloning into 'testing'...
OpenSSH_7.5p1, OpenSSL 1.0.2p  14 Aug 2018
debug1: Reading configuration data /home/max/.ssh/config
debug1: /home/max/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Failed dlopen: /usr/krb5/lib/libkrb5.a(libkrb5.a.so): Could not load module /usr/krb5/lib/libkrb5.a(libkrb5.a.so).
System error: No such file or directory

debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: resolving "bitbucket.org" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to bitbucket.org [18.205.93.2] port 22.
debug1: Connection established.
debug1: identity file /home/max/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/max/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version conker_1.1.17-9953920 app-133
debug1: no match: conker_1.1.17-9953920 app-133
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/max/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/max/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-dss,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128
debug2: MACs ctos: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug3: hostkeys_foreach: reading file "/home/max/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/max/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/home/max/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/max/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 18.205.93.2
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/max/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/max/.ssh/id_rsa (2008c568), explicit
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/max/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug2: input_userauth_pk_ok: fp SHA256:QHbX3ieex06ZuF8XtkQcBFoCfpCooxh2noxyGdTE1uI
debug3: sign_and_send_pubkey: RSA SHA256:QHbX3ieex06ZuF8XtkQcBFoCfpCooxh2noxyGdTE1uI
debug3: send packet: type 106
debug1: Sent ALLOW_PKCS12_KEYSTORE_CLIENT_FLAG packet
debug3: send packet: type 50
Authentication failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists
2
gitbitbucketaixopenssh
Just covering all the bases: have you associated that public key with your account on Bitbucket? Does that account have read access to that repository? – Jim Redmond
debug1: key_load_public: No such file or directory kind of looks like a "using errno without clearing it and without checking that the call failed" bug ... what happens if you touch(1) the file it tries to dlopen? Do you have a strace(1)-like tool on AIX? – o11c
@JimRedmond yes it's a public repo so I don't really need to specify any permissions. Moreover https clone works. – Max
@o11c touch works just fine. I'll try to get you some trace. – Max
You're right that public repos don't need permissions, but have you associated that public key with your account? SSH traffic still requires that, even for public repos. – Jim Redmond

2 Answers

2
votes

Here is what worked on AIX 7.1 Add following config to .ssh/config

AllowPKCS12keystoreAutoOpen=no

Why this is enabled by default on AIX 7.1 with SSH OpenSSH_7.5p1 is something I'll pursue further and will start a thread with AIX and Atlassian guys to see if they need to handle some scenarios here since this works perfectly fine for github.

1
votes

It looks like it is an issue with Apache Mina:

https://issues.apache.org/jira/browse/SSHD-886