Fixed deadlock between AsyncSshHandlerReader and AsyncSshHandler 24/81924/1
authorJaroslav Tóth <jtoth@frinx.io>
Sat, 30 Mar 2019 18:12:03 +0000 (19:12 +0100)
committerRobert Varga <nite@hq.sk>
Tue, 7 May 2019 05:36:54 +0000 (05:36 +0000)
commit0a078764e40c24d565098ddff5ec8be01b5e79f6
tree74b12be7e9f6e5da0a2426a84e90409392754d04
parenta8b245f1909fa175ab65f8d9b24d56ba2810be08
Fixed deadlock between AsyncSshHandlerReader and AsyncSshHandler

First was fix in AsyncSshHandlerReader but there was another
deadlock after that. Therefore a change in AsyncSshHandler is also
needed. Both ABBA deadlockw occur when AsyncSshHandler.disconnect
(..) and AsyncSshHandlerReader.invokeDisconnect(..) methods are
executed concurrently within the same SSH session that is going
to be closed.

Note: Snippets from stack-traces are from Oxygen distribution, so
the line numbers don't have to match exactly.

Deadlock #1:
"nioEventLoopGroupCloseable-3-20":
at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandlerReader.close(AsyncSshHandlerReader.java:96)
- waiting to lock <0x000000072ecffff8> (a org.opendaylight
.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerReader)
at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandler.disconnect(AsyncSshHandler.java:235)
- locked <0x000000072ed00020> (a org.opendaylight.netconf
.nettyutil.handler.ssh.client.AsyncSshHandler)
at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandler.close(AsyncSshHandler.java:215)
...

"sshd-SshClient[57f78db1]-nio2-thread-1":
at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandler.disconnect(AsyncSshHandler.java:221)
- waiting to lock <0x000000072ed00020> (a org.opendaylight
.netconf.nettyutil.handler.ssh.client.AsyncSshHandler)
...
org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandlerReader.operationComplete(AsyncSshHandlerReader
.java:64)
- locked <0x000000072ecffff8> (a org.opendaylight.netconf
.nettyutil.handler.ssh.client.AsyncSshHandlerReader)
...

Deadlock #2:
"nioEventLoopGroupCloseable-3-19":
at org.apache.sshd.client.session.ClientSessionImpl
.signalAuthFailure(ClientSessionImpl.java:138)
- waiting to lock <0x000000072c800218> (a java.lang.Object)
at org.apache.sshd.client.session.ClientSessionImpl
.preClose(ClientSessionImpl.java:126)
at org.apache.sshd.common.util.closeable.AbstractCloseable
.close(AbstractCloseable.java:95)
        at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandler.disconnect(AsyncSshHandler.java:251)
- locked <0x000000072c800480> (a org.opendaylight.netconf
.nettyutil.handler.ssh.client.AsyncSshHandler)
...

"sshd-SshClient[411a406d]-nio2-thread-8":
at org.opendaylight.netconf.nettyutil.handler.ssh.client
.AsyncSshHandler.disconnect(AsyncSshHandler.java:221)
- waiting to lock <0x000000072c800480> (a org.opendaylight
.netconf.nettyutil.handler.ssh.client.AsyncSshHandler)
...
org.apache.sshd.common.session.helpers.AbstractSession
.handleMessage(AbstractSession.java:499)
- locked <0x000000072c800218> (a java.lang.Object)
...

Change-Id: I42260213cad82250832451539d6ce11035ba37e2
Signed-off-by: Martin Sunal <msunal@frinx.io>
Signed-off-by: Jaroslav Tóth <jtoth@frinx.io>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
netconf/netconf-netty-util/src/main/java/org/opendaylight/netconf/nettyutil/handler/ssh/client/AsyncSshHandler.java
netconf/netconf-netty-util/src/main/java/org/opendaylight/netconf/nettyutil/handler/ssh/client/AsyncSshHandlerReader.java