Is it possible to configure SolrCloud with SSL and Basic Authentication?
I have configured 3 nodes of Solr in SolrCloud with SSL using this: https://cwiki.apache.org/confluence/display/solr/Enabling+SSL
and I have added authentication and authorization following this: https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin, https://cwiki.apache.org/confluence/display/solr/Rule-Based+Authorization+Plugin
when only SSL is enabled it works.
when only authentication + authorization is enabled it works
when both are enabled I get following stacktrace during startup:
2016-06-01 17:19:41.933 INFO (OverseerStateUpdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testowa/state.json version: 1350 2016-06-01 17:19:41.935 INFO (zkCallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testowa/state.json] for collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-01 17:19:41.937 INFO (zkCallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [testowa] from [1350] to [1351] 2016-06-01 17:19:43.557 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2016-06-01 17:19:43.557 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2016-06-01 17:19:43.557 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.SyncStrategy Sync replicas to https://172.30.92.66:8983/solr/testowa_shard1_replica3/ 2016-06-01 17:19:43.561 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr START replicas=[https://172.30.182.43:8983/solr/testowa_shard1_replica1/, https://172.30.182.44:8983/solr/testowa_shard1_replica2/] nUpdates=100 2016-06-01 17:19:44.580 WARN (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr exception talking to https://172.30.182.44:8983/solr/testowa_shard1_replica2/, failed org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://172.30.182.44:8983/solr/testowa_shard1_replica2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized request, Response code: 401</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /solr/testowa_shard1_replica2/get. Reason: <pre> Unauthorized request, Response code: 401</pre></p> </body> </html> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:198) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3.000000003C022970.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785) 2016-06-01 17:19:44.582 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr DONE. sync failed 2016-06-01 17:19:44.583 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2016-06-01 17:19:44.585 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery 2016-06-01 17:19:44.585 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ElectionContext Canceling election /collections/testowa/leader_elect/shard1/election/168013962670440512-core_node1-n_0000000882 2016-06-01 17:19:44.588 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2016-06-01 17:19:44.590 INFO (updateExecutor-2-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.DefaultSolrCoreState Running recovery 2016-06-01 17:19:44.592 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.LeaderElector Joined leadership election with path: /collections/testowa/leader_elect/shard1/election/168013962670440512-core_node1-n_0000000885 2016-06-01 17:19:44.594 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2016-06-01 17:19:44.597 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy ###### startupVersions=[[1535485004938739712, 1535485004934545409, 1535485004934545408, 1535485004930351104, 1535485004926156801, 1535485004926156800, 1535485004919865346, 1535485004919865345, 1535485004919865344, 1535485004914622464, 1535485004908331010, 1535485004908331009, 1535485004908331008, 1535485004902039552, 1535485004898893824, 1535485004894699521, 1535485004894699520, 1535485004891553792, 1535485004887359488, 1535485004883165185, 1535485004883165184, 1535485004878970880, 1535485004875825152, 1535485004871630849, 1535485004871630848, 1535485004867436544, 1535485004864290816, 1535485004860096513, 1535485004860096512, 1535485004855902208, 1535485004851707905, 1535485004851707904, 1535485004847513600, 1535485004843319297, 1535485004843319296, 1535485004837027841, 1535485004837027840, 1535485004832833538, 1535485004832833537, 1535485004832833536, 1535485004823396353, 1535485004823396352, 1535485004819202048, 1535485004816056321, 1535485004816056320, 1535485004811862016, 1535485004807667712, 1535485004803473409, 1535485004803473408, 1535485004799279104, 1535485004795084801, 1535485004795084800, 1535485004790890496, 1535485004787744768, 1535485004786696192, 1535485004783550464, 1535485004778307585, 1535485004778307584, 1535485004775161856, 1535485004770967552, 1535485004767821824, 1535485004766773248, 1535485004763627520, 1535485004759433217, 1535485004759433216, 1535485004754190337, 1535485004754190336, 1535485004748947456, 1535485004744753153, 1535485004744753152, 1535485004740558849, 1535485004740558848, 1535485004735315968, 1535485004731121664, 1535485004727975936, 1535485004726927360, 1535485004723781633, 1535485004723781632, 1535485004722733056, 1535485004714344448, 1535485004710150145, 1535485004710150144, 1535485004703858689, 1535485004703858688, 1535485004699664384, 1535485004695470080, 1535485004692324353, 1535485004692324352, 1535485004688130048, 1535485004684984320, 1535485004680790017, 1535485004680790016, 1535485004677644288, 1535485004673449985, 1535485004673449984, 1535485004668207105, 1535485004668207104, 1535485004664012800, 1535485004660867072]] 2016-06-01 17:19:44.599 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.LeaderElector Watching path /collections/testowa/leader_elect/shard1/election/240110433826439197-core_node3-n_0000000884 to know if I could be the leader 2016-06-01 17:19:44.603 INFO (OverseerStateUpdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { "operation":"leader", "shard":"shard1", "collection":"testowa"} current state version: 38 2016-06-01 17:19:44.607 INFO (OverseerStateUpdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testowa/state.json version: 1351 2016-06-01 17:19:44.611 INFO (zkCallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testowa/state.json] for collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-01 17:19:44.613 INFO (zkCallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [testowa] from [1351] to [1352] 2016-06-01 17:19:47.272 ERROR (qtp1185255965-22) [ ] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://172.30.182.43:8983/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:107) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:202) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:155) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-01 17:19:47.281 ERROR (qtp1185255965-22) [ ] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.a(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:172) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:159) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-01 17:19:47.288 WARN (qtp1185255965-22) [ ] o.a.s.s.PKIAuthenticationPlugin Failed to decrypt header, trying after refreshing the key Looks like all security plugins are working ok, but when they are all enabled Basic authentication is not using superuser and nodes between themselves cannot communicate. Any idea what can be wrong?