Currently we are in middle of upgrade process from v4.5.7 to v6.7.1.
SonarQube is successfully upgraded to v5.6.7 (including DB upgrade) and currently in the process of updating to v6.7.1.
New SonarQube version started on an empty DB (bundled H2 DB) worked fine. After which all the plugins and their versions are validated and updated accordingly.
Now Sonar service launching is failing with error Background initialization failed. Stopping SonarQube java.lang.IllegalStateException: Unrecoverable indexation failures.
I removed data/es5 folder and reattempted to start sonar but with no luck. Any assistance to resolve the issue is greatly appreciated.
Web log:
INFO web[][o.s.p.ProcessEntryPoint] Starting web INFO web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read INFO web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] INFO web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin] INFO web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] INFO web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin] INFO web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9002] INFO web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.7.1.35068 / 426519346f51f7b980a76f9050f983110550509d INFO web[][o.sonar.db.Database] Create JDBC data source for jdbc:oracle:thin:@host:****:instance INFO web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /.../sonar/sonarqube-6.7.1 INFO web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Branch / 1.0.0.507 / e9d161eaa2993f79890dd5d08230f3f566e6be93 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Checkmarx / 8.41.0-20161229-0945 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Developer / 1.0.0.240 / 25c709aeb128caa14725b943422cca19def50493 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Flex / 2.3 / 79167f762219bea1fd181ff5ef6ca4b34abd1675 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub / 1.4.2.1027 / dcfb02a5fa40c219d527563f5b8c793966523ca7 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Governance / 2.0.2.3011 / 4756b2f9d98d28cf2762d91eba1b48839a1edbe1 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Groovy / 1.5 / dcbcd1eeb8f61c8351e5dd1e51e11e52d186df35 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin License / 3.3.0.1341 / 6b63240af64c92afe9421ece35c99c70f1b97991 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Pitest / 0.9 / 0c011514ce025e5524fdcaa0c917eacf9d40f312 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SVG Badges / 3.0.1 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarABAP / 3.5.0.1080 / c37baa20eb9d24a27961ffe25d95ae4b716a3193 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 6.5.0.3766 / 6011a1e9fb002edb044ae3092d92bcfb7d9d8a2d INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCFamily / 5.0.0.9359 / 6001ee3bce59e4bb9a17e5edbe40b4b99bc2901a INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCOBOL / 4.1.1.2663 / 846e542a08e9e472f0216da6c30e66a96bcbc821 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 3.2.0.5506 / 86da09565f891121df248835665f5d8ed98772b6 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 4.15.0.12310 / 572454b93016ec73a53fe0e07b2ffdc356d21ba9 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 2.11.0.2485 / 741861a29e5f9a26c6c99c06268facb6c4f4a882 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLI / 1.7.0.1117 / 2e07afcb9ecb6a2e96c980b48f7195b82ffdd014 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLSQL / 3.2.0.1753 / 455f471cfde991b2cd1a86a939ff410bf3006059 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.8.0.1496 / 3fe3bc4d0273a5721ea2fb368dc45b1bb82fede3 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: Git / 1.3.0.869 / 4da53e3f9e55f4f2e5796625cb0c5768ed152079 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: SVN / 1.6.0.860 / 2111fdbd1dddda4ad6d4ed6486fd0b18c1010d3b INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRPG / 2.2.0.1005 / 146ae32dade755799da4fa9655e865d8757fddfa INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarSwift / 3.1.0.2067 / 22db8593d10c835675860598372d41ba8d5f0b8e INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.1.0.1079 / 042c9e65239a47d92d305f9767f730b3cc1e5ed3 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTSQL / 1.1.0.2177 / e8811dfecb71d328c4deca89297bc100b270313c INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 4.2.0.248 / 75e2175c7464b814699abd6719b964c027717683 INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB6 / 2.3.0.992 / 8c9e2016cb2e325b2fdeba4d303181b33bbbda4b INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 1.4.3.1027 / 39588245cecf538bb27be4e496ff303b0143d20b INFO web[][o.s.s.p.ServerPluginRepository] Deploy plugin Web / 2.5.0.476 / 636872f5d37fa7a440fe07d08d504e1a881225e5 INFO web[][c.c.sonar.CxPlugin] Checkmarx plugin loaded INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@6cdba960 [pattern=UrlPattern{inclusions=[/api/system/migrate_db/*, ...], exclusions=[/api/properties*, ...]}] INFO web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port **** INFO web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy) INFO web[][c.q.p.s.b.f.PreferredFontProvider] SVGImageGenerator will be using font 'Verdana' in order to compute SVG badges width. INFO web[][c.q.p.s.b.w.SVGImageGenerator] SVGImageGenerator is now ready. INFO web[][c.q.p.s.b.w.g.QualityGateBadgeGenerator] QualityGateBadgeGenerator is now ready. INFO web[][c.q.p.s.b.w.m.MeasureBadgeGenerator] MeasureBadgeGenerator is now ready. INFO web[][c.q.p.s.b.w.c.CeActivityBadgeGenerator] CeActivityBadgeGenerator is now ready. WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/editions/show_license WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/views/definition INFO web[][o.s.s.s.LogServerId] Server ID: ********* WARN web[][o.s.s.u.DeprecatedViews] Widget 'Checkmarx' (checkmarx-project-widget) is ignored. See org.sonar.api.web.page.PageDefinition to define pages. WARN web[][o.s.s.u.DeprecatedViews] Page 'Checkmarx' (/cx_config/index) is ignored. See org.sonar.api.web.page.PageDefinition to define pages. INFO web[][org.sonar.INFO] Security realm: LDAP INFO web[][org.sonar.INFO] Security realm started INFO web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled. INFO web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.) INFO web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index INFO web[][o.s.s.s.RegisterPlugins] Register plugins INFO web[][o.s.s.s.RegisterMetrics] Register metrics INFO web[][o.s.s.r.RegisterRules] Register rules INFO web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles INFO web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile cpp/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile c/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile rpg/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way Recommended INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile pli/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile tsql/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile vb/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile plsql/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile cobol/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile web/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2 INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile abap/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile grvy/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile objc/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile swift/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way INFO web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way recommended INFO web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates INFO web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@1c655890 [pattern=UrlPattern{inclusions=[/api/measures/component/*, ...], exclusions=[/api/properties*, ...]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter@4f45e5bd [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter@29719313 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@694be25 [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@61014c56 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@44a4512f [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@67e2e76 [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}] INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@12d61b62 [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}] INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] ... INFO web[][o.s.s.es.BulkIndexer] 201463 requests processed (3357 items/sec) INFO web[][o.s.s.es.BulkIndexer] 357701 requests processed (2603 items/sec) INFO web[][o.s.s.es.BulkIndexer] 506390 requests processed (2478 items/sec) INFO web[][o.s.s.es.BulkIndexer] 636193 requests processed (2163 items/sec) INFO web[][o.s.s.es.BulkIndexer] 766613 requests processed (2173 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (2220 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec) INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] done | time=815372ms INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] ... INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] done | time=3122ms INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [issues/issue] ... INFO web[][o.s.s.es.BulkIndexer] 758897 requests processed (12648 items/sec) INFO web[][o.s.s.es.BulkIndexer] 1395232 requests processed (10605 items/sec) INFO web[][o.s.s.es.BulkIndexer] 1989212 requests processed (9899 items/sec) INFO web[][o.s.s.es.BulkIndexer] 2567812 requests processed (9643 items/sec) INFO web[][o.s.s.es.BulkIndexer] 3159399 requests processed (9859 items/sec) INFO web[][o.s.s.es.BulkIndexer] 3320485 requests processed (2684 items/sec) INFO web[][o.s.s.es.BulkIndexer] 3320485 requests processed (0 items/sec) ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube java.lang.IllegalStateException: Unrecoverable indexation failures at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39) at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:117) at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:247) at org.sonar.server.issue.index.IssueIndexer.indexOnStartup(IssueIndexer.java:95) at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55) at java.util.Optional.ifPresent(Optional.java:159) at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:84) at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45) at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:80) at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:196) at org.sonar.server.platform.Platform.access$400(Platform.java:46) at org.sonar.server.platform.Platform$1.lambda$doRun$1(Platform.java:121) at org.sonar.server.platform.Platform$AutoStarterRunnable.runIfNotAborted(Platform.java:371) at org.sonar.server.platform.Platform$1.doRun(Platform.java:121) at org.sonar.server.platform.Platform$AutoStarterRunnable.run(Platform.java:355) at java.lang.Thread.run(Thread.java:745) INFO web[][o.s.p.StopWatcher] Stopping process