SonarQube 5.1 on Solaris doesn't show source files

Question

I've been trying to migrate from an outdated sonar installation to a newer SonarQube, version 5.1.2. The application is running on a Solaris platform and supported by an Oracle database.

Most everything works fine, except for the source code not being shown for any source file, in any context.

Any request to */api/sources/lines?uuid=...&from=1&to=1000 results in a 404, with a json object containing the following message:

"File '...' has no sources"

I've tried running the analysis from our Hudson server and from maven, with the same results. After having browsed the web for a few hours and found no question with the same symptoms, I'm turning to SO for help. In all the topics I've read, either the analysis doesn't run (e.g. Unable to run analysis on SonarQube 5.1) or the code is shown for some issues and not for others (e.g. Sonarqube Javascript Coverage doesn't show source files).

Out of sheer lassitude, I've tried installing the exact same version of SonarQube (5.1.2, same zip file as extracted on solaris), with the same configuration on my local machine (running Linux). The only differences are:

• the supporting OS (Solaris / Linux)
• the database (Oracle / H2)
• the JDK (although both are oracle's 1.8.0_45)

To my dismay, it works perfectly fine.

I've enabled DEBUG logs on the server and haven't been able to get any further details on how / why the elasticsearch index isn't finding the source for our projects.

Any information regarding how I can investigate further on this issue would be welcome.

Answer 1

Projects must be analyzed again when upgrading from (very) old versions such as 3.2. That should bring display of sources back.

Answer 2

I've been able to work around my issue following Simon's advice.

I've

1. restored the backup we had from the database before the upgrade from 3.2.1 to 5.1.2
2. performed an upgrade to the 4.5.6 LTS version
3. run a few analysis over several days on the 4.5.6 version, making sure everything worked as intended
4. created a backup of the database for the 4.5.6 version
5. performed the upgrade to version 5.1.2

During the migration to 5.1.2, the IndexSynchronizer ran into some trouble (due to elasticsearch apparently overloading the host's CPU):

 2015.11.03 13:21:23 INFO  web[o.s.s.s.IndexSynchronizer] Index source lines
 2015.11.03 13:22:10 INFO   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][766][38] duration [784ms], collections [1]/[1s], total [784ms]/[6.6s], memory [194mb]->[126.7mb]/[989.8mb], all_pools {[young] [58.6mb]->[2.1kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [126.8mb]->[118.2mb]/[682.6mb]}
 2015.11.03 13:22:23 INFO  web[o.s.s.es.BulkIndexer] 110525 requests processed (1842 items/sec)
 2015.11.03 13:22:30 WARN   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][784][51] duration [2.1s], collections [2]/[2.9s], total [2.1s]/[10.7s], memory [175.6mb]->[148.3mb]/[989.8mb], all_pools {[young] [42.2mb]->[37.3kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [124.9mb]->[139.7mb]/[682.6mb]}
 2015.11.03 13:23:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (269 items/sec)
 2015.11.03 13:24:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:25:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:26:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:27:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:28:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:29:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:30:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:31:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:32:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:33:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:34:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:35:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:36:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:37:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:38:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:39:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:40:07 INFO  web[o.e.client.transport] [sonar-1446552525578] failed to get node info for [#transport#-1][localhost][inet[/127.0.0.1:9001]], disconnecting...
 org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/127.0.0.1:9001]][cluster:monitor/nodes/info] request_id [3417] timed out after [5834ms]
    at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) ~[elasticsearch-1.4.4.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
 2015.11.03 13:40:07 WARN   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][788][54] duration [17.5m], collections [1]/[17.5m], total [17.5m]/[17.7m], memory [101mb]->[104.3mb]/[989.8mb], all_pools {[young] [7.8mb]->[380.8kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [84.7mb]->[95.5mb]/[682.6mb]}
 Wrapper Process has not received any CPU time for 978 seconds.  Extending timeouts.
 2015.11.03 13:40:07 ERROR web[o.s.s.ui.JRubyFacade] Fail to upgrade database
 java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Fail to execute ES search request '{"size":100,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"term":{"fileUuid":"3d674506-5a64-4b1e-a849-2c051e736e55","_cache":false}},{"range":{"line":{"from":250,"to":null,"include_lower":false,"include_upper":true},"_cache":false}}],"_cache":false}}}},"_source":false}' on indices '[sourcelines]' on types '[sourceline]'
    com.google.common.base.Throwables.propagate(Throwables.java:156)
    org.sonar.server.es.BaseIndexer.index(BaseIndexer.java:77)
    org.sonar.server.search.IndexSynchronizer.execute(IndexSynchronizer.java:90)

After that, it simply hung and wouldn't proceed with the upgrade nor show the index page (both / redirected to /maintenance and /setup showed an error message). After stopping and restarting sonar, it eventually finished migrating the source files index, and from there on it seems to work as expected.