Sonar stops working every few days: jdbc connection error

I have a Sonar server which is used once a day from maven/jenkins and every few days, say every 4 or 5 days, it crashes and and shows the message "We're sorry, but something went wrong".

In the log, the error is always about a jdbc connection problem. I thought that it was a problem with the database but then if I restart the sonar server everything goes fine again.

So, it looks like a memory leak or something in the sonar server that makes it crash every few days until someone restarts it. Does that makes sense?. This is the configuration I have:

sonar.jdbc.username: xxxx
sonar.jdbc.password: xxxx
sonar.jdbc.url: jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
#----- Connection pool settings
sonar.jdbc.maxActive:                      20
sonar.jdbc.maxIdle:                        5
sonar.jdbc.minIdle:                        2
sonar.jdbc.maxWait:                        5000
sonar.jdbc.minEvictableIdleTimeMillis:     600000
sonar.jdbc.timeBetweenEvictionRunsMillis:  30000

sonar.updatecenter.activate=true

http.proxyHost=xxxx
http.proxyPort=3128

sonar.notifications.delay=60

That's it. And this is the error log:

INFO  o.s.s.p.ServerImpl  SonarQube Server / 3.7.3 / 
INFO  o.s.c.p.Database  Create JDBC datasource for jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true

ERROR o.s.c.p.Database  Can not connect to database.
Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').

org.apache.commons.dbcp.SQLNestedException: 
Cannot create PoolableConnectionFactory (Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
.
.
.
Caused by: java.net.ConnectException: Connection refused
.
.
.
INFO  jruby.rack  An exception happened during JRuby-Rack startup
no connection available
--- System
jruby 1.6.8 (ruby-1.8.7-p357) (2012-09-18 1772b40) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_43) [linux-amd64-java]
Time: Thu Jan 02 08:04:08 -0500 2014
Server: jetty/7.6.11.v20130520
jruby.home: file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home

--- Context Init Parameters:
jruby.compat.version = 1.8
jruby.max.runtimes = 1
jruby.min.runtimes = 1
jruby.rack.logging = slf4j
public.root = /
rails.env = production
--- Backtrace
ActiveRecord::ConnectionNotEstablished: no connection available
  set_native_database_types at arjdbc/jdbc/RubyJdbcConnection.java:517
                 initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection.rb:61
                 initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/adapter.rb:31
            jdbc_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection_methods.rb:6
                       send at org/jruby/RubyKernel.java:2109
             new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:223
    checkout_new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:245
                   checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:188
                       loop at org/jruby/RubyKernel.java:1439
                   checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:184
            mon_synchronize at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home/lib/ruby/1.8/monitor.rb:191
                   checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:183
                 connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:98
        retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:326
        retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:123
                 connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:115
                 initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:440
                         up at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:401
                    migrate at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:383
          upgrade_and_start at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:62
            automatic_setup at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:74
                     (root) at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:213
                       load at org/jruby/RubyKernel.java:1087
           load_environment at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:23
           load_environment at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65
                     (root) at <script>:1

--- RubyGems
Gem.dir: /opt/sonar/war/sonar-server/WEB-INF/gems
Gem.path:
/opt/sonar/war/sonar-server/WEB-INF/gems
Activated gems:
  rake-0.9.2.2
  activesupport-2.3.15
  activerecord-2.3.15
  rack-1.1.3
  actionpack-2.3.15
  actionmailer-2.3.15
  activeresource-2.3.15
  rails-2.3.15
  color-tools-1.3.0
  i18n-0.4.2
  json-jruby-1.2.3-universal-java-1.6
  activerecord-jdbc-adapter-1.1.3
  fastercsv-1.4.0
--- Bundler
undefined method `bundle_path' for Bundler:Module

--- JRuby-Rack Config
compat_version = RUBY1_8
default_logger = org.jruby.rack.logging.StandardOutLogger@4fbbe4e1
err = java.io.PrintStream@d2284af
filter_adds_html = true
filter_verifies_resource = false
ignore_environment = false
initial_memory_buffer_size =
initial_runtimes = 1
jms_connection_factory =
jms_jndi_properties =
logger = org.jruby.rack.logging.Slf4jLogger@566dc8f0
logger_class_name = slf4j
logger_name = jruby.rack
maximum_memory_buffer_size =
maximum_runtimes = 1
num_initializer_threads =
out = java.io.PrintStream@6aeeefcf
rackup =
rackup_path =
rewindable = true
runtime_arguments =
runtime_timeout_seconds =
serial_initialization = false
servlet_context = ServletContext@o.e.j.w.WebAppContext{/,file:/opt/sonar/war/sonar-server/},file:/opt/sonar/war/sonar-server

ERROR jruby.rack  unable to create shared application instance
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException: 
(ConnectionNotEstablished) no connection available
.
.
.
ERROR jruby.rack  Error: application initialization failed
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available

Any help will be appreciated :)

Answers


This sounds familiar to me :P Try asking your operators if they have some automated 'cleansing' operation that just kills periodically open database connections automatically in order to prevent leaked connections to the database.


It happened to me, in a Windows 2012 Server and SQL Server 2012 database in a different server. It seems that sonar service maintains a connection open, created at start up time, so any disconnection (networking, database restart, etc.) causes this unrecoverable connection problem. Restarting the Sonar (SonarQube) Windows service solved the problem to me. But, if this problem is frequent, as in your case, it would be a good idea to schedule a service restart task, or find out what is taking the connection down.


Need Your Help

In MVC/Razor, how do I get the values of multiple checkboxes and pass them all to the controller?

c# asp.net-mvc asp.net-mvc-4 razor checkbox

I have a view with a list of items from a model. I need to add a checkbox to each row, have the user select multiple check boxes, and pass some identifier of what row was selected to the controlle...