SIP proxy slowly degraded towards finally not working

[ajlindy]

1. Brekeke Product Name and Version: Brekeke SIP Proxy 3.3.5.8/379-2

2. Java version: Java 1.7.0_67 32-bit

3. OS type and the version: Windows Server 2008
4. UA (phone), gateway or other hardware/software involved:

5. Your problem:

No Java updates happened, but after a reboot of the server when it came back up Brekeke would only handle a few SIP invites for our phones and then it eventually quit handling any calls.

Rebooting after that "appeared" to show Brekeke back up and running as a Service but we couldn't get to the webpage and we couldn't Stop the Service (it would time out). It would also not respond to any SIP invites.

What we eventually had to do was upgrade to Java 1.8.0_171 32-bit and point directly to the file path for that JVM to get it to resolve.

I have sv.logs for that day and it shows this:

svlistener: open session-log 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/log/2018/07/session.20180725.log'.
svlistener: open error-log 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/log/2018/07/error.20180725.log'.
svlistener: open dial-plan 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/etc/dialplan.tbl'.
svlistener: hostname=CH01-142510015 listen-port=5060
svlistener: IPv4: preferIPv6Addresses=false preferIPv4Stack=true
svlistener: interface={ REDACTED }
svlistener: classpath=C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\classes;C:\Program%20Files\Java\jre7\lib\tools.jar;C:\Program%20Files\Brekeke\sip\bin\bootstrap.jar;C:\Program%20Files\Brekeke\sip\bin\tomcat-juli.jar;C:\Program%20Files\Brekeke\sip\common\classes;C:\Program%20Files\Brekeke\sip\lib\annotations-api.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-ant.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-ha.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-tribes.jar;C:\Program%20Files\Brekeke\sip\lib\catalina.jar;C:\Program%20Files\Brekeke\sip\lib\ecj-4.2.2.jar;C:\Program%20Files\Brekeke\sip\lib\el-api.jar;C:\Program%20Files\Brekeke\sip\lib\jasper-el.jar;C:\Program%20Files\Brekeke\sip\lib\jasper.jar;C:\Program%20Files\Brekeke\sip\lib\jsp-api.jar;C:\Program%20Files\Brekeke\sip\lib\servlet-api.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-api.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-coyote.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-dbcp.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-es.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-fr.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-ja.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-jdbc.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-util.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\activation.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\hsqldb.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\jradiusclient.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\json_simple-1.1.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\log4j-core.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\log4j.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\mailapi.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondooss.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondosip.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondoutil.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\pop3.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\radiusplugin.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\smtp.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\TFplugin.jar;C:\Program%20Files\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext;
blockmgr: env : use default setting
Exception: 07/25/18 07:57:07.701
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0 (null:-2)
java.lang.Thread.start (null:-1)
com.brekeke.thread.ThreadCommon.start (null:-1)
com.brekeke.net.PacketEx.start (null:-1)
com.brekeke.net.sip.sv.Dispatcher.run (null:-1)


After the unsuccessful reboots to try and correct the problem, the logs don't look "wrong" to me, but maybe they are:

sv: open logging-file: 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/log/2018/07/sv.20180725.3.log'
sv: logging-plugin: com.brekeke.common.Logging
sv: 'your-sip-sv' at 'your-place' is starting...
sv: os=Windows Server 2008 (x86:6.0)
sv: java=1.7.0_67 (Java HotSpot(TM) Client VM, Oracle Corporation)
sv: java-home=C:\Program Files\Java\jre7
sv: total.mem=16252928 free.mem=14868688 cpu=2

svlistener: start at 07/25/18 08:45:05.614
tcp-listener: start
tcp-listener: listen-port=5060

svlistener: open session-log 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/log/2018/07/session.20180725.log'.
svlistener: open error-log 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/log/2018/07/error.20180725.log'.
svlistener: open dial-plan 'C:/Program Files/Brekeke/sip/webapps/sip/WEB-INF/work/sv/etc/dialplan.tbl'.
svlistener: hostname=CH01-142510015 listen-port=5060
svlistener: IPv4: preferIPv6Addresses=false preferIPv4Stack=true
svlistener: interface={ REDACTED }
svlistener: classpath=C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\classes;C:\Program%20Files\Java\jre7\lib\tools.jar;C:\Program%20Files\Brekeke\sip\bin\bootstrap.jar;C:\Program%20Files\Brekeke\sip\bin\tomcat-juli.jar;C:\Program%20Files\Brekeke\sip\common\classes;C:\Program%20Files\Brekeke\sip\lib\annotations-api.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-ant.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-ha.jar;C:\Program%20Files\Brekeke\sip\lib\catalina-tribes.jar;C:\Program%20Files\Brekeke\sip\lib\catalina.jar;C:\Program%20Files\Brekeke\sip\lib\ecj-4.2.2.jar;C:\Program%20Files\Brekeke\sip\lib\el-api.jar;C:\Program%20Files\Brekeke\sip\lib\jasper-el.jar;C:\Program%20Files\Brekeke\sip\lib\jasper.jar;C:\Program%20Files\Brekeke\sip\lib\jsp-api.jar;C:\Program%20Files\Brekeke\sip\lib\servlet-api.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-api.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-coyote.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-dbcp.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-es.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-fr.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-i18n-ja.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-jdbc.jar;C:\Program%20Files\Brekeke\sip\lib\tomcat-util.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\activation.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\hsqldb.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\jradiusclient.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\json_simple-1.1.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\log4j-core.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\log4j.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\mailapi.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondooss.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondosip.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\ondoutil.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\pop3.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\radiusplugin.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\smtp.jar;C:\Program%20Files\Brekeke\sip\webapps\sip\WEB-INF\lib\TFplugin.jar;C:\Program%20Files\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext;

[Tata]

The log indicates that you got OutOfMemoryError.

Are there any activities at the Session Log or/and Error log around at that time?
You can access these logs from [Logs]->[Session Logs] page and [Error Logs] page.

[jdkaplan]

I am a colleague of ajlindy, and working on this issue for our client with him.

In the Session and Error logs there is nothing at the time of the Out of Memory error. There is a gap of time during the period where Brekeke was not responding, but no errors were printed out into those logs.

[Tata]

Did you change any settings just before the issue started happening?
How often does the issue happen?
How many concurrent calls are they when the issue happens?

[jdkaplan]

There were no settings changes before this issue occurred, and it effected all calls in the system until we were able to update the java version.

Just before this happened, the server it was on was rebooted. Prior to the reboot there were reports of issues, but we were not notified until after the reboot and subsequent failure of the Brekeke SIP server to respond.

The issue has not repeated itself since.

[Tata]

Are you still using the version 3.3.5.8?
I recommend you to use the latest version. It is 3.8.5.2.

http://www.brekeke.com/wiki/BSS_Minor_Update
http://wiki.brekeke.com/wiki/BSS_History