No Audio Between PBXs that are connected to BSS.

[JRayfield]

1. Brekeke Product Name and Version:
Brekeke SIP Server - 3.14.5.17/563.2

2. Java version:
11.0.15

3. OS type and the version:
Windows Server 2012

4. UA (phone), gateway or other hardware/software involved:
3CX

5. Your problem:
I'm using BSS to connect 3CX IP PBX servers to SIP trunks. So BSS is acting as a proxy server.

It works great (has been online for 11 years). One of the features that I really like and use sometimes is to route media (audio) through the BSS.

However, I've recently come across a problem with this.

Let's say I've got two 3CX IP PBX servers connected to BSS and both of them are configured in BSS to proxy/route the audio through BSS. This works fine, until a client on one 3CX server attempts to call a client on the other 3CX server. The call will go through, however there is no audio between the two 3CX servers.

I thought setting up hairpinning in the router to which BSS is connected might fix this, but it didn't help.

Does anyone have any ideas as to what is happening here and how to fix this so that clients on 3CX IP PBX servers can talk to clients on other 3CX IP PBX servers, where BSS is handling audio for both servers?

John Rayfield, Jr.

[Laurie]

Are these 3 SIP entities on the same local network?
Are they behind a NAT?
Can both 3CX servers send ping packet each other directly?

[JRayfield]

No, the 3 SIP 'entities' are not on the same network. Each has it's own public IP address.

I looked at some packet captures of SIP messages in and out of the BSS.

If I disable RTP ($rtp = false) in the Dial Plan for one of the servers for outbound calls, then in the INVITE for that outbound call, I can see that the IP address for the Media, is the SDP portion of the INVITE, is the public IP address of the 3CX server through which the outbound call is being made.

If I enable the RTP ($rtp = true) in the Dial Plan for that same outbound call route from that same 3CX server, then the IP address for the Media in the SDP portion of that INVITE, is the public IP address of the BSS.

So that makes sense. With $rtp = true, the audio is being routed through the BSS.

The problem is when a call is being made from one 3CX server to another 3CX server that are both on the same BSS and $rtp = true for both 3CX server 'connections'. This results in no audio between these 3CX servers.

I've tried enabling NAT loopback on the router on this BSS server network but that didn't help the situation.

There doesn't seem to be a way to 'connect' the audio between two PBX servers that are both connected to one BSS, where $rtp is set to true for inbound and outbound calls for both PBX server connections.

[Laurie]

Does the BSS have more than one public IP address?

If you capture SIP packets when you make a call from one 3CX to another 3CX though the SIP Server, which IP address does SDP point?
Is it the BSS's public IP address or caller 3CX's public IP address?

[JRayfield]

Just one public IP address.

If the $rtp = false, then the public IP address in the SDP is the public IP address of the 3CX server.

If the $rtp = true, then the public IP address in the SDP is the public IP address of the BSS.

[Laurie]

You can analyze RTP packets with Wireshark.
First, capture packets when you make a call from one 3CX to another 3CX, and say something even if there is no audio.
Second, at Wireshark, go to [Telephony]>[RTP]>[RTP Streams].

If there is no RTP stream, $rtp=true is not applied.
If there are RTP streams, check the destination IP address and port.
Also you can listen streams with the "Play streams" button.

[JRayfield]

Here's how I tested. I hope this makes sense, the way I'm explaining it.

3CX-A: $rtp = true on inbound, $rtp = true on outbound
3CX-B: $rtp = true on inbound, $rtp = false on outbound

A call from 3CX-B to 3CX-A works - audio is fine, both directions. Packet captures on the BSS machine show audio packets from 3CX-A to BSS to 3CX-B and from 3CX-B to BSS to 3CX-A.

3CX-A: $rtp = true on inbound, $rtp = true on outbound
3CX-B: $rtp = true on inbound, $rtp = true on outbound

A call from 3CX-B to 3CX-A does not work - no audio either direction. Packet captures show audio packets from 3CX-A to BSS and 3CX-B to BSS, but no audio packets from BSS to either 3CX-A or 3CX-B.

So, in these tests, the connection between 3CX-A and BSS is always set to $rtp = true, both inbound and outbound.

If the connection between 3CX-B and BSS is set to $rtp = false, then audio is good both directions.

If the connection between 3CX-B and BSS is set to $rtp = true, then there is no audio passed between the two 3CX servers through BSS.

This doesn't make sense to me. I'm beginning to wonder if this is a but in the BSS software.

[Laurie]

If possible can you paste DialPlan rules which caused the no-audio call?
They will be two rules. 3CX-B outbound and 3CX-A inbound.

[JRayfield]

The inbound rules also involve doing a 302 to route calls through a STIR/SHAKEN service. I also use an alias list, with DID numbers in it that are assigned to the 3CX servers. Also, if a person tries to dial out, using a outbound caller id that is not in the alias list, the call will be blocked in BSS. And rInstance is added on Inbound calls.

If you look at the 3CX-A Inbound, you'll see where the inbound $rtp is set, according to what is in the alias list.

In a nutshell, if $rtp = true for both the Outbound of one server connection and the Inbound of the other server, then no audio flows between the two servers through BSS. ONLY if one server connection is set to $rtp = false (and it doesn't matter which one - inbound of one or outbound of the other), will audio flow between the two servers through BSS.

I just realized that this is working according to the way a NAND gate in electronics works.

IN1 = 0
IN2 = 0
OUT =1

IN1 = 0
IN2 = 1
OUT = 1

IN1 = 1
IN2 = 0
OUT = 1

IN1 = 1
IN1 = 1
OUT = 0


3CX-B Outbound:

Matching-
$request = ^INVITE
$registered("username") = true
$addr = ^3cx-server-ip$
To = sip:(.+)@
From = sip:1?(.{10})@
$alias.lookup("1%2") = (.+)

Deploy-
$auth = false
$rtp = false
$b2bua = true
To = sip:%1@STIR/SHAKEN-Service-ip
$session = failover
&failover.redirection.header-copy-pattern = ^identity
&failover.timer.provisional = 120

3CX-A: Inbound

Matching-
$request = ^INVITE
$addr = upstream-service-provider-ip
To = sip:(.+)@
$alias.lookup("%1") = (.+)/(.+)
$regAddr("%2") = (.+)
$param($regdb.URIparam("%2"),"rinstance") = (.+)

Deploy-
&inbound.to = sip:%1@%4
$rtp = %3
$auth = false
$request = INVITE sip:%1@%4;rinstance=%5 SIP/2.0
$continue = true

Matching-
$request = ^INVITE
&inbound.to = (.+)
Identity = .+
To = sip:(.+)@

Deploy-
$b2bua = true
To = sip:%2@STIR/SHAKEN-Service-IP
$session = failover %1
&failover.timer.provisional = 120
&failover.redirection = false

[Laurie]

Is the upstream service provider involved with this no audio issue?
This is because you have "$addr=upstream-service-provider-ip" in the rule "3CX-A: Inbound".

For having a detailed log, set "SIP Session" in [Diagnostics]>[Debug Logs] page and reproduce the issue. Then find the sv.xxxx.log (eg: sv.20240320.log) from a downloaded ZIP file. Search the SIP session you made from the log.

[JRayfield]

I don't think the upstream service provider is involved in this issue, however, I'll take a look at the logs and see if I'm missing something there.

[Laurie]

You can find a session in the log with timestamp, From-URI and To-URI.
Check "rulename" to see it is expected rule names, and
"mode:RTPrelay" to see it is on or off.

[JRayfield]

I can tell when it's on or off because when it's on I don't have audio and when it's off, I do.

I enabled SIP Session in the Diagnostic Logs, but I don't see anything like "mode:RTPrelay".

[Laurie]

You need to reproduce the issue after "SIP Session" log category is enabled.

[JRayfield]

I did.

[JRayfield]

Here's one line from the session.240321.log file. Is this the correct file?

This call was made from 3CX-B to 3CX-A. $rtp was set to 'false' on 3CX-B for outbound calls and to 'true' for 3CX-A inbound calls. Audio worked.

No reference to RTPin this line.

11153, sip:4171234567@provider-ip, sip:14179993456@provider-ip, 4837, 1711040219312, 1711040220271, 1711040225108, Success, -1, 3CX-B_ip:5060, provider-ip, 2, RCI JR 863285 Out, 3CXPhoneSystem 18.0.9.20 (20), , Talking, 0,

[Laurie]

Please find sv.xxxx.log such as sv.240321.log not session.xxxx.log.
The session.xxx.log is kind of CDR and sv.xxxx.log is a detailed log which can include SIP packets.

[JRayfield]

I found the file, but it had quit logging this morning. The file size was too low.

I tried deleting today's file and running the test again, but a new log file isn't being created. It looks like I'll have to wait until tomorrow.

[Laurie]

It seems the log setting is not applied yet.
Did you push [Update] button after you enabled "SIP Session" log category in [Debug Logs] page?
Once you applied the log setting, make a test call from one 3CX to another 3CX to reproduce the issue.

[JRayfield]

Yes. I did all of that. It just didn't work.

Meanwhile, I got more packet captures and looked at the rtp streams. Very interesting.

As long as 3CX-B is set to $rtp=false for either inbound or outbound calls, and 3CX-A is set to $rtp=true for either inbound or outbound, then audio works fine, flowing through BSS.

As long as 3CX-A is set to $rtp= false for either inbound or outbound calls, and 3CX-A inbound or outbound is set to $rtp=true, then audio works fine, flowing through BSS.

So, audio flows through BSS as long as at least one of the 3CX-->BSS connections is set for $rtp=false.

If 3CX-B inbound and outbound are set to $rtp=true AND 3CX-A inbound and outbound are set to $rtp=true (all $rtp settings are "true", then no audio flows.

The packet captures show what's happening.

When it works, the audio streams are being routed to/from the public IP address of the 3CX servers, from/to the BSS.

So, it looks like this:

3CX-B --> BSS (local IP address) --> 3CX-A
and
3CX-A --> BSS (local IP address) --> 3CX-B

This is good and everything works.

However, if rtp is set to true on both inbound and outbound, for both 3CX server connections (all $rtp settings are "true", then the IP addresses are 'messed up':

3CX-B --> BSS (local IP address) --> Public IP address of BSS
3CX-A --> BSS (local IP address) --> Public IP address of BSS

So audio from 3CX-B is never routed to 3CX-A and audio from 3CX-A is never routed to 3CX-B. BSS is attempting to route audio from each 3CX server to the public IP address of the BSS, not the other 3CX server.

So, the first question is:

Why is the public IP address of the BSS being used when all $rtp settings are set to "true" (and only then)? If either 3CX server connection is set to $rtp=false, for inbound or outbound calls, then audio is routed properly from/to BSS to/from each 3CX server.

Second question:

How do I keep BSS from using the BSS public IP address when all $rtp settings are "true" and use the proper 3CX server public IP addresses? If I can determine that a call is between two 3CX servers on BSS, can I modify the SDP information to make sure that the proper IP addresses are used?

Third question:

To me, with my limited knowledge of how BSS works internally, it almost appears to be a bug, where the BSS public IP address is used, instead of the correct 3CX server public IP addresses. Or is there a setting (or settings) in BSS that might have a bearing on this.

[Laurie]

It seems there are configuration issues.
Does the BSS have a local (private) IP address?
Is the BSS located behind a NAT?

If you reproduce the issue, is it shown as a single SIP session in the [Active Sessions] page? or two sessions?

[JRayfield]

Yes, the BSS is behind NAT, so it has a private (LAN) IP.

In the BSS Configuration, I've entered the LAN IP of the BSS into Internal IP address pattern.

And I've entered the Public IP of the BSS into Interface address 1 and External IP address pattern.

For a call between the two 3CX servers, whether the calls have audio or not, there are two entries in the Active Sessions list - one for the outbound portion of the call and one for the inbound portion of the call.

Looking at more packet captures, I see:

If $rtp=false, then the IP address for audio routing to a 3CX server (in the SDP header of an INVITE) is the public IP address of that 3CX server.

If $rtp=true, then the IP address for audio routing to that 3CX server (in the SDP header) of an INVITE is the public IP address of the BSS.

The problem occurs, then, if the routes for audio to both 3CX servers are to the BSS server. When this occurs, the BSS isn't passing audio on to each 3CX server.

It's as if BSS can't route audio through it, if it's supposed to be routing between two servers that are both connected to BSS with $rtp=true.

It would appear that, maybe, the BSS isn't able to differentiate between audio coming from the two 3CX servers into it.

In the BSS Configuration, in RTP Configuration, Port Mapping is set to SDP. Should it be set to Source Port?

[Laurie]

> For a call between the two 3CX servers, whether the calls have audio or not, there are two entries in the Active Sessions list - one for the outbound portion of the call and one for the inbound portion of the call.

For 3CX to 3CX case, it should be a single SIP session to avoid an issue if a router doesn't handle hairpinning.
Try another router or tune DialPlan rules.

[JRayfield]

Ohhhh!

The routers that I use are MikroTik, which are VERY flexible in their programming - and somewhat complicated because of that.

I'll dig a bit more into the router and see what I might find that might be causing or might help this situation. I tried enabling Hairpinning, but I'm not sure that it was really working properly.

So, I have Dial Plan entry ahead of all other entries for Outbound calls from BSS. It looks like this:

Matching:
$request = ^INVITE
To = sip:.*(\d{11})@
$alias.lookup("%1") = (.+)

Deploy:
$continue = true

This checks all outbound calls from 3CX servers, through BSS, to see if the number that has been dialed is in an Alias List (and therefore is another 3CX server on this BSS).

This works very nicely. If I try to call 3CX-A from 3CX-B, the Session Log file in BSS shows that the Outbound call from 3CX-B triggered the above Dial Plan and the Dial Plan for outgoing calls from 3CX-B.

So, this let's me determine if a call is from one 3CX server connected to this BSS, to another 3CX server that is connected to this BSS.

What additional functions in the above Dial Plan can I add that will cause a call like this to 'loop back' through BSS, instead of trying to exist BSS and come back in? I'm quite 'stuck' at this point.

[JRayfield]

I got it fixed! I was able to make one change in my router/firewall that allows it to work now.

Your help, Laurie, was invaluable to me. I really appreciate it. I've also learned a lot more about how BSS works. I like that, too.

John