Suddenly Poor Call Quality

FreeSWITCH Troubleshooting
1

Hello everyone,

Our setup is as follows we have 2 servers running freeswitch together with our own software to controll freeswitch via ESL. The servers get incomming calls via kamailio which are then evenly distributed among the 2 servers.

We noticed a problem with our voice-servers lately. For some reason the first server of our voice-servers started to produce many calls with bad quality up to around 20% of all the calls during peak hours. This started when our software segfaulted and we restarted it. If the amount of calls is not very high, calls with poor quality wont appear as much.

On the second server this problem also started to appear but with up to around 50% of the calls affected during peak hours. On this server our software did not have any segfaults. The problem started to appear after we updated our software. After the update we initially didnt restart freeswitch but after we restarted freeswitch the problem was fixed.

But we still dont know why this has happend. As you can see in the screenshots, during that time there were also many UDP buffer errors and IPv4 errors. Maybe this is related somehow. A large amount of the poor calls has a MOS of between 4 and 4.3.
The affected calls always had 5-20 missing UDP pakets.

The Hardware:
Dell PowerEdge R420
2x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz
32GB Ram
Debian GNU/Linux 11 (bullseye)
Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe

The First Server:

vs1-grafana
vs1-grafana1920×1080 223 KB

vs1-voipmon
vs1-voipmon1920×1080 179 KB

The Second Server:

vs2-grafana
vs2-grafana1920×1080 238 KB

vs2-voipmon
vs2-voipmon1920×1080 181 KB

2

Do you HEAR it or are you trusting the graphs to tell you its bad? And do you have a pcap or example?

/b

3

You can clearly hear the drop outs in the recording. The calls are often missing some UDP pakets like this:

wireshark
wireshark1897×903 82.3 KB

The questions is why this is happening so often?

4

This seems like packet loss, is this inbound to FreeSWITCH? Can you share a pcap?

/b

5

Is there a way to send you a pcap privatly? i cant post it here because of legal reasons.

6

brian@signalwire.com

/b

7

thanks, i have send you the pcap file

8

did you received the file?

9

I haven’t seen it in my inbox, seems like you had some packet loss.

/b

10

Can you guess why this happend? since after restarting freeswitch(not rebooting the server) it went away. So we think it might have something to do with freeswitch.

11

That sounds interesting, it could be some buffer issue. Please keep an eye on it.
/b

12

which buffers do you mean? so we can keep an eye on them

13

Unknown, if I knew it would be solved.

14

What information(core dump, etc) would be helpful to further investigate the problem, it it occures again?

15

Nothing is of help from a core file unless you have a crash.

/b