switch_core_session.c:1861 Thread Failure!

FreeSWITCH Troubleshooting
1

It seems I met a interest issue raised by LUKE :slight_smile:

[CRIT] switch_core_session.c:1861 Thread Failure!

[CRIT] switch_core_session.c:1817 LUKE: I’m hit, but not bad.

[CRIT] switch_core_session.c:1818 LUKE’S VOICE: Artoo, see what you can do with it. Hang on back there…

Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,

Artoo reduces the max sessions to 4051 thus, saving the switch from certain doom.

I’m wondering how it could be?

In what stuation it will be trigger?

This issue could be reproduced on certain host(36core, CentOS7) during sipp test, whenever peak calls nearly reach 4000, it takes place.

On another(20core, CentOS7) host, it never appear, although there are totally using equal fs release and config.

runtime command echo is here

CentOS

cat /proc/cpuinfo |grep processor|wc -l

72

cat /proc/sys/kernel/threads-max

1029374

ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 514687

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 1048567

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 65535

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

2

Based on the code, it appears freeswitch is alerting that ut cannot create any new threads, and is reducing the max-sessions value to keep the system from crashing. It would appear that around 4000 calls is what the server is able to handle safely.

 1726 static void thread_launch_failure(void)
 1727 {
 1728         uint32_t sess_count;
 1729 
 1730         switch_mutex_lock(session_manager.mutex);
 1731 
 1732         sess_count = switch_core_session_count();
 1733 
 1734         if (sess_count > 110) {
 1735 
 1736                 switch_core_session_limit(sess_count - 10);
 1737                 switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_CRIT, "LUKE: I'm hit, but not bad.\n");
 1738                 switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_CRIT,
 1739                                                   "LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....\n"
 1740                                                   "Green laserfire moves past the beeping little robot as his head turns.  "
 1741                                                   "After a few beeps and a twist of his mechanical arm,\n"
 1742                                                   "Artoo reduces the max sessions to %d thus, saving the switch from certain doom.\n", sess_count - 10);
 1743 
 1744         }
 1745         
 1746         switch_mutex_unlock(session_manager.mutex);
 1747 }