KB 2285: ALL NTLM AUTHENTICATORS ARE BUSY
The symptoms of busy authenticators are either unwanted authentication request pop-ups, or a sudden and complete shutdown of the squid service.
SYMPTOMS
The symptoms of busy authenticators are either unwanted authentication request pop-ups, or a sudden and complete shutdown of the squid service.
Context
The default number of Olfeo NTLM authenticators is 15.
This number of instances has been defined as an average, corresponding to an architecture of less than 500 client workstations with reasonable web traffic.
This number of instances may be insufficient for an architecture with more than 500 workstations and/or a population generating a lot of web traffic (schools, universities, libraries, professions using web applications, etc.), and may therefore need to be increased.
TEST
Prerequisites
Install the squid-client utility with apt-get install squidclient :
TESTOLFEO:~# apt-get install squidclient Reading package lists... Done Building dependency tree... Done The following NEW packages will be installed: squidclient 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 102kB of archives. After unpacking 188kB of additional disk space will be used. WARNING: The following packages cannot be authenticated! squidclient Install these packages without verification [y/N]? y Get:1 http://debian.olfeo.com etch/main squidclient 3.1.5-2olfeo01 [102kB] Fetched 102kB in 0s (578kB/s) Selecting previously deselected package squidclient. (Reading database ... 16949 files and directories currently installed.) Unpacking squidclient (from .../squidclient_3.1.5-2olfeo01_i386.deb) ... Setting up squidclient (3.1.5-2olfeo01) ...
Step 1: Squid client analysis of the number of NTLM authenticators pending.
The following command will tell us whether NTLM authenticators are busy.
Example where all NTLM authenticators are free (requests column) :
TESTOLFEO:~# squidclient -h 127.0.0.1 -p 3129 mgr:ntlmauthenticator
HTTP/1.0 200 OK
Server: squid/3.1.5
Mime-Version: 1.0
Date: Tue, 06 Dec 2011 08:18:37 GMT
Content-Type: text/plain
Expires: Tue, 06 Dec 2011 08:18:37 GMT
Last-Modified: Tue, 06 Dec 2011 08:18:37 GMT
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3129
Via: 1.0 localhost (squid/3.1.5)
Proxy-Connection: close
NTLM Authenticator Statistics:
program: /usr/bin/ntlm_auth
number active: 15 of 15 (0 shutting down)
requests sent: 178
replies received: 178
queue length: 0
avg service time: 4 msec
# FD PID # Requests Flags Time Offset Request
1 254 4226 0 0.000 0 (none)
2 256 4227 0 0.000 0 (none)
3 258 4228 0 0.000 0 (none)
4 260 4229 0 0.000 0 (none)
5 262 4230 0 0.000 0 (none)
6 264 4231 0 0.000 0 (none)
7 266 4232 0 0.000 0 (none)
8 268 4233 0 0.000 0 (none)
9 270 4234 0 0.000 0 (none)
10 272 4235 0 0.000 0 (none)
11 274 4236 0 0.000 0 (none)
12 276 4237 0 0.000 0 (none)
13 278 4238 0 0.000 0 (none)
14 280 4239 0 0.000 0 (none)
15 282 4240 0 0.000 0 (none)
Flags key:
B = BUSY
C = CLOSING
R = RESERVED
S = SHUTDOWN PENDING
P = PLACEHOLDER
Step 2: Log analysis in /var/log/squid3/cache.log
WARNING lines can be filtered for easier reading.
After searching these lines, all NTLM authenticators are busy and 15 requests that could not be processed could have caused the squid service to stop.
Command to type :
egrep -i '(busy|many|fatal)' /opt/olfeo5/data/proxy/log/cache.log
Result:
2011/10/04 07:30:21| WARNING: All ntlmauthenticator processes are busy. 2011/10/04 07:30:21| WARNING: 15 pending requests queued 2011/10/04 07:30:21| Consider increasing the number of ntlmauthenticator processes in your config file.
RESOLUTION
The number of requests for the last authenticator should not exceed 0.
Otherwise, we recommend increasing the number of authenticators to 30 instances (for an architecture with more than 500 workstations).
In most cases, it is therefore advisable to increase the number of instances from 15 to 30 to solve this problem.
If 30 isn't enough, increase in steps of 10 or 20 until there's no problem.