KB 2285: ALL NTLM AUTHENTICATORS ARE BUSY
Symptoms in the case of busy authenticators are either untimely pop-ups requesting authentication or a sudden and complete shutdown of the squid service.
SYMPTOMS
Symptoms in the case of busy authenticators are either untimely pop-ups requesting authentication or a sudden and complete shutdown of the squid service.
Background
The number of Olfeo NTLM authenticators is set to 15 instances by default.
This number of instances has been defined as an average, corresponding to an architecture with fewer than 500 client workstations and reasonable web traffic.
This number of instances may be insufficient for an architecture with more than 500 workstations and/or a population generating a lot of web traffic (schools, universities, libraries, professions using web applications, etc.) and may therefore need to be increased.
TEST
Prerequisites
Install thesquid-clientutility with the command apt-get install squidclient:
TESTOLFEO:~# apt-get install squidclient Reading package lists... Done Building dependency tree... Done The following NEW packages will be installed: squidclient 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 102kB of archives. After unpacking 188kB of additional disk space will be used. WARNING: The following packages cannot be authenticated! squidclient Install these packages without verification [y/N]? y Get:1 http://debian.olfeo.com etch/main squidclient 3.1.5-2olfeo01 [102kB] Fetched 102kB in 0s (578kB/s) Selecting previously deselected package squidclient. (Reading database ... 16949 files and directories currently installed.) Unpacking squidclient (from .../squidclient_3.1.5-2olfeo01_i386.deb) ... Setting up squidclient (3.1.5-2olfeo01) ...
Step 1: Analyze the number of pending NTLM authenticators usingSquid Client.
The following command will allow us to determine whether the NTLM authenticators are busy.
Example where all NTLM authenticators are free (requests column):
TESTOLFEO:~# squidclient -h 127.0.0.1 -p 3129 mgr:ntlmauthenticator
HTTP/1.0 200 OK
Server: squid/3.1.5
Mime-Version: 1.0
Date: Tue, 06 Dec 2011 08:18:37 GMT
Content-Type: text/plain
Expires: Tue, 06 Dec 2011 08:18:37 GMT
Last-Modified: Tue, 06 Dec 2011 08:18:37 GMT
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3129
Via: 1.0 localhost (squid/3.1.5)
Proxy-Connection: close
NTLM Authenticator Statistics:
program: /usr/bin/ntlm_auth
number active: 15 of 15 (0 shutting down)
requests sent: 178
replies received: 178
queue length: 0
avg service time: 4 msec
# FD PID # Requests Flags Time Offset Request
1 254 4226 0 0.000 0 (none)
2 256 4227 0 0.000 0 (none)
3 258 4228 0 0.000 0 (none)
4 260 4229 0 0.000 0 (none)
5 262 4230 0 0.000 0 (none)
6 264 4231 0 0.000 0 (none)
7 266 4232 0 0.000 0 (none)
8 268 4233 0 0.000 0 (none)
9 270 4234 0 0.000 0 (none)
10 272 4235 0 0.000 0 (none)
11 274 4236 0 0.000 0 (none)
12 276 4237 0 0.000 0 (none)
13 278 4238 0 0.000 0 (none)
14 280 4239 0 0.000 0 (none)
15 282 4240 0 0.000 0 (none)
Flags key:
B = BUSY
C = CLOSING
R = RESERVED
S = SHUTDOWN PENDING
P = PLACEHOLDER
Step 2: Analyze the logs in /var/log/squid3/cache.log
WARNING lines can be filtered for easier reading.
After searching these lines, all NTLM authenticators are busy and 15 requests that could not be processed could have caused the squid service to shut down.
Command to type:
egrep -i '(busy|many|fatal)' /opt/olfeo5/data/proxy/log/cache.log
Result:
2011/10/04 07:30:21| WARNING: All ntlmauthenticator processes are busy. 2011/10/04 07:30:21| WARNING: 15 pending requests queued 2011/10/04 07:30:21| Consider increasing the number of ntlmauthenticator processes in your config file.
RESOLUTION
The number of requests for the last authenticator should not exceed 0.
Otherwise, it is recommended to increase the number of authenticators to 30 instances (for an architecture with more than 500 workstations).
In most cases, it is therefore advisable to increase from 15 instances to 30 instances to solve this problem.
If 30 is not a sufficient number, increase by increments of 10 or 20 until there are no problems.