One of my company’s servers is hosted with Liquid Web. Yesterday one of my co-workers tries to log into WHM and sees the following message:
Brute Force Protection
This account is currently locked out because a brute force attempt was detected. Please wait 10 minutes and try again. Attempting to login again will only increase this delay. If you frequently experience this problem, we recommend having your username changed to something less generic.
He tries to log in a bit later and receives the message again. He contacts Liquid Web’s Heroic Support, and the support person “helpfully” recommend a server reboot to fix the problem. My co-worker asked for another solution, and support said that it was the only way.
My co-worker tells me about the situation, and I tell him that the support guy is an idiot. It takes all of about ten seconds to quickly find a solution to the problem on google. I then tell my co-worker that I’ll fix the problem, so he “thanks” the support guy and closes the chat.
I logged into the box via SSH and had the problem fixed in a couple of minutes.
Since access to the box wasn’t urgent, I did what I recommend that everyone does in situations like this: gather information first. I wanted to know what caused the lock out first. In other words, who or what was brute forcing the box and caused this issue.
It was quite possible that my box was still under attack. If I simply turned off brute force protection to bypass the block, I could have opened up my box to being compromised.
cPHulk stores all of its information in a database called
cphulkd. There are two tables of interest:
brutes. The logins table stores login authentication failures. The brutes table stores excessive authentication failures indicative of a brute force attack.
Here is what I saw on the server:
[chris@office ~]$ ssh server Last login: Wed Oct 14 11:02:14 2009 from host [user@server ~]$ mysql -u user -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is id Server version: version Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> connect cphulkd Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Connection id: id Current database: cphulkd mysql> select IP, BRUTETIME from brutes order by BRUTETIME; Empty set (0.00 sec) mysql> select IP, LOGINTIME FROM logins order by LOGINTIME; +---------------------------------+---------------------+ | IP | LOGINTIME | +---------------------------------+---------------------+ | 188.8.131.52 | 2009-10-14 11:23:10 | | 184.108.40.206 | 2009-10-14 11:23:10 | | 220.127.116.11 | 2009-10-14 11:23:10 | | 18.104.22.168 | 2009-10-14 11:23:40 | | 22.214.171.124 | 2009-10-14 11:23:40 | | 126.96.36.199 | 2009-10-14 11:23:40 | | djdeatheater.liquidweb.com | 2009-10-14 11:24:03 | | 188.8.131.52 | 2009-10-14 11:24:07 | | 184.108.40.206 | 2009-10-14 11:24:07 | | 220.127.116.11 | 2009-10-14 11:24:07 | | djdeatheater.liquidweb.com | 2009-10-14 11:24:09 | | djdeatheater.liquidweb.com | 2009-10-14 11:24:15 | | mail.ingener.com | 2009-10-14 11:24:53 | | mail.ingener.com | 2009-10-14 11:24:57 | | 18.104.22.168 | 2009-10-14 11:25:16 | | 22.214.171.124 | 2009-10-14 11:25:16 | | 126.96.36.199 | 2009-10-14 11:25:16 | | 188.8.131.52 | 2009-10-14 11:25:41 | | 184.108.40.206 | 2009-10-14 11:25:41 | | 220.127.116.11 | 2009-10-14 11:25:41 | | pomme.sai.msu.ru | 2009-10-14 11:26:13 | | pomme.sai.msu.ru | 2009-10-14 11:26:13 | | pomme.sai.msu.ru | 2009-10-14 11:26:13 | | 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 | | 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 | | 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 | | 18.104.22.168 | 2009-10-14 11:27:23 | | 22.214.171.124 | 2009-10-14 11:27:23 | | 126.96.36.199 | 2009-10-14 11:27:23 | | 188.8.131.52 | 2009-10-14 11:27:47 | | 184.108.40.206 | 2009-10-14 11:27:47 | | 220.127.116.11 | 2009-10-14 11:27:47 | +---------------------------------+---------------------+ 32 rows in set (0.00 sec)
As you can see, a distributed brute force login attempt was launched starting at 11:23am. Fortunately, cPHulk quickly recognized the attack and had completely shut down login access for the user account.
Thus, when we tried to log in, it of course didn’t work.
Since I now knew the problem, and now felt good about how cPHulk protected our box, I decided to open up access again so we could go about our work. There are two different ways that I could have done this.
The first way would be to disable cPHulk to regain access, log into WHM, clear out the the block by using the “Flush DB” option in the cPHulk settings page, and then re-enable cPHulk. A number of people recommended this method, but I didn’t like it. I certainly don’t want to disable a security measure that successfully protected the box just to be able to regain access. What would happen if a huge wave of brute force authentication attempts hit the box in the time between disabling and re-enabling cPHulk? The answer is that the box wouldn’t protest and would tell the attacking program whether each attempt was successful or not.
If you need to use this method, the two commands you will want to use are:
/usr/local/cpanel/bin/cphulk_pam_ctl --disable and
/usr/local/cpanel/bin/cphulk_pam_ctl --enable. These two commands will disable and enable cPHulk, respectively.
I decided to use another method. This method didn’t require disabling cPHulk, and thus, didn’t require reducing protection to regain access. Essentially, I cleared the tables manually, so that I could log in once again. Since this opens up the login process again, a brute force could still proceed but would be quickly shut down again.
While still connected to the database through the MySQL monitor, I ran a couple more queries.
Note: Before I ran these queries. I stored the details of these attacks in a file on the server in case I needed to refer to the information later. Never delete attack data without keeping a record of it somewhere. You never know when you may need it.
mysql> delete from brutes; Query OK, 0 rows affected (0.00 sec) mysql> delete from logins; Query OK, 32 rows affected (0.00 sec)
Now, we can log back into the box.
I’m sure that many people encountering this situation would just pat themselves on the back and go about their day, but I wasn’t satisfied.
I wanted to ensure that the office wouldn’t be blocked in the event that another brute force attack happens, so I added our office to the whitelist. The whitelist supports both IP numbers and hostnames. So, if you have a dynamic IP, you can use a DDNS (Dynamic DNS) service to get your own hostname to use. Both DynDNS and No-IP offer reliable free solutions that provide you with a hostname to use. In addition, EveryDNS is a free DNS service provider that also offers the ability to create subdomains of your personal domain that can link to dynamic IPs. Many current routers support either DynDNS, No-IP, or both so that the router can handle updating the hostname automatically. Some router firmwares, such as Tomato support a much larger variety of DDNS services.
Another nice feature about using the DDNS services and whitelisting a hostname is that you can update that DNS pointer remotely. This means that if you get locked out of WHM when you are away from the location that is whitelisted, you can update the IP on the DDNS system, wait a few minutes for the IP to update, and then log into WHM via the whitelist.
One final thing I did was to change the account’s password to an even more complicated and much longer one. Good luck cracking that in 32 tries.
I have to say that I was very pleased with the performance of cPHulk, a piece of software that I didn’t configure and which did its job admirably with no intervention by us. I wasn’t as pleased with the performance of the Liquid Web support person that “helped” us. Maybe I’ve just been pampered with the amazing support quality that I’ve experience at Host Gator, but I don’t think that “reboot the server” is an appropriate response to “the security software locked us out”. In fact, I’m not even sure if a reboot would have done anything other than just piss us off.
I sent a tweet about this yesterday and was pleased to get a response from Nick Campbell, a manager at Liquid Web, who wanted to know who recommended the reboot solution so that he could “thwack them if needed”. Thanks Nick.
The terminal output styling is new. I worked on it last night, and I’m quite pleased with the results. I think it makes the output much easier to scan and understand. I’d love to know what you think about it.
Did I help you?
The help desk was obviously wrong. Here is another link that is close to how you did it. http://www.mydigitallife.info/2009/07/01/account-locks-out-due-to-brute-force-protection-in-cpanel-webhost-manager-whm/
Good to see you at gaarai.com Danny. 🙂
Thanks for sharing the link.
[…] Limpiar la base de datos del Brute Force de Cpanel cuando te ha bloqueado a ti mismo. […]
Great Article… it helps me a million since there had been so many attempts it was set to a lockout of 2 weeks :S
I’ve since whitelisted my IP will that then enable me access to WHM ?
I didn’t try out whitelisting, so I can’t comment on that. I’m satisfied with just being able to clear out blacklists.
Just wanted say thanks for the great post, saved me a tonne of time and work just now with the same problem.
You’re my hero, I don’t have to contact my Hosting provider to fix the stupid Brute Force Protection…Just followed your instructions, walah it’s gone and now I’m connecting!
You’re awesome…thanks again!
great write up. i locked myself out on a new server, dont know how, but could still get into ssh, but not whm. i just used the easy way out, and disbled cphulk long enough for me to get in whm and flush the database. i then promptly added my IP to the whitelist, which is typical behavior, i guess i got wrapped up in the move and overlooked adding my IP.
i wanted to drop a thank you to you here, and let you know it was a great write up.
thanks, and good job.
Since you asked, the terminal output you’ve shown here looks great. I actually took it to be a screen-dump at first glance, which perhaps suggests that it’s very clear what it’s supposed to show. Well done.
Thanks. Great help!
We had this happen on a remote dns-only node… tried stopping cphulkd (as described). Did not allow us to gain access via web. Zeroed the logins, and brutes tables. Did not allow access. Finally added our IP address into the whitelist table directly and that worked:
insert into whitelist values (“999.999.999.999”,0);
(put your IP in place of the 999’s)
Then worked immediately.
Logged in, added additional IPs to whitelist and re-enabled cphulkd.
Thanks for the info. The database clearing helped me once I had kvm access to my server. Like many others, I’ve added our company to the whitelist. Can’t believe I overlooked that…. *facepalm*
A pure revelation!!! Thank you so much. I’m appreciating CPanel more and more every single day. It’s the best tool for a job with the best user community.
thank a lot !! i was really stuck in this issue
Just a note … you can also whitelist yourself by inserting your IP to CPHulkd’s mysql “whitelist” table via SSH.
# mysql -u root -p
mysql> connect cphulkd;
mysql> insert into whitelist (IP) VALUES (‘YOUR_IP_ADDRESS_HERE’);
Hope that helps!
Hello Chris Jean
Your steps is correct and work for me.
[…] http://chrisjean.com/2009/10/15/fix-whm-cpanel-cphulk-brute-force-protection-lock-out-via-ssh/ […]
[…] http://chrisjean.com/2009/10/15/fix-whm-cpanel-cphulk-brute-force-protection-lock-out-via-ssh/ […]
This is very useful post in simple words,
Excellent Article! it was very well written and very helpfull. THanks.
I wonder why using URL :2087/scripts2/doautofixer?autofix=disable_cphulkd
comand line /usr/local/cpanel/bin/cphulk_pam_ctl –disable
… does not work? Only, flush via MySQL works.
They should add this in more details to docs:
I have found that often the easiest solution is to use a cellphone (not connected on your Wi-Fi, therefore not sharing your current IP); you login to your whm (www.yourdomain/whm) go to the brute force protection as mentioned… and then you remove your ip address, which is what was blocking you. You log off from the cellphone and then try to log into your WHM console using your PC and it should all work… you can easily do this in under five minutes!
I have been checking my logs and today I just randomly noticed I was being hit more then the normal. I was near to just banning all the IPs is this something that I should be ok with doing or should i just remove the IPs from the IP tables? At first I saw 127.0.0.1 but no that cant be right…So I go into mySQL and check exactly what you have done. I got frustrated with the whole list. Now I am able to get into my machine just fine but I really don’t want to be seeing either that someone is trying to attempt to get into my machine still. So to summarize should I ban these IPs on the list?
I’m not sure what logs you are specifically referring to nor do I know the specific requirements of your server, so I couldn’t comment on what you should do. I should point out that 127.0.0.1 refers to the server itself and can pop up in logs because some processes request services from the server itself via networking and not via sockets. This results in 127.0.0.1 being logged.
cphulk seems to have too many false positives. CSF is a better alternative IMO and installs and integrates into cPanel nicely.