As my knowledge in deploying Linux servers grew so did my confidence. This confidence saw me offering to setup webservers for others. In this particular case I had built out a server for my fiance, who wanted to run a blog and store for her crafts. A few months into it being setup and running well she ran into problems where her website would be extremely slow or unresponsive. Rebooting the server would sometimes yield “error establishing database connection”. Here are some of my troubleshooting attempts and eventual fix to her unresponsive wordpress website.
I initially started by rebooting the server like all the cliche tech jokes infer. When the server came back up everything seemed fine again. However, shortly after it started to become slower and slower until again it was unresponsive. I connected back to the server and noticed that just logging in seemed slow. I checked the processes using the top command, which showed several Apache2 instances running and a high MySQL process. Both these services seemed to be taxing the servers resources. I noticed after killing and restarting services that eventually MySQL would crash giving an “Error establishing database connection” when trying to visit the wordpress site.
(Generic top command screenshot)
After browsing through Apache2 and MySQL logs I found nothing out of the ordinary. MySQL was crashing because because there were no available resources but I didn’t see any specific reason for the crash. I adjusted some of the apache2.conf settings for max clients and max requests per clients, which I thought resolved it. Approximately an hour later the issue reoccurred. I began thinking either Apache2 was broken or her site was hacked, typical wordpress. At this point, I assumed rebuilding the webserver was the easiest out since it was only hosting 1 website. Before a rebuild take a good backup!
Backing up WordPress Site
To backup a wordpress site you need the folder structure and the SQL file. This is where PHPmyadmin comes in handy! First, I connected to the webserver using my go to FTP client Filezilla. I navigated to /var/www/html/ where the website folders lived and downloaded the whole parent folder for her website. Next, I used PHPmyadmin to download the SQL file. This is simple. Just log into PHPmyadmin find the SQL database for the website, select it and click export.
Everything I needed was now backed up. The rest I wanted to re configure like apache2.conf, php.ini and mysql.cnf. So, it was time to rebuild the server. Her webhost had an easy utility to log in, delete the server and deploy a new one!
Finding the actual problem
I had a fresh new server completely reconfigured and the database/folder structure uploaded. However, after about another hour the website became unresponsive again. MySQL also started to crash. I scoured the internet for information or clues on the issue until an epiphany happened. All the apache2 instances were coming from sessions connected to the website. I checked the apache2 access.log file which by default is found in /var/log/apache2/ unless this location has been manually moved in the apache2.conf file. Boom! I found it. The log file showed an IP hitting the site constantly. A quick who is by IP showed it was a Russian IP from a server farm. At this point I was not very sufficient with IP tables but I knew enough to drop an IP.
- sudo iptables -A INPUT -s xxx.xxx.xxx.xxx -j DROP
After running this command the issue was cleared. The site remained up and running normally. I am unsure why her website was targeted for this DOS attack or brute force attack seeing as it didn’t contain much content. But the problem was resolved.
Yet again it was time to celebrate. But also it was time to reflect. It is pretty crazy how deep you get into something technically and how far you look into something to find the answer right under your nose. I spent hours attempting to fix an issue that could be resolved in minutes. You live and learn though and I will never forget this experience. Nor will I forget to check for something like this next time. The unresponsive wordpress website was fixed!