Randomly at work, we started getting emails from one of our routers that the “Data Disk” was almost full. This is the first time anyone on the team has experienced this issue. After fumbling through the settings and options on the network appliance I realized we would most likely need to connect via SSH to troubleshoot it. One of my co-workers wasn’t familiar with this or generating SSH Keys. So, I decided to write a quick post about it and the overall resolution.
Creating SSH Keys
There are several ways to do this. I typically use putty to connect via SSH and password but I am not a fan of it with keys. But it is possible, you need to get extra software (PuTTYgen), which I believe will generate .ppk versions of keys that PuTTY is compatible with. I much prefer using bash. If you are on windows you can get bash here. In bash the commands to create new SSH Key pairs are simple and as follows:
First navigate to where you want to store the key pairs on you workstation. A directory like this will work or the default directory of /c/users/username/.ssh will be used.
- cd /c/users/username/keys
Then generate the keys:
- ssh-keygen -t rsa -C “email@example.com”
Next it’ll prompt for a location/name. If you navigated to the directory you want to save it in simply type a name for it. If not you can navigate from this part as well and create the name.
Finally, it will ask for a passphrase which you can enter for extra security or choose to leave blank.
Connecting via ssh with keys
This portion can be different depending on what you are connecting to. Most hardware appliances will provide an option in their web interface to upload your freshly made Key. You want to upload the version that ends with .pub. Next open your fancy bash terminal again and navigate to the directory your keys are stored:
- cd /c/users/username/.ssh
Then run the following command to connect.
- ssh -i “keyname” loginName@ipaddress
After running this you should see a successful connection and be able to run your commands in the terminal.
Resolving the issue
Now that we had gone through creating and connecting via SSH to the network appliance, we can solve the issue. Using the typical linux based diskspace commands I found the folder mainly taking up the most space. After some research on their forums the files in this were just log files. During this research they also suggested to rebuild the postgress database using the command:
- /etc/init.d/postgresssql92 rebuild
Running this reduced the disk space about 10% which was a great initial start. After more thorough research I found it was safe to delete the files I found with my initial disk space commands:
- df -h – I like this initial command to find how much disk space is used by which filesystems. You can then follow navigate to a full one and run:
- du -h – This goes through all the directories and returns the directory size. The -h flag makes it more readable instead of returning large numbers (usually in bytes) it condenses them to KB, MB, GB etc…
- du -ah – This version returns file sizes as well. The a (all) flag I recommend running once in the problematic directory.
Once in the directory with the problematic files:
- rm * – removes all files from current directory.
Creating SSH Keys isn’t overly complicated and is a nice added security feature for many appliances that offer shell access. Appliances that face the outside are at bigger risk and this type of authentication is safer than typical password authentication.