GPU Based Password Cracking with Amazon EC2 and oclHashcat

Password cracking is an activity that comes up from time to time in the course of various competitions. While it would be nice to have a dedicated password cracking rig, like anything from Sagitta HPC, it's just not practical for many people myself included. Having access to a GPU cracking machine would be nice from time to time however and the GPU systems that Amazon EC2 supports offers a decent compromise. Doing research on how to setup a password cracking program such as oclHashcat on EC2 came up with results that gave all sorts of conflicting directions. These instructions seem way more straight forward then anything I've come across so hopefully this will be useful for you too. 

First things first. Let's establish a baseline to determine how much better using EC2 will be versus what I typically use during CTFs, a Kali Linux VM. Using hashcat and running it in benchmark mode here are the stats it produces:
Nothing special obviously. Let's see how much of a jump we can get by spinning up a GPU enabled instance on Amazon EC2. I'll assume you've at least gone through the effort of creating an Amazon AWS account. If not, go here first and register yourself. 

First thing is to launch a new instance. From your EC2 dashboard click on the Launch Instance button. 

This will take you to a list of available Amazon Machine Images (AMI). The one we will be using is called the Amazon Linux GRID AMI. Click on the AWS Marketplace tab on the left hand side and search for "Amazon Linux GRID AMI". 

Click the Select button to choose this AMI.

The next screen shows you the available types of EC2 instances that the AMI can run on. Since we're interested in an EC2 instance that supports the NVIDIA GRID GPU driver, the only ones that are available to us are the g2.2xlarge (which has 1 video card) and the g2.8xlarge (which has 4 video cards). For now we will choose the the g2.2xlarge instance. All of the steps in this post work for the g2.8xlarge instance as well. The only difference being that you select g2.8xlarge at this step.

Click the g2.2xlarge row and click on the "Review and Launch " button. You'll be asked if you want to use an SSD or magnetic as the boot volume. I think the SSD boot volume may incur additional costs so I opted to use the magnetic boot volume.
You can click on the "Next: Configure Instance Details" button if you need to configure additional options such as subnets or other specific behaviors. You can also adjust the amount of storage the AMI will have. The default is 8gb which won't hold most decent sized wordlists so you may want to bump that up if you'd like. This won't be necessary for this tutorial though.

The next screen will allow you to review the details of the instance you are about to launch. Nothing special here and you can proceed to click the "Launch" button.

At this point you'll then be asked which key pair you want to use to connect to this instance. Amazon will force you to use a private key file to authenticate to instances. If you already have a PEM key pair you can select that option. I'll assume you have never done this before so I'll go through creating a new key pair. Select "Create a new key pair" from the top drop down box and enter a name for the new key pair you are about to create and click the "Download Key Pair" button. I named mine nova-test-oclhashcat. Keep this file somewhere safe. If you lose it you cannot re-download the file. This means you will never be able to authenticate to the instance again.

Once you have the PEM file downloaded you can click the "Launch Instances" button. Amazon will start the process of spinning up the instance. You will be taken to another site that has a link to your specific instance. In my case, my instance was named i-55b95da9. Click on the link that corresponds to your instance.

Clicking on that link will take you to a status site for your instance. This will give you the information necessary to authenticate to your instance. Take note of the public IP. You will need that piece of information.

The next steps on how to connect to the instance will be from OS X. It should be the same from Linux. However, if you're using Windows and Putty, you'll need to follow a different set of instructions to connect. You can read more about that here.

Open a terminal and navigate to the directory where you downloaded your PEM file. If you do a ls -l on the file you'll see something like this: 

If you go ahead and try to connect using that file, you'll be rejected because the permissions are too lax on it. It has to be a read only file. To fix that we'll need to chmod the file so it's readable only by the user.

Now we can authenticate to the instance via SSH.

Next we will go about installing the latest NVIDIA drivers on our instance by basically following this guide.
First we will uninstall the current NVIDIA driver by typing:
[ec2-user@ip-172-31-9-193 ~]$ sudo yum erase nvidia cuda -y

Next we will download the latest NVIDIA driver. Go here and select these options from the drop down boxes and press the "Select" button:

• Product Type: GRID
• Product Series: GRID Series
• Product: K520
• Operating System: Linux 64-bit
• Language: English (US)
• Recommended/Beta: Recommended/Certified

It should look like this:

Select the newest driver, in my case its version 346.72. Click through the next screen and when you get through to the download page, right click and copy the link the "Agree and Download" button links to.

Go back to your SSH session and download the driver to the instance:

[ec2-user@ip-172-31-9-193 ~]$ wget http://us.download.nvidia.com/XFree86/Linux-x86_64/346.72/NVIDIA-Linux-x86_64-346.72.run

Next we will update all of the packages on our instance

[ec2-user@ip-172-31-9-193 ~]$ sudo yum update -y

Then we will reboot our instance so we load the latest kernel version if it happened to be updated. 

[ec2-user@ip-172-31-9-193 ~]$ sudo reboot

Give the instance a few moments to reboot and reconnect. After connecting install the Development Tools package group.

Davids-MBP:~ dave$ ssh -i nova-test-oclhashcat.pem.txt ec2-user@52.7.150.101

[ec2-user@ip-172-31-9-193 ~]$ sudo yum groupinstall -y "Development tools"

Then we will install additional files.

[ec2-user@ip-172-31-9-193 ~]$ sudo yum install kernel-devel-`uname -r`

Now we will install the NVIDIA drivers you downloaded earlier. My command will be different from yours if the driver you downloaded is of a different version.

[ec2-user@ip-172-31-9-193 ~]$ sudo /bin/bash NVIDIA-Linux-x86_64-346.72.run

The install process for the NVIDIA driver will ask you a few configuration questions. Here are configurations I performed:

• Accepted the license
• Yes to registering the kernel module sources with DKMS
• Yes to installing the 32-bit compatibility libraries
• OK to the disclaimer about the libvdpau and libvdpau_trace libraries
• Yes to running the nvidia-config utility
• OK to acknowledge the X config file was updated

Then we will once again reboot the instance. 

[ec2-user@ip-172-31-9-193 ~]$ sudo reboot

Give the instance a few moments to reboot and reconnect. After connecting we will verify that the driver if functional:

Now that we have an update to date NVIDIA driver we will install oclHashcat. First we will go here and copy the download link to the latest NVIDIA version of oclHashcat. At the time of this writing it was v1.36. Download this file to your instance.

[ec2-user@ip-172-31-9-193 ~]$ wget http://hashcat.net/files/cudaHashcat-1.36.7z

In order to extract this file we need to install p7zip. In order to do that we need to run a few commands to install it.

[ec2-user@ip-172-31-9-193 ~]$ sudo yum-config-manager --enable epel

[ec2-user@ip-172-31-9-193 ~]$ sudo yum install -y p7zip

Now we can extract oclHashcat.

[ec2-user@ip-172-31-9-193 ~]$ 7za x cudaHashcat-1.36.7z

Change directory into where you extracted cudaHashcat-1.36 to and you now have access to oclHashcat:

Let's run some benchmarks to see what the performance is like on this machine:

[ec2-user@ip-172-31-9-193 cudaHashcat-1.36]$ ./cudaHashcat64.bin -b

As we can see on a few of these figures the Amazon instance blows away what I can do on my modest VM. My VM can hash194 words a second of sha512crypt while the Amazon EC2 instance can do 13,457 hashes a second. Similarly my VM can MD5 hash 6.06 million words a second but the EC2 instance can hash 2,492.9 million words a second. Definitely a big jump in performance. If you step up to the 4 GPU EC2 instance the performance scales linearly so you get 4x whatever numbers you see for the 1 GPU instance. 

Don't forget to stop your instance after you're done with it. Go back to the browser tab that has the status of the instance and click on the "Actions" button and select "Instance State" and then "Stop".