Monday, May 18, 2015

GPU Based Password Cracking with Amazon EC2 and oclHashcat

Password cracking is an activity that comes up from time to time in the course of various competitions. While it would be nice to have a dedicated password cracking rig, like anything from Sagitta HPC, it's just not practical for many people myself included. Having access to a GPU cracking machine would be nice from time to time however and the GPU systems that Amazon EC2 supports offers a decent compromise. Doing research on how to setup a password cracking program such as oclHashcat on EC2 came up with results that gave all sorts of conflicting directions. These instructions seem way more straight forward then anything I've come across so hopefully this will be useful for you too. 

First things first. Let's establish a baseline to determine how much better using EC2 will be versus what I typically use during CTFs, a Kali Linux VM. Using hashcat and running it in benchmark mode here are the stats it produces:
Nothing special obviously. Let's see how much of a jump we can get by spinning up a GPU enabled instance on Amazon EC2. I'll assume you've at least gone through the effort of creating an Amazon AWS account. If not, go here first and register yourself. 

First thing is to launch a new instance. From your EC2 dashboard click on the Launch Instance button. 

This will take you to a list of available Amazon Machine Images (AMI). The one we will be using is called the Amazon Linux GRID AMI. Click on the AWS Marketplace tab on the left hand side and search for "Amazon Linux GRID AMI". 

Click the Select button to choose this AMI.

The next screen shows you the available types of EC2 instances that the AMI can run on. Since we're interested in an EC2 instance that supports the NVIDIA GRID GPU driver, the only ones that are available to us are the g2.2xlarge (which has 1 video card) and the g2.8xlarge (which has 4 video cards). For now we will choose the the g2.2xlarge instance. All of the steps in this post work for the g2.8xlarge instance as well. The only difference being that you select g2.8xlarge at this step.

Click the g2.2xlarge row and click on the "Review and Launch " button. You'll be asked if you want to use an SSD or magnetic as the boot volume. I think the SSD boot volume may incur additional costs so I opted to use the magnetic boot volume.
You can click on the "Next: Configure Instance Details" button if you need to configure additional options such as subnets or other specific behaviors. You can also adjust the amount of storage the AMI will have. The default is 8gb which won't hold most decent sized wordlists so you may want to bump that up if you'd like. This won't be necessary for this tutorial though.

The next screen will allow you to review the details of the instance you are about to launch. Nothing special here and you can proceed to click the "Launch" button.

At this point you'll then be asked which key pair you want to use to connect to this instance. Amazon will force you to use a private key file to authenticate to instances. If you already have a PEM key pair you can select that option. I'll assume you have never done this before so I'll go through creating a new key pair. Select "Create a new key pair" from the top drop down box and enter a name for the new key pair you are about to create and click the "Download Key Pair" button. I named mine nova-test-oclhashcat. Keep this file somewhere safe. If you lose it you cannot re-download the file. This means you will never be able to authenticate to the instance again.

Once you have the PEM file downloaded you can click the "Launch Instances" button. Amazon will start the process of spinning up the instance. You will be taken to another site that has a link to your specific instance. In my case, my instance was named i-55b95da9. Click on the link that corresponds to your instance.

Clicking on that link will take you to a status site for your instance. This will give you the information necessary to authenticate to your instance. Take note of the public IP. You will need that piece of information.

The next steps on how to connect to the instance will be from OS X. It should be the same from Linux. However, if you're using Windows and Putty, you'll need to follow a different set of instructions to connect. You can read more about that here.

Open a terminal and navigate to the directory where you downloaded your PEM file. If you do a ls -l on the file you'll see something like this: 

If you go ahead and try to connect using that file, you'll be rejected because the permissions are too lax on it. It has to be a read only file. To fix that we'll need to chmod the file so it's readable only by the user.

Now we can authenticate to the instance via SSH.

Next we will go about installing the latest NVIDIA drivers on our instance by basically following this guide.
First we will uninstall the current NVIDIA driver by typing:
[ec2-user@ip-172-31-9-193 ~]$ sudo yum erase nvidia cuda -y

Next we will download the latest NVIDIA driver. Go here and select these options from the drop down boxes and press the "Select" button:

  • Product Type: GRID
  • Product Series: GRID Series
  • Product: K520
  • Operating System: Linux 64-bit
  • Language: English (US)
  • Recommended/Beta: Recommended/Certified
It should look like this:

Select the newest driver, in my case its version 346.72. Click through the next screen and when you get through to the download page, right click and copy the link the "Agree and Download" button links to.
Go back to your SSH session and download the driver to the instance:
[ec2-user@ip-172-31-9-193 ~]$ wget

Next we will update all of the packages on our instance
[ec2-user@ip-172-31-9-193 ~]$ sudo yum update -y

Then we will reboot our instance so we load the latest kernel version if it happened to be updated. 
[ec2-user@ip-172-31-9-193 ~]$ sudo reboot

Give the instance a few moments to reboot and reconnect. After connecting install the Development Tools package group.
Davids-MBP:~ dave$ ssh -i nova-test-oclhashcat.pem.txt ec2-user@
[ec2-user@ip-172-31-9-193 ~]$ sudo yum groupinstall -y "Development tools"

Then we will install additional files.
[ec2-user@ip-172-31-9-193 ~]$ sudo yum install kernel-devel-`uname -r`

Now we will install the NVIDIA drivers you downloaded earlier. My command will be different from yours if the driver you downloaded is of a different version.
[ec2-user@ip-172-31-9-193 ~]$ sudo /bin/bash

The install process for the NVIDIA driver will ask you a few configuration questions. Here are configurations I performed:
  • Accepted the license
  • Yes to registering the kernel module sources with DKMS
  • Yes to installing the 32-bit compatibility libraries
  • OK to the disclaimer about the libvdpau and libvdpau_trace libraries
  • Yes to running the nvidia-config utility
  • OK to acknowledge the X config file was updated
Then we will once again reboot the instance. 
[ec2-user@ip-172-31-9-193 ~]$ sudo reboot

Give the instance a few moments to reboot and reconnect. After connecting we will verify that the driver if functional:
Now that we have an update to date NVIDIA driver we will install oclHashcat. First we will go here and copy the download link to the latest NVIDIA version of oclHashcat. At the time of this writing it was v1.36. Download this file to your instance.
[ec2-user@ip-172-31-9-193 ~]$ wget

In order to extract this file we need to install p7zip. In order to do that we need to run a few commands to install it.
[ec2-user@ip-172-31-9-193 ~]$ sudo yum-config-manager --enable epel
[ec2-user@ip-172-31-9-193 ~]$ sudo yum install -y p7zip

Now we can extract oclHashcat.
[ec2-user@ip-172-31-9-193 ~]$ 7za x cudaHashcat-1.36.7z

Change directory into where you extracted cudaHashcat-1.36 to and you now have access to oclHashcat:
Let's run some benchmarks to see what the performance is like on this machine:
[ec2-user@ip-172-31-9-193 cudaHashcat-1.36]$ ./cudaHashcat64.bin -b

As we can see on a few of these figures the Amazon instance blows away what I can do on my modest VM. My VM can hash194 words a second of sha512crypt while the Amazon EC2 instance can do 13,457 hashes a second. Similarly my VM can MD5 hash 6.06 million words a second but the EC2 instance can hash 2,492.9 million words a second. Definitely a big jump in performance. If you step up to the 4 GPU EC2 instance the performance scales linearly so you get 4x whatever numbers you see for the 1 GPU instance. 

Don't forget to stop your instance after you're done with it. Go back to the browser tab that has the status of the instance and click on the "Actions" button and select "Instance State" and then "Stop".


  1. Thank you so much for this. Got it running on the g2.8xlarge :D

    1. Thanks for the post, I am techno savvy. I believe you hit the nail right on the head. I am highly impressed with your blog. It is very nicely explained. Your article adds best knowledge to our Java Online Training from India. or learn thru Java Online Training from India Students.

  2. Thank you very much! Was 100% helpful

  3. hello

    i want to crack md5 hashes generated under 3 minutes, because the hash is changed under 4 min.

    here is a sample with the result :

    MD5 Hash : e4bbbc684cd1a9d8801a3fe4e854511d
    Result : aJ0DOc2uoXPL4v9khxWZ-1

    *** i using many software like hashcat_cuda and bars but i think with and bruteforce method no one do that .
    those of "-1" or "-0" dash and one or zero digits was fixed at the end of any generated hashes .
    i just need the end number of result (( "1" or "0" )) .

    many hash is exist and i need a developed software or decryption method to do this for many time .
    it's very very important for me .

    does anyone exist to do that ????
    awaiting for answer ...

    if you cracked this i sent $1000 per month for you

  4. This comment has been removed by the author.

  5. it says my instance limit is 0. i had to request an increase. are they catching on to this?

    1. No we don't, its just a limit that "we" had here, ask for the increase and it will be increased :)

  6. I love aws but find aws interface very overwhelming and confusing. They should simplify the interface.

  7. Viewing applications, monitoring chat messangers, etc. you can have here have a glimpse at this link to find more information.

  8. Thanks for providing this information. Maybe you will also be willing to try to learn how to hack someone’s phone for free.

  9. Really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
    Aws Online Training

  10. Expected to form you a next to no word to thank you once more with respect to the decent recommendations you've contributed here.
    Hadoop Training in Marathahalli

  11. I simply wanted to write down a quick word to say thanks to you for
    those wonderful tips and hints you are showing on this site.

    AWS Training in Chennai

    AWS Training in Bangalore

    AWS Training in Bangalore

  12. • Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updatingAzure Online Training

  13. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.
    devops training in bangalore

  14. I was searching for the better blog on Sales force CRM techniques to understand what CRM is and what is the use? Your blog is the right one I found to get depth knowledge. Thanks for sharing your wonderful ideas.
    Salesforce Training in Chennai | Salesforce Training

  15. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.

    java training in bangalore

  16. It’s great to come across a blog every once in a while that isn’t
    the same out of date rehashed material. Fantastic read.
    dotnet training in bangalore

  17. Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post that is written in well manner.
    mobile website builder

  18. • Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.

    AWS Training in Bangalore|

  19. Appreciating the persistence you put into your blog and detailed information you provide.

    It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    AWS Training in Chennai

  20. Expected to form you a next to no word to thank you once more with respect to the decent recommendations you've contributed here.
    diploma in fire and safety course in chennai

  21. As this blog looking so interesting I would like to read regularly so that I can get more stuff around this area.
    Best Online Software Training Institute | AWS Training

  22. I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.

    Best Hadoop Training Institute In chennai

  23. AWS EC2 known as Amazon Web Service Elastic Compute Cloud is a modern development in the web service that assures the user in getting a securable, resizable compute capability in the cloud version. It has been designed in such a way that it will help the developers to create web-scale computing straightforwardly. This new version will help you to take control of all your computer system and run on the proved computing environment of Amazon. With minimal resistance the simple service of the Amazon AWS EC2 edge allows you to attain and organize the capacity. The Best AWS Technical Training

  24. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.

    Amazon Web Services Training in Chennai

  25. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here. aws training in chennai

  26. This website is really a walk-by way of for all of the information you wished about this and didn’t know who to ask. Glimpse right here, and also you’ll undoubtedly uncover it.
    Aws Training in Hyderabad

  27. I wish to show thanks to you just for bailing me out of this particular trouble. As a result of checking through the net and meeting techniques that were not productive, I thought my life was done.
    Diploma in Fire and Safety Course in Chennai

  28. Very useful information and interesting topic.

    aws training in Hyderabad

  29. Informative Post! Thank you such a great amount for sharing. This pretty post, it was so great to peruse and helpful to enhance my insight as refreshed one, continue blogging... Vendor Reconciliation | CA Firms | Warehouse Audit

  30. Very useful information to everyone thanks for sharing, learn the latest updated Technology at Best Training institutions
    Salesforce Lightning is the latest updated technology
    Salesforce Online Training in Bangalore
    Salesforce Training in india

  31. I wish to show thanks to you just for bailing me out of this particular
    trouble.As a result of checking through the net and meeting
    techniques that were not productive, I thought my life was done.
    CCNA Training in Chennai | CCNA Training in Velachery

  32. DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on but have the second-highest talent deficit.
    Are you seeing DevOps in your future? Perhaps you are already exploring where to start learning DevOps, choose myTectra the market leader in DevOps Training.