MPI and Password Cracking Author: Jason R. Davis Site: TMTO[dot]ORG Date: May 28th, 2009 Table of Contents: Introduction Directory Structure Methodology Hardware and Software Installing OpenMPI Installing John the Ripper MPI Utilizing CPU Cores on a Standalone PC Outro Introduction After sauntering through the web for many hours searching for GOOD documentation on this topic, I realized that it simply does not exist. I’ve been using MPI, both OpenMPI and MPICH2, for a little over two years and have become familiar with implementing it effectively in production environments. I also have experience with setting up scalable password cracking utilities that operate through MPI. With that said, I couldn’t sit on this knowledge anymore, and went through the laborious task of documenting it for public release. I will be updating this documentation in the future to include instructions on how to scale this to a full cluster. For now, it’ll let you utilize all the cores on a single PC for cracking, instead of just one core. The goals: Setup a cluster like directory structure that is scalable Install OpenMPI Install John the Ripper MPI Run a simple test and crack a MD5 hash Directory Structure Methodology This is nearly the most important part of building a system that is scalable and won’t break if you upgrade to a newer version of an application. I am going to quickly outline how I build a directory structure for a single machine. That structure is then scalable to a cluster. /apps/ /apps/myapp1/ /apps/myapp1/v1.x/ /apps/myapp1/v1.x/install/ /apps/myapp1/v1.x/x86_32/ /apps/myapp1/v1.x/x86_64/ /apps/myapp1/v2.x/ /apps/myapp1/v2.x/install/ /apps/myapp1/v2.x/x86_32/ /apps/myapp1/v2.x/x86_64/ /apps/myapp1/latest ? v2.x/ /apps/myapp1/stable ? v1.x/ /apps/myapp1/env/ /apps/ This is the base directory for all our cluster applications. /apps/myapp1/ For each application we create a subdirectory of our base directory, in this case "myapp1" /apps/myapp1/v1.x/ The stable version of the application will go below this directory structure. This is where the production application will reside. You will use the actual version number of the application for the directory name. You will not use "v1.x". /apps/myapp1/v1.x/install/ The source code for this specific version will be downloaded, unpacked, and compiled from this directory. /apps/myapp1/v1.x/x86_32/ If the application was compiled on a 32-bit processor, then this is where the application will be installed and ran from. /apps/myapp1/v1.x/x86_64/ If the application was compiled on a 64-bit processor, then this is where the application will be installed and ran from. /apps/myapp1/v2.x/ The latest version of the application will go below this directory structure. This is where the alpha, beta, or unstable release will reside. This includes applications that need to be tested and are not ready for production. You will use the actual version number of the application for the directory name. You will not use "v2.x". /apps/myapp1/v2.x/install/ The source code for this specific version will be downloaded, unpacked, and compiled from this directory. /apps/myapp1/v2.x/x86_32/ If the application was compiled on a 32-bit processor, then this is where the application will be installed and ran from. /apps/myapp1/v2.x/x86_64/ If the application was compiled on a 64-bit processor, then this is where the application will be installed and ran from. /apps/myapp1/latest ? v2.x This is a symbolic link "latest" pointing to the directory named after the unstable installed version. /apps/myapp1/stable ? v1.x This is a symbolic link "stable" pointing to the directory named after the stable installed version These symbolic links will allow you to upgrade software and not need to rebuild the environment to accommodate the change. Simply modify the symbolic link to point to the newer version, still under the name of latest. If I have "/apps/myapp1/v1.x/bin/" in the $PATH variable for all users .bashrc, then I would have to update all the user .bashrc files if I wanted to upgrade everyone to "/apps/myapp1/v2.x/bin/". By using the "stable" and "latest" symbolic links, you only need to put "/apps/myapp1/stable/bin/" in each users .bashrc file. And when you want to migrate all users to "/apps/myapp1/v2.x/bin/" simply modify the "stable" symbolic link to point to it. All the users will automatically start using the newest version on a simple change to the symbolic links. This is very important for scalability and allows you to test new releases without affecting the other users by putting "/apps/myapp1/latest/bin/" into your $PATH. /apps/myapp1/env/ This is where scripts and files that modify the user environment will be placed Hardware and Software For this process I’ve chosen Gentoo as the GNU/Linux distribution. I’ve installed it onto a server containing a single "Intel(R) Xeon(R) E5405 @ 2.00GHz" which happens to be a Quad Core processor. Which is why I use "-np 4" in the mpirun commands. Also note, I am using a 64-bit OS and software. Pay attention to your version and make the minor changes to the commands to accommodate that. Installing OpenMPI OpenMPI v1.3.2 (Latest) / OpenMPI v1.3.1 (Stable) ~ # mkdir /apps ~ # mkdir /apps/openmpi Install v1.3.2: ~ # mkdir /apps/openmpi/v1.3.2 ~ # ln -s /apps/openmpi/v1.3.2 /apps/openmpi/latest ~ # mkdir /apps/openmpi/v1.3.2/install ~ # mkdir /apps/openmpi/v1.3.2/x86_32 ~ # mkdir /apps/openmpi/v1.3.2/x86_64 ~ # cd /apps/openmpi/v1.3.2/install ~ # wget http://www.open-mpi.org/software/ompi/v1.3/downloads/openmpi-1.3.2.tar.gz ~ # tar -xf openmpi-1.3.2.tar.gz ~ # cd openmpi-1.3.2 For 32-bit systems: ~ # ./configure --prefix= /apps/openmpi/v1.3.2/x86_32 For 64-bit systems: ~ # ./configure --prefix=/apps/openmpi/v1.3.2/x86_64 ~ # make ~ # make install Install v1.3.1: ~ # mkdir /apps/openmpi/v1.3.1 ~ # ln -s /apps/openmpi/v1.3.1 /apps/openmpi/stable ~ # mkdir /apps/openmpi/v1.3.1/install ~ # mkdir /apps/openmpi/v1.3.1/x86_32 ~ # mkdir /apps/openmpi/v1.3.1/x86_64 ~ # cd /apps/openmpi/v1.3.1/install ~ # wget http://www.open-mpi.org/software/ompi/v1.3/downloads/openmpi-1.3.1.tar.gz ~ # tar -xf openmpi-1.3.1.tar.gz ~ # cd openmpi-1.3.1 For 32-bit systems: ~ # ./configure --prefix= /apps/openmpi/v1.3.1/x86_32 For 64-bit systems: ~ # ./configure --prefix=/apps/openmpi/v1.3.1/x86_64 ~ # make ~ # make install ~ # mkdir /apps/openmpi/env ~ # cd /apps/openmpi/env ~ # echo ‘export PATH=${PATH}:/apps/openmpi/latest/x86_64/bin’ > prepenv_latest ~ # echo ‘export PATH=${PATH}:/apps/openmpi/stable/x86_64/bin’ > prepenv_stable Edit /etc/env.d/00basic and modify the LDPATH to include the OpenMPI libraries: LDPATH="/usr/local/lib:/apps/openmpi/latest/x86_64/lib:/apps/openmpi/stable/x86_64/lib" ~ # env-update && source /etc/profile Running a OpenMPI Sanity Check: ~ # source /apps/openmpi/env/prepenv_latest ~ # mpirun -np 4 hostname The last command above will print the computers hostname four times. Not less, not more, four times. If it does, then your "latest" OpenMPI version is working. ~ # env-update && source /etc/profile ~ # source /apps/openmpi/env/prepenv_stable ~ # mpirun -np 4 hostname The last command above will print the computers hostname four times. Not less, not more, four times. If it does, then your "stable" OpenMPI version is working. At this point you should have a functioning OpenMPI install. Two versions, latest and stable, are available based upon which file you source. The reason I force users to login and "source" the "prepenv_xxxxxx" file is because I don’t want to add bloat to user profiles. I despise $PATH variables that are 30+ directories long, etc. Less is more in this case. I don’t add any information to users .bashrc in this case. Some users actually put the "source /apps/openmpi/env/prepenv_stable" command in their .bashrc, which makes OpenMPI available at the time they login by setting the $PATH. Installing John the Ripper MPI Note: You cannot install John the Ripper MPI unless you have a functioning MPI environment. So do not continue unless you have successfully installed OpenMPI and completed the sanity check. John the Ripper MPI v1.7.3.1 (Latest) ~ # mkdir /apps/jtr Install v1.7.3.1: ~ # mkdir /apps/jtr/v1.7.3.1 ~ # ln -s /apps/jtr/v1.7.3.1 /apps/jtr/latest ~ # mkdir /apps/jtr/v1.7.3.1/install ~ # mkdir /apps/jtr/v1.7.3.1/x86_32 ~ # mkdir /apps/jtr/v1.7.3.1/x86_64 ~ # cd /apps/jtr/v1.7.3.1/install ~ # wget http://www.bindshell.net/tools/johntheripper/john-1.7.3.1-all-2-mpi8.tar.gz ~ # tar -xf john-1.7.3.1-all-2-mpi8.tar.gz ~ # cd john-1.7.3.1-all-2-mpi8/src For 32-bit systems: ~ # make linux-x86-sse2 For 64-bit systems: ~ # make linux-x86-64 ~ # cd /apps/jtr/v1.7.3.1/install/john-1.7.3.1-all-2-mpi8/run ~ # cp * /apps/jtr/v1.7.3.1/x86_64/ ~ # mkdir /apps/jtr/env ~ # cd /apps/jtr/env ~ # echo ‘export PATH=${PATH}:/apps/jtr/latest/x86_64’ > prepenv_latest ~ # echo ‘source /apps/openmpi/env/prepenv_latest’ >> prepenv_latest ~ # echo ‘cd /apps/jtr/latest/x86_64’ >> prepenv_latest Running a John the Ripper Sanity Check: ~ # env-update && source /etc/profile ~ # source /apps/jtr/env/prepenv_latest ~ # mpirun -np 4 ./john -test The last command above will self test John the Ripper MPI and run a benchmark to see how fast your computer can run it. Utilizing CPU Cores on a Standalone PC At this point you should have a functioning MPI environment and a functioning version of John the Ripper. So now that the foundation is laid, let us move on to the fun part: Cracking Passwords! The first task we need to complete is to create a cleanup script for John the Ripper, because it creates a lot of files that don’t get cleaned up after its runs. Only run this if you no longer need the john_log and john.pot files, because it removes them as well. ~ # cd /apps/jtr/env ~ # echo "#!/bin/bash" > cleanup_latest.sh ~ # echo "rm /apps/jtr/latest/x86_64/john_rec.rec*" >> cleanup_latest.sh ~ # echo "rm /apps/jtr/latest/x86_64/john_log" >> cleanup_latest.sh ~ # echo "rm /apps/jtr/latest/x86_64/john.pot" >> cleanup_latest.sh ~ # chmod + x cleanup_latest.sh The entire platform is setup now. Next we will begin a test run, with a simple MD5 hash. ~ # env-update && source /etc/profile ~ # source /apps/jtr/env/prepenv_latest ~ # echo "root:e80b5017098950fc58aad83c8c14978e" >> md5s.txt ~ # mpirun -np 4 ./john -format=raw-md5 -incremental:alpha ./md5s.txt Loaded 1 password hash (Raw MD5 [raw-md5]) Loaded 1 password hash (Raw MD5 [raw-md5]) Loaded 1 password hash (Raw MD5 [raw-md5]) Loaded 1 password hash (Raw MD5 [raw-md5]) abcdef (root) Process 0 completed loop. thread: 0 guesses: 1 time: 0:00:00:00 c/s: 1582K trying: abaflm - abcdch The last command above initiated John the Ripper MPI utilizing all 4 cores and successfully broke the password in under a second. The password was "abcdef". If you have questions about the syntax for running John the Ripper, please read the related docs. After each run, I usually copy the "john.pot" file to safe spot, and then cleanup: ~ # /apps/jtr/env/cleanup_latest.sh Outro As mentioned in at the beginning of this document, I will be amending updates to show users how to expand to a cluster, spanning many PCs and many CPU cores. If you have trouble, utilize Google as much as possible; every error I came across had an answer and a quick Google search brought it right to me. Jason R. Davis TMTO[dot]ORG # milw0rm.com [2009-06-08]