Exploring XOR Decryption Methods

The use of XOR encryption to keep anti-virus, web filters, and even researchers at-bay have been used for many years. While there are stronger encryption algorithms, the XOR cipher is very easy to add to a project, has low overhead, and is still quite effective even today.

Converter has four ways to help you figure out what the XOR key is. For a single XOR key, you can use Converter's Key Search/Convert. If it's a PE file it may have the string "This program cannot be run in DOS mode" in the header. Let's try that. We just need the first 1KB so import the file and leave the "Import First KB" checkbox checked. Make those setting changes and click on the Search button.

The XOR key of 0x8E was found!

If you're not sure if it is a PE file, you can have Converter enumerate all the XOR values to a text file. To do this, check the "Output First..." box then click on the Search button. Scrolling through the results, you may come across the PE magic and part of the header. The value of 142 is the decimal equivalent of 0x8E.

For XOR keys with more than one value, there's a couple of ways to find the key besides brute forcing each value. One way is import 1KB of the file once again then copy that to the main screen. Choose Stats > Hex Frequency. The output will show you the number of times each hex value has been seen in the input. Notice that eight hex values show up over 80 times. This is likely the XOR key in no particular order. You can now start brute forcing the file with these values. :)

The second way is to paste the imported data and then choose Tools > Pattern Finder. Converter will try to find the longest repeating pattern in the data string. This will take awhile but you will hopefully get decent results like this:

This isn't the actual XOR key. You could copy the output to the input box and have it look for the next longest repeating pattern again (and again). But you don't need to. Here is the result of the first pass:

DCD43089E77A630FDCD43089E77A630FDCD43089E77A630FD
CD43089E77A630FDCD43089E77A630FDCD43089E77A630FDC
D43089E77A630FDCD43089E77A630FDCD43089E77A630FDCD
43089E77A630FDCD43089E77A630FDCD43089E77A630FDCD4
3089E77A630FDCD43089E77A630FDCD43089E77A630FDCD43
089E77A630FDCD43089E77A630FDCD43089E77A630FDCD430
89E77A630FDCD43089E77A630FDCD43089E77A630FDCD4308
9E77A630FD

You can see a pattern with the following string being repeated many times:

DCD43089E77A630F

They look like hex values but we don't know where one ends and the other begins. One clue is to look at the hex frequency screenshot above. It's the same values! But we don't know what is the first value of the XOR key is.

Going back to Key Search/Convert, delete all of the input data except for the first two characters which is 0x80 in this case. Search for "M". We get the result 0xCD which is the first value in the XOR key.

Now we can figure out what the 8-byte key is:

0xCD 0x43 0x08 0x9E 0x77 0xA6 0x30 0xFD

Let's test it out...it works!

A couple of weeks ago, I read an article called, "Efficient Detection of XOR-Encoded Traffic" by Josh Homan (http://www.cloudshield.com/blog/advanced-malware/how-to-efficiently-detect-xor-encoded-content-part-1-of-2/). This concept is not new but for some reason, his excellent write-up caught my eye so I started to study this more closely. I tried using other algorithms besides XOR but I couldn't come up with anything more efficient and effective.

Please check out his article as he did a great job explaining the concept (thanks Josh!). The basic idea is to look for the same distance between two values in the input data. Let's use the same string Josh used in his example, This program cannot be run in DOS mode.

The distance between "T" and "h", "h" and "i", "i" and "s", etc would be the same even though the string has been XOR-encrypted with a single XOR value. If the XOR key is four bytes for example, then you need to offset by the same value and look for the same distance between "T" and " ", "h" and "p", "i" and "r", etc.

I've added the concept to the next version of Converter which I'm still testing. I had to modify the Key Search section to accommodate for this but it seems to work pretty well. I do get false positives when the search string is short or the offset is large.

To use this feature, import 1KB of your data and choose "Calculate Distance...". Converter will pre-fill the Match String for you. If you toggle between Single Key Search and Calculate Distance, the Match String will switch between "This program cannot be run in DOS mode." and "This program must be run under Win32" for you.

If you leave the offset at 0, it will automatically go through the length of your search string and will try to find the first match. Enter an offset if you want to try a specific XOR key length.

The same 8-byte XOR key is found but again, we're not sure what value goes first. Paste the key, set the options, and click on Convert. If that doesn't work, you need to shift the XOR key over by one so just click on the new "Rotate Key" button and it does it automatically.

Once you have your key, use Converter's File > Convert Binary File to process the entire file.

Again, it's not perfect and you will encounter false positives but should give you a great starting point.

So then I thought, why stop at this -- why not have a tool that can automatically XOR-decrypt a file? I needed to find a good live example to test my theory on...

By now I'm sure you have read about GameOver ZeuS now using encryption to get their files past network security. If not, you can read these two writeups:

CyberCrime & Doing Time - GameOver Zeus now uses Encryption to bypass Perimeter Security
CrySyS Blog - GameOver Zeus now uses Encryption to bypass Perimeter Security – .enc encryption

Based on CrySyS' writeup, the downloader program downloads an ".enc" file, XORs the file with a four-byte key, then decompresses it using LZNT1. If I had to do this manually, I would either extract the key from RAM after running the downloader or figuring out the key in Converter as described above. But by using Josh's technique, I can automatically find the key.

Here's the plan for a POC:

  • Open the ".enc" file and check for the "ZZT " magic (0x5A5A5000)
  • Remove the magic bytes
  • Calculate the distance for "program" then shift the XOR key over by one
  • Decrypt the file using the XOR key
  • Write it to a temp file (for debugging purposes)
  • Uncompress the file using LZNT1
  • Write out the PE file

The LZNT1 tool they used isn't able to decompress everything for some reason. I had the same problem too but this works most of the time. There is a debug checkbox that will show you the XOR key it found and preserve that temp file so you can check it out.

I found three live samples here:

hxxp://farmyarddog[.]co[.]uk/images/pdf.enc (97B200826B7A526D91FDA4C56DC438AE)
hxxp://svsmills[.]com/images/pdf.enc (2FF56C80DAE376FED059E12033690DFC)
hxxp://mararu[.]ro/Media/07UKex.enc (9DF4F00EF32806905C3E92286FDE7F0B)

Here's the hex view of one of the files showing its magic bytes:

I run the program on each of the files:

And I get back three decrypted and uncompressed files automatically:

decrypted1.exe (542A5A6F04DDCAD3EFFC72121C59E332)
decrypted2.exe (11E9BF6E9FEDD3EC64163594371BA790)
decrypted3.exe (49686AB2AE3ECF219024A6E4F990AB79)

Here's the hex view of the temp file (after it's XOR'd and before it's decompressed). If you look closely you'll see that there are some null bytes in the DOS header which is why the program isn't searching for the entire header string, just the word "program".

Few notes:

  • If the .enc file doesn't have the magic bytes then the program will end
  • If the XOR keys cannot be found automatically (probably because the file is not a PE), the program will end
  • If the decompression process fails for some reason then you will might a small file with partial results

Posted on: 02/11/2014