Friday, January 16, 2009

Memory Registry Tools!

Hi everyone! I know it's a bit late, but I made you all a Christmas present: tools for accessing registry data in Windows memory dumps. This the work that I presented at DFRWS 2008; it took a while to release because I had to find time to port it to Volatility 1.3.

To use them, grab either the zip or the tarball and extract it to your Volatility directory. You'll get the following new plugins (along with some supporting files):
  • hivescan: finds the physical address of CMHIVE structures, which represent a registry hives in memory
  • hivelist: takes a physical address of one CMHIVE, returns the virtual address of all hives, and their names
  • printkey: takes a virtual address of a hive and a key name (e.g., 'ControlSet001\Control'), and display the key's timestamp, values and subkeys
  • hashdump: dump the LanMan and NT hashes from the registry (deobfuscated). See this post for more details on how this is accomplished.
  • lsadump: dump the LSA secrets (decrypted) from the registry. See this post for more information.
  • cachedump: dump any cached domain password hashes from the registry. This will obviously only work if the memory image comes from a machine that was part of a domain. See this post for more details.
In general, when you start working with an image you should:
  1. Use hivescan to find the physical address of a hive.
  2. Pass one of those (any of them should do) to hivelist using the -o option. This will give you the virtual address of each of the hives in memory.
  3. With this list in hand, you can now examine individual keys within a hive using printkey, or use hashdump, lsadump, or cachedump to extract credentials from the memory image. The latter three need the virtual addresses of specific hives (SYSTEM and SAM for hashdump, SYSTEM and SECURITY for lsadump and cachedump).
The full list of options available for each command can be obtained with --help. Note that hashdump, lsadump, and cachedump require PyCrypto in order to de-obfuscate the various credentials!

If you're curious how registry access works under the hood, you can read my DFRWS paper or presentation, or check out the blog posts I wrote while doing that research: Enumerating Registry Hives, Reading Open Keys, and Cell Index Translation. And, of course, you can always just look at the source code :)

Some final, closing thoughts. Harlan Carvey's RegRipper has been a huge sucess, and has shown a lot of people how much forensic information is stored in the registry. It would be awesome if this great tool could be applied to memory forensics, but unfortunately I have not yet figured out a way to integrate the two. However, this perl module looks promising; perhaps someone could write a Python wrapper around my registry code that would expose an interface compatible with Parse::Win32Registry, allowing RegRipper to run against a memory image with only slight modifications.

Of course, since RegRipper plugins are all written in Perl, you could also just see what they're doing and reimplement it as a Volatility plugin! To make that easier, here's a quick guide to using the new registry API in Volatility:
  1. Start by opening a hive and assigning it to an address space. The HiveAddressSpace represents a hive in memory, and takes a virtual address space and offset to a hive as a base. Also, because the registry types are not part of Volatility's standard types, you need to add them in.
    from forensics.win32.hive2 import HiveAddressSpace
    from forensics.win32.regtypes import regtypes
    types.update(regtypes)
    syshive = HiveAddressSpace(addr_space, types, syshive_address)

  2. Next, use the get_root() function to find the root key of the hive. You need to pass in the current profile.
    from forensics.win32.rawreg import *
    from forensics.object2 import Profile
    theProfile = Profile()
    root = get_root(syshive, theProfile)

  3. To open a key, use open_key():
    key = open_key(root, ["ControlSet001", "Control", "Lsa"])
    print key.Name

  4. To get a list of subkeys of a key, use subkeys():
    for sk in subkeys(key):
    print sk.Name

  5. To get the values of a key, use values():
    for v in values(key):
    print v.Name

  6. To parse the value data and return it in a more meaningful representation, use value_data():

    for v in values(key):
    val_type, val_data = value_data(v)
    print v.Name, val_type, val_data


That's pretty much all you need to know! You can access other attributes of keys and values as members of the Python objects returned. The attributes available can be found in regtypes.py; keys are _CM_KEY_NODE objects and values are _CM_KEY_VALUE objects.

I don't have time to write any more, unfortunately, but I hope these will be useful to people! I'll try to post an example of a RegRipper plugin converted to a Volatility one soon so there's a concrete example to work with. Until then, enjoy and feel free to e-mail me (or leave comments) with any questions!

10 comments:

Forensics said...

I'm looking for some advice as to what I might be doing wrong. I captured a memory image using mdd from an XP Pro SP2 workstation. Below are the results. The hivescan and hivelist commands work as documented. However, I get numerous python errors with hashdump, lsadump, and cachedump. Any thoughts would be appreciated. I tried this on multiple memory images with the same results from different machines all running XP Pro SP2.

dlh0640s@wkg840sc1 /cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta
$ python volatility hivescan -f mem.dd
Offset (hex)
119184224 0x71a9b60
119449432 0x71ea758
125493256 0x77ae008
283810656 0x10ea9b60
283814752 0x10eaab60
283872096 0x10eb8b60
284084056 0x10eec758
332682664 0x13d455a8
345204144 0x149365b0
347350720 0x14b426c0
348203872 0x14c12b60
452257192 0x1af4e5a8
493259616 0x1d668b60
521558880 0x1f165b60
543526920 0x20659008
654027616 0x26fbab60
795981224 0x2f71b5a8
925971888 0x373135b0
932587200 0x379626c0
953403224 0x38d3c758
956902240 0x39092b60

dlh0640s@wkg840sc1 /cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta
$ python volatility hivelist -o 0x39092b60 -f mem.dd
Address Name
0xe25245b0 \Documents and Settings\NetworkService\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat
0xe25396c0 \Documents and Settings\NetworkService\NTUSER.DAT
0xe17c0b60 \WINDOWS\system32\config\software
0xe17c4758 \WINDOWS\system32\config\default
0xe17c9b60 \WINDOWS\system32\config\SAM
0xe17bcb60 \WINDOWS\system32\config\SECURITY
0xe15f8008 [no name]
0xe1037758 \WINDOWS\system32\config\system
0xe1022b60 [no name]
0xe2561b60 \Documents and Settings\LocalService\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat
0xe23d55a8 \Documents and Settings\LocalService\NTUSER.DAT


dlh0640s@wkg840sc1 /cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta
$ python volatility hashdump -f mem.dd -y 0xe1037758 -s 0xe17c9b60
Traceback (most recent call last):
File "volatility", line 219, in "OPEN BRACKET" module "CLOSE BRACKET"
main()
File "volatility", line 215, in main
command.execute()
File "memory_plugins/registry/hashdump.py", line 67, in execute
dump_memory_hashes(addr_space, types, self.opts.syshive, self.opts.samhive, Profile())
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/win32/hashdump.py", line 302, in dump_memory_hashes
dump_hashes(sysaddr, samaddr, profile)
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/win32/hashdump.py", line 289, in dump_hashes
bootkey = get_bootkey(sysaddr,profile)
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/win32/hashdump.py", line 117, in get_bootkey
cs = find_control_set(sysaddr,profile)
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/win32/hashdump.py", line 105, in find_control_set
root = get_root(sysaddr, profile)
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/win32/rawreg.py", line 67, in get_root
return Object("_CM_KEY_NODE", ROOT_INDEX, address_space, profile=profile)
File "/cygdrive/e/toolkit/FirstResponse/Volatility-1.3_Beta/forensics/object2.py", line 64, in __new__
if MemoryRegistry.OBJECT_CLASSES.objects.has_key(theTypeName):
AttributeError: 'NoneType' object has no attribute 'objects'

Paul Bobby said...

Similar errors

$ python volatility lsadump -y 0xe1035b60 -s 0xe1a3b008 -f rt-4713.001
ERR: Couldn't find subkey PolSecretEncryptionKey of Policy
Traceback (most recent call last):
File "volatility", line 219, in (module)
main()
File "volatility", line 215, in main
command.execute()
File "memory_plugins/registry/lsadump.py", line 79, in execute
secrets = get_memory_secrets(addr_space, types, self.opts.syshive, self.opts.sechive, Profile())
File "/cygdrive/e/Downloads/Volatility-1.3_Beta/forensics/win32/lsasecrets.py", line 139, in get_memory_secrets
return get_secrets(sysaddr, secaddr, profile)
File "/cygdrive/e/Downloads/Volatility-1.3_Beta/forensics/win32/lsasecrets.py", line 130, in get_secrets
secret = decrypt_secret(enc_secret[0xC:], lsakey)
File "/cygdrive/e/Downloads/Volatility-1.3_Beta/forensics/win32/lsasecrets.py", line 70, in decrypt_secret
block_key = key[j:j+7]
TypeError: 'NoneType' object is unsubscriptable

moyix said...

@Paul Bobby:

Your bug is easier, so I'll address it first ;) The hint is the message printed out at the beginning:

ERR: Couldn't find subkey PolSecretEncryptionKey of Policy

My best guess is that the PolSecretEncryptionKey was not in main memory (it had been paged out to disk, or not loaded from the hive in the first place). In this case, open_key returns None, and I need to add a check to that portion of lsadump to bail out gracefully if this happens. Unfortunately it means that the LSA won't be accessible without the original (on-disk) hive.

@Forensics:

Your bug is rather more strange. MemoryRegistry.OBJECT_CLASSES is created when forensics.registry.Init() is called in Volatility, which happens inside the main volatility script -- it should never be None.

Does the pslist_ex_3 command work for you on that image? And, is the version of Volatility you're using the stock 1.3_Beta downloaded from Volatile Systems?

Thanks for the bug reports from you both!

Jon said...
This comment has been removed by the author.
Ago said...

\xe0\xbe one should search for: \xb1\x0cCM10\xe0\xbe\xe0\xbe

Or just simply comment the line "self.add_constraint(self.check_blocksize_equal)" in the file hive2.py.

I hope the autheor will write a win2k3 compatible version sooner or later, but until that this can be a "hotfix".

Or change the self.pool_size = 0x4a8 in the same file to
self.pool_size = 0x588

Jon said...

Brendan,

Any ideas on this error:

C:\Python25>python volatility hashdump -f "C:\Dump\xp-laptop-2005-06-25.img" -y
0xe1035b60 -s 0xe1986008>Password_Hash.txt
Traceback (most recent call last):
File "volatility", line 219, in module
main()
File "volatility", line 215, in main
command.execute()
File "memory_plugins\registry/hashdump.py", line 78, in execute
dump_memory_hashes(addr_space, types, self.opts.syshive, self.opts.samhive,
prof)
File "C:\Python25\forensics\win32\hashdump.py", line 302, in dump_memory_hashe
s
dump_hashes(sysaddr, samaddr, profile)
File "C:\Python25\forensics\win32\hashdump.py", line 290, in dump_hashes
hbootkey = get_hbootkey(samaddr,bootkey,profile)
File "C:\Python25\forensics\win32\hashdump.py", line 155, in get_hbootkey
md5 = MD5.new()
AttributeError: 'module' object has no attribute 'new'

C:\Python25>

On the line 219 area I had to take out a less than and great sign around the word module because the blog thought it was unacceptable html code

I've tried this with both python 2.5 and 2.6. I was also using volatility 1.3 beta.

Thanks in advance.

James said...
This comment has been removed by a blog administrator.
Mark said...

Hello,

I keep getting [Error: Invalid Module [hivescan]. Do I dump all the py files into the main volatility directory? I am currently running this on Windows.

Thanks

moyix said...

Mark,

You should be able to just unpack the zip file or tarball into the main Volatility directory, yes. Could you post the full output when you try to run hivescan? Volatility often prints more detailed errors at the very beginning of the output, which may have scrolled off screen.

Thanks,
Brendan

Mark said...

Moyix to respond to you question see below.

I currently have 1.3 Beta and 1.1.2. First is the output from 1.3 Beta. There are two different commands. Directly below is the "ident" output. As you can see I am running this on a Vmware image memory. I have also tried this on a regular dd image. I get the same output when run on 1.1.2 with out the "\forensics\Win32\crashdump.py31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha".
I have noticed that there is no hivescan.py, just a hivescan2. I have tried both.

Image Name: image.vmem
Image Type: Service Pack 3
VM Type: pae
DTB: 0x319000
Datetime: Fri Jun 25 13:06:12 2010


#1
C:\Volatility-1.3_Beta>python volatility hivescan -h
Error: Invalid module [hivescan].

Volatile Systems Volatility Framework v1.3
Copyright (C) 2007,2008 Volatile Systems
Copyright (C) 2007 Komoku, Inc.



#2
C:\Volatility-1.3_Beta>\forensics\Win32\crashdump.py31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha
Error: Invalid module [hivescan].

Volatile Systems Volatility Framework v1.3
Copyright (C) 2007,2008 Volatile Systems
Copyright (C) 2007 Komoku, Inc.