Sometimes people ask me for career advice related to information security in general and, more specifically, digital forensics and incident response. I’ve written a few articles on this topic, as did many other respected professionals. Below are pointers to some of these tips.
Digital forensics in general:
Specific to malware analysis:
Broader IT and information security career tips:
I’m sure I missed many other excellent articles with practical career tips for digital forensics and related fields. If you’d like to recommend your favorite references, kindly leave a comment.
In the field of IT in general and digital forensics in particular, you become obsolete the moment you stop learning. Here are several free recorded webcasts related to reverse-engineering and malware analysis that will help you keep your skills up to date:
The need to define custom, incident-specific signatures is slowly gaining traction in the mainstream enterprise. A few years ago this concept, often called Indicators of Compromise (IOCs), was mostly discussed by government organizations and defense contractors who were coming to terms with Advanced Persistent Threat (APT) attacks.
Madiant began popularizing the term IOC around 2007. Kris Kendall’s paper Practical Malware Analysis mentioned IOCs in the context of malware reversing at Black Hat DC 2007. For a precursor to this, see Kevin Mandia’s Foreign Attacks on Corporate America slides from Black Hat Federal 2006. At the time, few organizations saw the need to go beyond antivirus-based detection by analyzing the adversary’s artifacts to define custom host-level signatures.
Now, several years later, the term IOC is pretty well-known in the infosec industry. More companies are adding malware and related analysis skills to incident response teams. As Jake Williams put it, such firms know how to examine new malware and extract IOCs. “These are then fed back into the system and scans are repeated until no new malware is found.” Automated analysis products from vendors such as Norman, Mandiant, FireEye and HB Gary are being increasingly positioned as IR triage-enablers.
That said, the knowledge and skills for deriving and using IOCs is far from being mainstream. Anton Chuvakin highlighted the distinction between security haves and have-nots along the lines of this capability. The haves know how to reverse-engineer malware to “extract the IOCs FAST (or get those IOCs shared with you by trusted friends) and then look for them on other systems.”
IOC techniques haven’t entered the mainstream just yet. But we’re heading in that direction, as more people attain forensics skills and as more tools become available for defining and making use of such custom, incident-specific signatures.
To learn how to define and make use of IOCs, take a look at:
I had the pleasure of speaking with Jake Williams, my colleague at SANS Institute, about his perspective on various malware analysis and reverse-engineering topics. You can read the interview in three parts:
Jake is highly experienced in this space and shared helpful insights in the interview above. Jake will be teaching FOR610: Reverse-Engineering Malware on several occasions at SANS this year.
Incident responders and forensic investigators need to be careful when using 32-bit tools to examine file system artifacts on 64-bit Windows. Christian Wojner documented the issue in a paper titled The WOW-Effect. He demonstrated how the WOW64 File System Redirector built into 64-bit Windows transparently redirects 32-bit tools’ access to core OS directories and registry values. This is likely to confuse forensics personnel performing live analysis of the system.
Christian explained that “64‐bit versions of Windows provide backward compatibility for 32‐bit executables.” For instance, a 32-bit application accessing %windir%\System32 will be transparently redirected by the OS to access a corresponding file in the %windir%\SysWOW64 directory. “In other words, a 32‐bit application will never see any file stored in System32, it will always access SysWOW64 instead.”
Consider the situation, described in Christian’s paper, where the investigator uses the classic md5sum tool (which is usually 32-bit) to look at file that corresponds to a suspicious process…
On my 64-bit Windows 7 laboratory system, I called a demo file “myfile.exe” (0d07469c97791d2d51a6317f84f75025) and placed it in the C:\Windows\System32 directory. I placed a different file (ec75883fdc02c259026e0585e720dfe7) under the same name in C:\Windows\SysWOW64.
I used 32-bit md5sum to read the file from C:\Windows\System32. The OS redirected the program to read from C:\Windows\SysWOW64 without any visual indications that this occurred:
However, when I use a 64-bit tool for calculating the MD5 hash of the same file, I get a different value. This value corresponds to the actual file under C:\Windows\System32 that I intended to examine:
That’s because when the 32-bit md5sum program attempted to read C:\Windows\System32\myfile.exe, the OS transparently redirected it to C:\Windows\SysWOW64\myfile.exe.
You can see a video of my little demo on YouTube.
Christian pointed out another scenario that might mislead incident responders: It’s not uncommon for investigators to upload suspicious files to VirusTotal for malware scanning. However, if the investigator uses a 32-bit browser, then he or she will be uploading a different file than what would be uploaded using a 64-bit browser.
If you’re concerned about this issue, stick to using 64-bit tools when performing live analysis of 64-bit Windows systems. If you’re using 32-bit tools, there is another approach to accessing contents of C:\Windows32, though this doesn’t work for other files and registry items that Windows might redirect. Instead of attempting to access %windir%\System32, access %windir%\Sysnative to avoid redirection. Thanks to Tyler Hudak for telling me about this:
For more details and examples regarding this issue, see Christian’s The WOW-Effect paper.
The field of digital forensics and incident response (DFIR) is attracting a lot attention among information security professionals and law enforcement officers seeking to progress in their careers. One of the challenges of entering this field is that employers often limit their recruitment efforts to experienced forensicators. What can people seeking to get into this industry do?
It seems that organizations rarely want to invest into growing the skills of a beginner forensics or IR analyst. As the result, individuals seeking to get into DFIR should look for opportunities to pick up relevant skills as part of their current job responsibilities. Some ideas and examples:
The idea is to obtain some baseline DFIR knowledge by building upon what you already know. Look for ways to do this in the context of your current job responsibilities without undermining your commitments to your employer. Supplement the research and experimentation you can do at work with studying and exploring on your own time. Read books on the relevant topics, keep up with DFIR blogs and take formal training if your budget allows. Participate in online forms and informal meet-ups. Talk to people who currently work in DFIR.
Once you learn a bit about DFIR through informal exploration, reading and studying, start looking for a job—in your organization or elsewhere—that can provide you with experiences and mentoring in the aspect of digital forensics and interest response that interests you. Don’t forget to incorporate what you’ve learned about DFIR into your resume, of course.
There are many ways to enter a given field, and everyone’s approach might be different. What are your tips for people interested in getting into DFIR? What has worked for you?
Update: For a perspective on this topic from Harlan Carvey, see his Getting Started post.
Hand-picked related posts:
Just so you know, I teach the malware analysis course at SANS Institute.
Some time ago I wrote about the importance of deliberate practice for developing information security skills. Practice is scritical to improving expertise in all aspects of infosec, including malware analysis. You will get better if you take the time to experiment with malware in a laboratory environment, building upon what you may have read or were taught to practical obtain hands-on expertise.
That’s why I’m a big fan of puzzles and exercises designed for experimenting with malware. If you like learning in this manner as well, you might enjoy the malware analysis challenge I published on the SANS forensics blog:
About a year ago I collaborated with the folks at Lake Missoula Group to create a malware-themed network forensics puzzle. That contest is now over; however, I would like to provide an opportunity to learn from the scenario defined in that puzzle to strengthen your malware analysis skills.
There is no prize for correctly answering the 7 questions in that challenge. It’s merely an opportunity for you to learn malware analysis through practice.
For additional exercises related to analyzing and reverse-engineering malicious software, see the Honeynet Project Challenges page, which lists several challenges related to malware and forensics.
Mandiant’s free Redline tool is designed for “triaging hosts suspected of being compromised or infected while supporting in-depth live memory analysis.” The new utility is meant to replace Audit Viewer, which was Mandiant’s earlier memory analysis tool. Both programs rely on Memoryze for capturing the memory image of the live Windows host, though they can also examine “dead” memory image files.
Redline’s present version mostly mimics the functionality of Audit Viewer—albeit in a more streamlined interface. However, Mandiant plans to expand Redline “to include significant new capabilities unavailable in Audit Viewer,” according to the FAQ.
After launching Redline, the user is presented with several options, which include using an existing memory image file and examining the local computer:
When scanning a local system, the user can select what type of data Redline will extract from memory. The options are Quick Scan, Standard Scan, Full Audit and Custom. I recommend using Full Audit despite it being slow, in part because it includes the extraction of strings:
Once the scan completes, the user is presented with an interactive interface for examining the findings. One of the tool’s useful features is its ability to automatically assign a “Malware Risk Index” (MRI) to memory artifacts based on several built-in heuristics. Redline flips the rating in comparison to Audit Viewer, 100 as the riskiest value and 0 as the least risky one. (For more on MRI, see the Redline FAQ.)
Another feature of the tool allows it to locate and examine memory sections that may have been injected into processes:
Redline includes numerous other features that should prove useful to computer forensic analysts and incident responders. I’m looking forward to seeing this tool evolve.
I speak with a lot of security professionals who are seeking to enter or grow in the field of digital forensics and incident response. There is a lot of confusion regarding the work that such individuals do and what opportunities exist to join or excel within their ranks. This is worth discussing.
Computer and Digital Forensics
Digital forensics is often seen as a subset of computer forensics, which typically involves examining computers for signs of malicious or illegal actions. The actions might have been taken by the computer’s user who is being accused of a crime or by a person or malware that compromised the system. Historically, that meant examining the computer’s hard drive.
Today’s IT ecosystem encompasses more than computer systems or their hard drives, which is why digital forensics is seen as a broader practice of examining digital artifacts in databases, memory, network traffic and mobile devices. Sometimes it also involves e-discovery work to support litigation efforts.
Computer Security Incident Response
Incident response (IR) is often viewed as a subset of or a complement to digital forensics. After all, handling a suspected malware infection, system compromise or a data breach usually involves looking at digital artifacts to assess the situation. Performing IR may also include examining a live system by running commands on it to survey the affected host.
The IR process may also involve coordinating actions of the individuals and organizations involved in or affected by the incident—a task that involves strong communication skills, business savvy and a perspective on public relations.
When performing digital forensics and/or incident response, the examiner might come across malware in the form of browser scripts, exploit-ridden documents or malicious executables. Examining these artifacts to understand their capabilities requires a specialized malware analysis and reverse-engineering skill-set. (I teach a malware analysis course at SANS.)
The malware analyst (a.k.a. reverse engineer) needs to examine the malware specimen to determine how it its environment and what capabilities are built into its code base. The analyst may be asked to document the program’s malicious capabilities, understand its propagation characteristics, and define signatures for detecting its presence.
Digital Forensics and IR Career Options
As you can see, there are several disciplines within the overall field of digital forensics and incident response. As the result, there are many opportunities for specialization. The numerous options, while initially confusing, also make it easier for individuals to find an area that interests them, building upon their prior experiences while leaving room for professional growth.
One of the challenges for those looking to enter this field is that employers often focus their recruitment efforts on experienced forensicators, rather than investing into personnel who could mature as part of the group. Also, the relative youth of the field makes it hard for the beginners to identify mentors or to define a path for sharpening their skills. In some cases, individuals burn out from working hectic hours that leave little time for professional development.