OpenCourser brand icon
Sign in
We may earn an affiliate commission when you visit our partners.
Course image
Udemy logo

Reverse Engineering & Malware Analysis - Intermediate Level

4.7 Filled star Filled star Filled star Filled star Half star
Based on 338 ratings
see reviews
Paul Chin, PhD

If you already have some basic reverse engineering and malware analysis knowledge and wish to go further, then this course is for you. I will take you from basic to intermediate level in reverse engineering and analyzing malware. You will learn using plenty of practical walk-throughs. The focus of this course will be on how to unpack malware. Most modern malware are packed in order to defeat analysis. Hence, this Intermediate Level Course provides the required knowledge and skills to unpack malware. All the needed tools will be introduced and explained. By the end of this course, you will have the intermediate level skill in malware analysis under your belt to further your studies in this field. Even if you do not intend to take up malware analysis as a career, still the knowledge and skills gained in reverse engineering and analysis would be beneficial to you to reverse software as well.

Read more

If you already have some basic reverse engineering and malware analysis knowledge and wish to go further, then this course is for you. I will take you from basic to intermediate level in reverse engineering and analyzing malware. You will learn using plenty of practical walk-throughs. The focus of this course will be on how to unpack malware. Most modern malware are packed in order to defeat analysis. Hence, this Intermediate Level Course provides the required knowledge and skills to unpack malware. All the needed tools will be introduced and explained. By the end of this course, you will have the intermediate level skill in malware analysis under your belt to further your studies in this field. Even if you do not intend to take up malware analysis as a career, still the knowledge and skills gained in reverse engineering and analysis would be beneficial to you to reverse software as well.

Everything is highly practical.  No boring theory or lectures. More like walk-throughs which you can replicate and follow along.  We will focus on API Hooking and Memory Analysis and Tracing to determine where and when to dump memory after a malware has unpacked its payload into memory. In this course, we will be using Oracle Virtual Machine installed with Flare-VM.  Take note that all software used in this course are free.

Topics include:

  1. Types of Malware and Terminologies

  2. Dynamic and Static Analysis

  3. Assembly Language Refresher and Malicious APIs

  4. API Hooking, Process Hijacking, Dumping Memory

  5. Fixing Section Alignments, Un-mapping and Re-Basing Dumped Files

  6. Enumerating Breakpoints and Memory Tracing

  7. Hooking VirtualProtect, VirtualAlloc, GetProcAddress, CreateProcessInternalW and other common API's

  8. Using Scylla Plugin to Dump Memory

  9. Using Delphi Interactive Reconstructor

  10. Dumping Memory from Memory Viewer, Process Hacker and Memory Maps

  11. API Enumeration Count Trick To Know When to Dump

  12. Self-Injection and Remote Thread Injection

  13. and more...

This course is suitable for:

  • Students who has already done a basic level malware analysis course

  • Hackers looking for additional tools and techniques to reverse software

  • Reverse Engineers who want to venture into malware analysis

The prerequisites:

  • Some basics in malware analysis or software reverse engineering.

  • Windows PC with Virtual Machine and Flare-VM Installed.

Note:

If you do not have the basics of malware analysis, it is recommended to take my earlier course first, which is entitled:

Reverse Engineering & Malware Analysis Fundamentals

Go ahead and enroll now. I will see you inside.

Enroll now

What's inside

Learning objectives

  • Types of malware and terminologies
  • Static analysis
  • Dynamic analysis
  • Assembly language refresher and malicious apis
  • Api hooking, process hijacking, dumping memory
  • Identifying standard and custom packers
  • Unpacking packed malware
  • Enumerating breakpoints and memory tracing
  • Hooking virtualprotect, virtualalloc, getprocaddress, createprocessinternalw and other common api's
  • Using scylla plugin to dump memory, fixing iat tables
  • Using delphi interactive reconstructor
  • Dumping memory from memory viewer, process hacker and memory maps
  • Api enumeration count trick to know when to dump
  • Self-injection and remote thread injection
  • Fixing section alignments, unmapping and re-basing dumped files
  • And more...
  • Show more
  • Show less

Syllabus

Static Analysis of .NET Trojan - Part 1

Introduction

Intro to the course.

Types of Malware and Malware Analysis Terminologies
Read more

Types of Malware

Malware Analysis Terminologies

Dynamic Analysis of .NET Trojan - Part 1

Static Analysis of .NET Trojan - Part 2

How to solve the problem when the malware hangs in a Running State when debugging with dnSpy

Assembly Language Basics

Malicious APIs

API Hooking, Process Hijacking and Dumping Memory

Tracing Process Hijacking and Dumping Memory

Fixing Section Alignment, Unmapping, fixing IAT and Re-basing

Unpacking Part 1: Static Analysis of Emotet Trojan

Unpacking Part 2: Debugging of Emotet Trojan to Hunt For Unpacked Code

Unpacking Part 3: Dumping Memory and Unmapping Dumped File

IDA Static Analysis and xdbg Enumerating Breakpoints

API Hooking and Memory Tracing

Dumping Memory and Unmapping File

API Hooking with VirtualProtect, VirtualAlloc and GetProcAddress

Memory Tracing and Scylla Dumping

PE-Studio and Interactive Delphi Reconstructor (IDR)

Unpacking part 1: API Hooking

Unpacking part 2: Dumping from Memory Map

Unpacking part 3: Un-mapping Dumped File

Dridex - part 1 - Initial Analysis

Dridex - part 2 - API Enumeration Count

Dridex - part 3 - Self-Injection and Process Hacker Dumping

Dridex - part 4 - Unmapping the Dumped File

Lab: Unpacking Ramnit Trojan

Ramnit - part 2 - Tracking VirtualAlloc to Identify When To Dump

Ramnit - part 3 - Unpacking UPX with CFF Explorer

Remcos - part 1 - exploring .NET with xdbg

Some students may experience the xdbg getting Terminated when they open Remcos trojan. This lesson shows how to solve that issue by setting xdbg to Ignore Exceptions.

Remcos - part 3 - Analysis with PE-Bear and PE-Studio

Remcos - part 4 - Unpacking with dnSpy by tracing Invoke

Zloader - part 1 - PE-Studio and API Hooking until VirtualProtect

Zloader - part 2 - Tracing Pointer to Unpacked Code for Dumping

Zloader - part 3 - PE-Studio and PE-Bear Analysis

Bonus Lecture

Traffic lights

Read about what's good
what should give you pause
and possible dealbreakers
Covers API Hooking and Memory Analysis and Tracing, which are essential for identifying where and when to dump memory after a malware has unpacked its payload
Uses Flare-VM, a popular and powerful Windows-based security distribution, providing a pre-configured environment for reverse engineering and malware analysis
Focuses on unpacking malware, a critical skill since most modern malware are packed to evade detection and analysis
Requires prior basic knowledge of malware analysis or software reverse engineering, indicating it builds upon existing skills
Recommends taking a fundamentals course first, suggesting that learners without prior experience may struggle with the intermediate content
Explores techniques like fixing section alignments, un-mapping, and re-basing dumped files, which are necessary for working with unpacked malware

Save this course

Create your own learning path. Save this course to your list so you can find it easily later.
Save

Reviews summary

Practical malware unpacking techniques

According to students, this course offers highly practical training focused squarely on unpacking malware. Many learners found the hands-on labs to be the strongest aspect, walking through techniques like API hooking and memory dumping on various malware families. The instructor is knowledgeable and the demos are effective, providing invaluable, real-world applicable skills. However, some students noted that the pace can be fast and the explanations occasionally feel rushed. A few reviewers emphasized that while labeled intermediate, the course requires a solid, perhaps more than 'basic', understanding of assembly and low-level concepts to fully keep up.
Requires solid prior knowledge of RE/assembly.
"Requires solid basics beforehand. Definitely intermediate level."
"It assumes a very strong grasp of assembly and low-level concepts, maybe more than just 'basic' RE knowledge."
"The 'basic' prerequisite feels misleading; you need to be quite comfortable with debuggers and assembly."
Delivers key malware unpacking skills effectively.
"Covers important unpacking techniques like API hooking and memory dumping."
"Fantastic course focusing on unpacking!"
"I appreciated the coverage of various packers and techniques like Scylla and IDR."
"Unpacking malware is a crucial skill, and this course delivers exactly that."
Hands-on labs are highly practical and valuable.
"The practical labs walking through unpacking different malware families were incredibly helpful."
"The labs are the strongest part and very practical."
"Top-notch practical training! Unpacking malware is a crucial skill, and this course delivers exactly that."
"Solid intermediate course. The hands-on labs are the main value proposition here."
Pace may be too fast for some learners.
"Some explanations felt a little fast-paced at times, especially if you're rusty on assembly."
"I struggled a bit with the pace."
"The instructor rushes through explanations. If you don't already know exactly what he's doing, you get lost quickly."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Reverse Engineering & Malware Analysis - Intermediate Level with these activities:
Review Assembly Language Fundamentals
Reinforce your understanding of assembly language, which is crucial for reverse engineering and understanding malware behavior at a low level.
Browse courses on Assembly Language
Show steps
  • Review basic assembly instructions and syntax.
  • Practice reading and interpreting simple assembly code snippets.
  • Familiarize yourself with common assembly idioms used in malware.
Read 'Reversing: Secrets of Reverse Engineering'
Expand your understanding of reverse engineering principles with a book that delves into disassembly, debugging, and code analysis.
View Reversing: Secrets of Reverse Engineering on Amazon
Show steps
  • Focus on chapters related to code analysis and debugging techniques.
  • Relate the book's concepts to the malware analysis techniques taught in the course.
  • Consider how the book's insights can improve your unpacking skills.
Read 'Practical Malware Analysis'
Supplement the course material with a comprehensive guide to malware analysis, covering both static and dynamic techniques.
View Practical Malware Analysis: The Hands-On Guide... on Amazon
Show steps
  • Read the chapters relevant to unpacking and memory analysis.
  • Work through the examples provided in the book.
  • Compare the book's techniques with those taught in the course.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Create a Personal Malware Analysis Toolkit
Improve your efficiency by compiling a collection of useful tools and resources for malware analysis, making them readily accessible for future projects.
Show steps
  • Gather all the tools used in the course, such as debuggers, disassemblers, and memory viewers.
  • Organize these tools into a single directory or virtual machine.
  • Add any additional tools or resources that you find helpful.
  • Document the purpose and usage of each tool in your toolkit.
Practice Unpacking with Publicly Available Malware Samples
Solidify your unpacking skills by practicing on real-world malware samples, reinforcing the techniques learned in the course.
Show steps
  • Download malware samples from reputable sources like VirusTotal or Hybrid-Analysis.
  • Apply the unpacking techniques taught in the course to these samples.
  • Document your process and findings for each sample.
  • Compare your results with online reports and analyses.
Write a Blog Post on a Specific Unpacking Technique
Deepen your understanding of a specific unpacking technique by explaining it in detail in a blog post, reinforcing your knowledge and improving your communication skills.
Show steps
  • Choose a specific unpacking technique covered in the course.
  • Research the technique thoroughly, including its variations and limitations.
  • Write a clear and concise blog post explaining the technique, including examples and diagrams.
  • Publish your blog post on a platform like Medium or your own website.
Develop a Simple Unpacker for a Specific Packer
Apply your knowledge by creating a tool that automates the unpacking process for a specific packer, demonstrating your mastery of the concepts.
Show steps
  • Choose a specific packer covered in the course or one you encounter in your practice.
  • Analyze the packer's structure and unpacking routine.
  • Write code to automate the unpacking process, using tools like Python or C++.
  • Test your unpacker on various samples packed with the chosen packer.
  • Document your unpacker's functionality and limitations.

Career center

Learners who complete Reverse Engineering & Malware Analysis - Intermediate Level will develop knowledge and skills that may be useful to these careers:
Malware Analyst
A malware analyst investigates and analyzes malicious software to understand its functionality, origin, and potential impact. This individual often performs both static and dynamic analysis on malware samples. The Reverse Engineering & Malware Analysis course helps build the practical skills to unpack malware, a crucial ability, as modern malware is often packed to evade detection. This course, with its practical walk-throughs on API Hooking and Memory Analysis, may be particularly useful for those looking to dissect and understand complex malware threats. This course can provide a solid foundation in unpacking malware.
See salaries and explore the career path for Malware Analyst
Digital Forensics Analyst
Digital forensics analysts investigate cybercrimes and security incidents, collecting and analyzing digital evidence to identify perpetrators and understand the scope of the damage. Knowledge of malware analysis is crucial for examining infected systems and understanding the tools and techniques used by attackers. The Reverse Engineering & Malware Analysis course may help learners develop the skills to dissect malware, trace its activity, and extract relevant forensic artifacts. The skills taught related to memory analysis may be useful.
See salaries and explore the career path for Digital Forensics Analyst
Reverse Engineer
A reverse engineer analyzes software or hardware to understand its design, functionality, and operation, often without access to the source code or original documentation. The Reverse Engineering & Malware Analysis course equips you with the expertise to dissect and comprehend software, even when obfuscated. The course's focus on assembly language, API hooking, and memory analysis may be especially helpful. The reverse engineer often will use the techniques taught in a course such as this one to conduct their work. This course may be useful.
See salaries and explore the career path for Reverse Engineer
Intelligence Analyst
Intelligence analysts gather and analyze information related to cyber threats and actors, providing insights to inform decision-making and proactive security measures. Malware analysis is a key component of understanding the capabilities and tactics of threat actors. The skills taught in the Reverse Engineering & Malware Analysis course regarding the identification of malware types, unpacking packed malware samples, and analyzing malicious APIs can help intelligence analysts track and understand emerging threats. This may be a useful course.
See salaries and explore the career path for Intelligence Analyst
Security Researcher
Security researchers investigate vulnerabilities in software and systems to improve overall security posture. They often analyze malware to understand attack vectors and develop defenses. A relevant course such as Reverse Engineering & Malware Analysis helps develop crucial skills in dynamic and static analysis, unpacking malware, and understanding malicious APIs. The course's practical labs on unpacking various trojans, such as Emotet and Hancitor, can provide this person with valuable real-world experience. The security researcher performs reverse engineering as part of their job.
See salaries and explore the career path for Security Researcher
Penetration Tester
A penetration tester, or ethical hacker, simulates cyberattacks to identify vulnerabilities in a system's security. Understanding malware and reverse engineering techniques aids in mimicking attacker behavior and discovering potential exploits. The Reverse Engineering & Malware Analysis course helps learn how malware operates and how to bypass security measures, thus enhancing a penetration tester's skills in identifying and exploiting weaknesses. This course's teachings on memory tracing and API hooking may be helpful in uncovering vulnerabilities.
See salaries and explore the career path for Penetration Tester
Security Architect
Security architects design and build secure computer systems and networks, ensuring that security measures are integrated throughout the infrastructure. A strong understanding of malware analysis can help security architects design more robust and resilient systems. The Reverse Engineering & Malware Analysis course may help security architects understand the tactics that malware uses to compromise systems, which can inform their design decisions and security implementations. This may be a useful course.
See salaries and explore the career path for Security Architect
Incident Responder
An incident responder handles security breaches and cyberattacks, working to contain the incident, investigate its cause, and restore systems to normal operation. Knowledge of malware analysis is invaluable for understanding the nature of an attack and developing effective remediation strategies. The Reverse Engineering & Malware Analysis course helps understand malware behavior, how it spreads, and its potential impact on systems. Understanding malicious APIs, as taught in this course, may be useful in rapidly identifying attack vectors.
See salaries and explore the career path for Incident Responder
Vulnerability Analyst
Vulnerability analysts identify and assess weaknesses in software, hardware, and networks. They work to prevent security breaches and improve system resilience. A course like Reverse Engineering & Malware Analysis, with its focus on unpacking and dissecting malware, can help vulnerability analysts understand how exploits work and identify potential vulnerabilities in software. The course's coverage of assembly language and memory analysis may be especially helpful in identifying low-level vulnerabilities.
See salaries and explore the career path for Vulnerability Analyst
Exploit Developer
An exploit developer creates code that takes advantage of vulnerabilities in software or systems to gain unauthorized access. This role often requires extensive knowledge of reverse engineering and malware analysis to understand how to target vulnerabilities effectively. While this course does not explicitly teach exploit development, the Reverse Engineering & Malware Analysis course may help to dissect and understand software weaknesses, which is a prerequisite for developing exploits. Knowledge gained from the course may prove useful.
See salaries and explore the career path for Exploit Developer
Cybersecurity Engineer
Cybersecurity engineers design, implement, and manage security systems and infrastructure to protect organizations from cyber threats. While the role is broad, a solid understanding of malware analysis helps in creating more robust defenses. The Reverse Engineering & Malware Analysis course provides a deeper understanding of malware behavior and techniques, which can inform the design and implementation of more effective security measures. The knowledge of dynamic and static analysis may prove helpful..
See salaries and explore the career path for Cybersecurity Engineer
Software Developer
A software developer designs, codes, and tests software applications. While not always directly related, understanding reverse engineering and malware analysis can help developers write more secure code and prevent vulnerabilities. The Reverse Engineering & Malware Analysis course exposes learners to common malware techniques and vulnerabilities which can inform better coding practices and security awareness. Knowledge of assembly language concepts discussed in the course may be useful.
See salaries and explore the career path for Software Developer
Security Consultant
A security consultant advises organizations on how to improve their security posture, assess risks, and implement security solutions. Understanding malware and reverse engineering provides a deeper perspective on potential threats and vulnerabilities. A course in Reverse Engineering & Malware Analysis helps by providing the consultant with a stronger technical understanding of malware, which can inform their recommendations and strategies. A consultant who understands reverse engineering may find this course beneficial.
See salaries and explore the career path for Security Consultant
Cryptographer
Cryptographers design and analyze encryption algorithms and security protocols to protect sensitive information. While not directly related to malware analysis, understanding reverse engineering techniques can help cryptographers assess the security of existing systems and identify potential weaknesses. The Reverse Engineering & Malware Analysis course may help cryptographers understand how attackers might attempt to bypass or compromise cryptographic protections. This may be a useful course.
See salaries and explore the career path for Cryptographer
System Administrator
System administrators are responsible for maintaining and managing computer systems and networks. While they typically do not analyze malware directly, having a basic understanding of malware analysis can help them better protect their systems and respond to security incidents. The Reverse Engineering & Malware Analysis course may help system administrators understand the different types of malware, how they spread, and what actions they can take to prevent infections. This may be a helpful course.
See salaries and explore the career path for System Administrator

Reading list

We've selected two books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Reverse Engineering & Malware Analysis - Intermediate Level.
Cover image
Practical Malware Analysis
Save
Comprehensive guide to malware analysis techniques. It covers both static and dynamic analysis, providing practical examples and step-by-step instructions. It is highly recommended as a reference text for this course, as it provides a solid foundation for understanding the concepts and techniques covered in the lectures. This book is commonly used as a textbook at academic institutions and by industry professionals.
Practical Malware Analysis: The Hands-On Guide to...
Paperback
$$
Practical Malware Analysis: The Hands-On Guide to...
Kindle Edition
$$
Cover image
Reversing
Save
Provides a deep dive into the world of reverse engineering. It covers a wide range of topics, including disassembly, debugging, and code analysis. While some parts may be more advanced, it offers valuable insights into the underlying principles of reverse engineering, which are essential for malware analysis. This book is more valuable as additional reading than it is as a current reference.
Reversing: Secrets of Reverse Engineering
Paperback
$$
Reversing: Secrets of Reverse Engineering
Kindle Edition
$$

Share

Help others find this course page by sharing it with your friends and followers:
Facebook
LinkedIn
Reddit
X
Link
Email

Similar courses

Similar courses are unavailable at this time. Please try again later.
Course image
Rating
4.7 Filled star Filled star Filled star Filled star Half star
Effort
5.5 total hours
Via
Udemy
Instructor
Paul Chin, PhD
Language
English
Traffic lights
Read about what's good
what should give you pause
and possible dealbreakers
Covers API Hooking and Memory Analysis and Tracing, which are essential for identifying where and when to dump memory after a malware has unpacked its payload
Uses Flare-VM, a popular and powerful Windows-based security distribution, providing a pre-configured environment for reverse engineering and malware analysis
Focuses on unpacking malware, a critical skill since most modern malware are packed to evade detection and analysis
Requires prior basic knowledge of malware analysis or software reverse engineering, indicating it builds upon existing skills
Recommends taking a fundamentals course first, suggesting that learners without prior experience may struggle with the intermediate content
Explores techniques like fixing section alignments, un-mapping, and re-basing dumped files, which are necessary for working with unpacked malware
Share this
Share to help others discover this course.
Facebook LinkedIn X Reddit Link Email
Begin learning today
Enroll now to gain full access to Reverse Engineering & Malware Analysis - Intermediate Level.
Enroll now
Save for later
Add this course to your list. Find it anytime.
Save
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2025 OpenCourser