OpenCourser brand icon
Sign in
We may earn an affiliate commission when you visit our partners.

x64dbg Script Programming For Reverse Engineering

Paul Chin, PhD

Welcome to "x64dbg Scripting For Reverse Engineering and Memory Hacking." This comprehensive course is designed for both new and experienced students in the field of ethical hacking and reverse engineering. Whether you've taken my previous x64dbg courses or are familiar with the basics of reverse engineering, this course will take your skills to the next level by diving deep into the power of xdbg scripting.

Read more

Welcome to "x64dbg Scripting For Reverse Engineering and Memory Hacking." This comprehensive course is designed for both new and experienced students in the field of ethical hacking and reverse engineering. Whether you've taken my previous x64dbg courses or are familiar with the basics of reverse engineering, this course will take your skills to the next level by diving deep into the power of xdbg scripting.

In this course, you'll learn how to automate reverse engineering tasks using x64dbg scripting, a critical skill for any serious security researcher. We'll start with the fundamentals of scripting within x64dbg and gradually move on to more advanced topics, such as modifying process memory to alter control flow logic, patching memory without the need to unpack executables, and test complex packing protections and anti-tampering mechanisms.

The course is structured to provide you with hands-on experience through real-world scenarios and challenges, allowing you to apply what you've learned in a practical, problem-solving context. If you're someone who enjoys solving puzzles and facing challenges head-on, this course will provide you with the tools and knowledge to do just that.

By the end of this course, you'll have the skills to automate and enhance your reverse engineering process, making you more efficient and effective in your work. Whether you're a security researcher looking to sharpen your skills or someone passionate about ethical hacking, this course is for you.

Join me in "x64dbg Scripting For Reverse Engineering and Memory Hacking" and start mastering the art of scripting in x64dbg today.

Sign up today and I will see you inside.

Enroll now

What's inside

Learning objectives

  • Script programming
  • Learn new reverse engineering techniques never taught before
  • Practice and hone your assembly language hacking skills
  • Hacking process memory
  • Patch memory of packed program without unpacking
  • Automating reverse engineering
  • Debugging packed program without unpacking
  • Evaluating, analyzing and testing upx, vmprotect, enigma and themida protectors
  • Automation using autoit
  • Using aob (array of bytes) scan to solve aslr (address space layout randomization) issues
  • Finding critical regions by tracing input strings in memory
  • Using handle proc algorithm to pause debugger to run scripts
  • Searching for critical regions even though there are no visible strings
  • Tracing input strings in the stack and parameter windows to find critical region
  • Enabling disabled buttons
  • Modifying message boxes with scripts and aob scans
  • Using instruction pointers eip and rip to modify flow control
  • Two step input validation and x64 calling conventions
  • Automatic rtu (run to user) to speed up search for critical regions
  • Creating keygens via scripting
  • Patching stack parameters on the fly
  • And more . . .
  • Show more
  • Show less

Syllabus

Introduction

Introduction and Welcome to the course.

Installing Oracle Virtual Box and Windows Virtual Machine, x64dbg and DiE (Detect It Easy)

Read more

Understand the need and benefits for script programming

The power of scripting to defeat ASLR protection

Installing Visual Studio Code, configuring syntax highlighting,  and Writing Your First Hello World Script

Create a Crackmes Project folder

Basic Introductory skills and knowledge

Where to go for help and documentation and also how to use the command input box in x64dbg

Where to go for help and documentation and how to use x64dbg's command input box

This tests your knowledge on x64dbg documentation and x64dbg Command Input Box

Use the MOV, INC, DEC, ADD, SUB and XOR instructions in a script

Use the MOV, INC, SUB, ADD, SUB and XOR instructions in a script

How to use scripting to push and pop

Use scripting to push registers to the stack and to pop the stack back to registers, understand significance of the order of pushes and pops.

Use TEST and do branching with JE and JNE instructions

Using the TEST instruction to do branching with JE and JNE instruction

Use the CMP instruction in scripting along with JE, JG and JL to branch

Use the CMP instruction in scripting along with JE, JG and JL to branch

Init and Attach

Init and attach to processes

Run and serun

run and serun and the differences between them

sto Stepover

Stepping over code using sto command

Stepping into calls

Simple single step into without parameters

Perform recursive Step Into

How to ignore exceptions when stepping over and stepping into

Ignore exceptions when stepping over or stepping into code

Use running to RET (rtr)

Run to RET for every function.

Skip Instructions

Skip instructions using the skip command

Undo Instructions

Undo instructions using the command InstrUndo

Understand User Modules and System Modules, Set BP on API calls and Run to User Code

Understand User Modules and System Modules, Set BP on API calls and Run to User Code

Step until system call and loops

Understand User and System Modules, Step Until System Call. Use Loops.

Understand how to dynamically get the Base Address and do Array of Bytes Scans

Get Base Address dynamically and do AOB scans.

Patching a Jump with NOPs using Memory Patching

AoB Scan followed by patching a jump with NOPs

Search For Strings On The Stack

Search For Strings On The Stack

Search For Strings In the Registers

Search For Strings In the Registers.  Useful for tracing where the password is about to be checked for either the correct length or to be compared with the real password.

Enable a disabled button by tracing breakpoint counts

Enable a disabled button by tracing breakpoint counts, then manually modify the SetWindowLong parameters.

Enable a disabled button by patching the SetWindowLong parameters using scripts

Enable a disabled button by patching the SetWindowLong parameters using scripts

Modify MessageBox by Tracing Memory Where Parameters Originally Load

Modify MessageBox by Tracing Memory Where Parameters Originally Load

Modify MessageBox With Scripts and AoB Scans

Modify MessageBox With Scripts and AoB Scans

Trace the Stack and Register for Input Strings (Username and Password)

Introduction to 04-Crypt0sBeginnerCrackMe - Preliminary Analysis

Trace Input Strings On The Stack and Understand The Differences Between ESP & EBP stack view

Tracing Password Input String On The Stack

Tracing Input Strings in Registers

Trace Input String on the Stack, Register and Patch Memory to to Reverse Jump Logic

Trace Input String on the Stack and Register

Patch Memory With NOPs to Reverse Jump Logic

Trace CMP Instructions

Trace CMP instructions for comparison between Input and Actual Password

Patch memory with NOP to reverse a jump logic

Handle-Proc algorithm and Two-in-one Patch for MessageBox and Reversing Jump

GUI App Basic Input String Search

Handle-Proc Algorithm for Searching For Critical Functions

Two-in-one Patch for MessageBox and Reversing Jump

Do Variations on the Handle-Proc Algorithm and Searching For UTF-16 Input Strings

Understand that the Handle-Proc algorithm can be varied and also that not all buttons can be used to pause the debugger

Using Various Different Proc To Pause The Debugger and Also How to Search For Unicode (UTF-16) Input Strings.

Patch memory to change the serial key to your own

Create Keygen From Results of Search For Input String

Introduction to Keygenthis and Handle-Proc algorithm

Handle-Proc Variations and Stack Search For Input String
Creating a Keygen From Results of Search For Input String
Create Keygen Where There Are No Error MessageBoxes

Create Keygen Where There Are No Error MessageBoxes

Use the Message Breakpoint Method, EBX Register Search and BP on GetDlgItemText

Preliminary Analysis of Trope crackme and trying Handle-Proc Algorithm on it

Use the Message Breakpoint Method to Locate the Critical Function

Fishing out the actual password by tracing the individual character comparison. Use the ZF flag to prevent early exit.

Set Breakpoint On GetDlgItemText API To Locate Critical Function

Understand  Concept Behind The Message Breakpoint Method For Finding The Critical Function

Deal with Dynamic Bytes in AOB and Modify EIP To Jump To Exit

Preliminary analysis on Patchme-Revamped

Avoiding Dynamic Bytes and Modifying EIP to Exit App

Identifying Dynamic Bytes In AOB and Replacing Them With Wildcard Placeholders

Understand Two-Step Input Validation and x64 Calling Conventions in Parameter Window

Understand the two-step input validation process

Understand x64 calling convention and parameter windows

Solve a UPX packed program - fishing for password and patching memory - without unpacking

Looking for a place to put bp in a packed program

Using VirtualProtect to locate unpacked region to insert breakpoint after input

Using scripts and AOB scans to set breakpoints in the unpacked region and to locate the Critical Region

Patching with NOPs to always show the Good Message

Solve UPX packed program - fishing for serial and patching - without unpacking

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Provides hands-on experience through real-world scenarios and challenges, allowing learners to apply what they've learned in a practical, problem-solving context
Teaches how to automate reverse engineering tasks using x64dbg scripting, a critical skill for any serious security researcher
Explores modifying process memory to alter control flow logic, patching memory without unpacking executables, and testing complex packing protections and anti-tampering mechanisms
Requires installing Oracle Virtual Box and Windows Virtual Machine, x64dbg and DiE (Detect It Easy), which may require additional setup and resources
Covers using AOB (Array of Bytes) scan to solve ASLR (Address Space Layout Randomization) issues, which is a common technique in reverse engineering
Teaches debugging packed programs without unpacking, which is a valuable skill for analyzing malware and protected software

Save this course

Save x64dbg Script Programming For Reverse Engineering to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in x64dbg Script Programming For Reverse Engineering with these activities:
Review Assembly Language Fundamentals
Solidify your understanding of assembly language, which is crucial for reverse engineering and understanding how programs work at a low level.
Show steps
  • Study x86-64 assembly language syntax and instructions.
  • Practice writing simple assembly programs.
  • Debug assembly code using a debugger like GDB.
Read 'Reverse Engineering for Beginners'
Gain a solid foundation in reverse engineering principles before diving into x64dbg scripting.
View Alter Ego: A Novel on Amazon
Show steps
  • Obtain a copy of 'Reverse Engineering for Beginners'.
  • Read the book, focusing on chapters related to debugging and disassembly.
  • Take notes on key concepts and techniques.
Practice x64dbg Scripting Basics
Reinforce your understanding of x64dbg scripting syntax and commands through repetitive exercises.
Show steps
  • Write scripts to perform basic operations like setting breakpoints and modifying registers.
  • Experiment with different x64dbg commands and observe their effects.
  • Debug your scripts to identify and fix errors.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Document Your Scripting Discoveries
Solidify your understanding by creating a personal knowledge base of x64dbg scripting techniques and solutions.
Show steps
  • Create a document or wiki to store your scripting notes.
  • Document each script you write, including its purpose, code, and any challenges you encountered.
  • Organize your notes for easy reference.
Automate a Simple Reverse Engineering Task
Apply your x64dbg scripting skills to automate a real-world reverse engineering task, such as patching a simple crackme.
Show steps
  • Choose a simple crackme or vulnerable program.
  • Identify a specific task to automate, such as bypassing a license check.
  • Write an x64dbg script to perform the task automatically.
  • Test your script thoroughly and refine it as needed.
Contribute to x64dbg Scripting Community
Share your knowledge and contribute to the x64dbg scripting community by creating and sharing scripts or documentation.
Show steps
  • Find an x64dbg scripting forum or repository.
  • Share your scripts and documentation with the community.
  • Provide feedback and assistance to other users.
Read 'Practical Reverse Engineering'
Expand your knowledge of reverse engineering techniques beyond the scope of the course.
View Practical Reverse Engineering on Amazon
Show steps
  • Obtain a copy of 'Practical Reverse Engineering'.
  • Read the book, focusing on chapters related to your areas of interest.
  • Experiment with the techniques described in the book.

Career center

Learners who complete x64dbg Script Programming For Reverse Engineering will develop knowledge and skills that may be useful to these careers:
Reverse Engineer
A reverse engineer analyzes software or hardware to understand its design, functionality, and operational principles. Reverse engineers often work in cybersecurity, software development, and vulnerability research. This course in x64dbg scripting helps a reverse engineer automate reverse engineering tasks, modify process memory to alter control flow logic, and patch memory without unpacking executables. The course will allow the reverse engineer to evaluate, analyze, and test complex packing protections and anti-tampering mechanisms.
See salaries and explore the career path for Reverse Engineer
Security Researcher
A security researcher investigates vulnerabilities and security flaws in software and systems. This course helps security researchers automate reverse engineering tasks using x64dbg scripting. The knowledge gained from this course regarding x64dbg, process memory modification and tracing input strings in memory helps a security researcher become more efficient and effective. The course's coverage of address space layout randomization, as well as debugging and memory patching, is relevant to a security researcher.
See salaries and explore the career path for Security Researcher
Malware Analyst
A malware analyst examines malicious software to understand its behavior, purpose, and potential impact. Malware analysts use reverse engineering techniques to dissect and analyze malware samples. This course in x64dbg scripting may be helpful to a malware analyst, because it covers automating reverse engineering tasks, modifying process memory, and patching memory. The course's focus on debugging packed programs without unpacking can be applied to malware analysis.
See salaries and explore the career path for Malware Analyst
Exploit Developer
An exploit developer creates code that takes advantage of vulnerabilities in software or systems to gain unauthorized access or execute malicious code. This course would be useful to an exploit developer because it covers x64dbg scripting, memory hacking, and assembly language. An exploit developer can use this course to learn techniques for automating reverse engineering tasks, modifying process memory, and bypassing security measures. The skills of automating and enhancing the reverse engineering process is useful to the exploit developer.
See salaries and explore the career path for Exploit Developer
Vulnerability Analyst
A vulnerability analyst identifies and assesses weaknesses in software, hardware, and network systems. Vulnerability analysts require a strong foundation in reverse engineering and debugging to discover and analyze vulnerabilities comprehensively. This course provides useful skills in automating reverse engineering tasks, modifying process memory, and patching memory. The focus on debugging packed programs without unpacking is applicable to vulnerability analysis.
See salaries and explore the career path for Vulnerability Analyst
Penetration Tester
A penetration tester assesses the security of computer systems, networks, or web applications by simulating attacks to identify vulnerabilities and weaknesses. Penetration testers use reverse engineering techniques to understand how software works and find potential entry points for attacks. This course may be useful for a penetration tester through learning new reverse engineering techniques. The course's lessons on memory patching can be valuable to a penetration tester.
See salaries and explore the career path for Penetration Tester
Cybersecurity Engineer
A cybersecurity engineer designs, implements, and manages security measures to protect computer systems, networks, and data from cyber threats. Cybersecurity engineers use knowledge of reverse engineering, debugging, and assembly language to understand and mitigate security risks. The knowledge from this course can help the cybersecurity engineer automate reverse engineering tasks and modify process memory to alter control flow logic. A cybersecurity engineer can also test complex packing protections and anti-tampering mechanisms.
See salaries and explore the career path for Cybersecurity Engineer
Software Developer
A software developer designs, codes, and tests software applications. While not always a primary skill, reverse engineering knowledge can be valuable for understanding legacy code, debugging complex issues, and ensuring software security. This course may be useful to the software developer as it covers automating reverse engineering tasks using x64dbg scripting. The course's focus on debugging packed programs without unpacking can be applied to software development.
See salaries and explore the career path for Software Developer
Game Hacker
A game hacker modifies video game code or data to gain an unfair advantage, create cheats, or bypass copy protection mechanisms. Game hackers use reverse engineering to understand how games work and identify areas to modify. This course introduces the game hacker to new reverse engineering techniques and automation using x64dbg scripting. The focus on hacking process memory and patching packed programs may be useful to the aspiring game hacker.
See salaries and explore the career path for Game Hacker
Firmware Engineer
A firmware engineer develops and tests low-level software that controls hardware devices. Reverse engineering is useful for analyzing and modifying firmware images. This course in x64dbg scripting may be useful to the firmware engineer, because it helps automate reverse engineering tasks, modify process memory, and patch memory. The course lessons regarding address space layout randomization and debugging and memory patching will be helpful to the firmware engineer.
See salaries and explore the career path for Firmware Engineer
Embedded Systems Engineer
An embedded systems engineer designs, develops, and tests software and hardware for embedded systems. Embedded systems engineers use reverse engineering techniques to analyze and debug existing systems. This course focusing on x64dbg, tracing input strings in memory, and modifying process memory benefits an embedded systems engineer. This enables them to evaluate, analyze, and test complex packing protections and anti-tampering mechanisms.
See salaries and explore the career path for Embedded Systems Engineer
Technical Support Engineer
A technical support engineer provides technical assistance and troubleshooting to customers or internal users. Reverse engineering skills can be valuable for diagnosing and resolving complex software issues. The lessons from this course that discuss new reverse engineering techniques may be useful to the technical support engineer. The course's focus on debugging packed programs without unpacking can be applied to software troubleshooting.
See salaries and explore the career path for Technical Support Engineer
Quality Assurance Tester
A quality assurance tester is responsible for identifying and reporting software defects and ensuring software quality. Reverse engineering skills can be valuable for understanding software behavior and identifying potential vulnerabilities. The quality assurance tester may find this course teaches valuable skills, as it helps automate reverse engineering tasks using x64dbg scripting. The course lessons of memory patching can be valuable to a quality assurance tester.
See salaries and explore the career path for Quality Assurance Tester
Data Recovery Specialist
A data recovery specialist retrieves lost or damaged data from storage devices. Reverse engineering might be used to understand file systems or data structures to aid in recovery efforts. This course may be useful to a data recovery specialist in that it teaches reverse engineering techniques and automation using x64dbg scripting. The focus on memory patching techniques can be applied to data recovery.
See salaries and explore the career path for Data Recovery Specialist
Computer Forensics Investigator
A computer forensics investigator examines digital evidence to uncover facts related to cybercrimes or security incidents. Reverse engineering techniques can be valuable for analyzing malware, tracing data flows, and understanding software behavior. This course focused on x64dbg scripting helps a computer forensics investigator automate reverse engineering tasks and modify process memory to alter control flow logic. The investigator will also be able to evaluate, analyze, and test complex packing protections and anti-tampering mechanisms.
See salaries and explore the career path for Computer Forensics Investigator

Reading list

We've selected two books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in x64dbg Script Programming For Reverse Engineering.
Cover image
Practical Reverse Engineering
Save
Provides a comprehensive guide to reverse engineering, covering x86, x64, and ARM architectures, as well as the Windows kernel. It delves into reversing tools and obfuscation techniques, making it highly relevant for understanding the challenges and solutions presented in the x64dbg scripting course. It serves as a valuable reference for understanding the underlying principles and tools used in reverse engineering.
Practical Reverse Engineering
Paperback
Check
Practical Reverse Engineering
Kindle Edition
Check
Cover image
Alter Ego
Save
Provides a comprehensive introduction to reverse engineering concepts and techniques. It covers essential topics such as disassemblers, debuggers, and program analysis. It is particularly useful for beginners as it explains complex concepts in a clear and accessible manner. This book serves as a great reference for understanding the fundamentals before diving into x64dbg scripting.
Alter Ego: A Novel
Paperback
$$

Share

Help others find this course page by sharing it with your friends and followers:
Facebook
LinkedIn
Reddit
X
Link
Email

Similar courses

Similar courses are unavailable at this time. Please try again later.
Effort
10 total hours
Via
Udemy
Instructor
Paul Chin, PhD
Language
English
Good to know
Provides hands-on experience through real-world scenarios and challenges, allowing learners to apply what they've learned in a practical, problem-solving context
Teaches how to automate reverse engineering tasks using x64dbg scripting, a critical skill for any serious security researcher
Explores modifying process memory to alter control flow logic, patching memory without unpacking executables, and testing complex packing protections and anti-tampering mechanisms
Requires installing Oracle Virtual Box and Windows Virtual Machine, x64dbg and DiE (Detect It Easy), which may require additional setup and resources
Covers using AOB (Array of Bytes) scan to solve ASLR (Address Space Layout Randomization) issues, which is a common technique in reverse engineering
Teaches debugging packed programs without unpacking, which is a valuable skill for analyzing malware and protected software
Share and help others discover this course.
Facebook LinkedIn X Reddit Link Email
Don't miss out
Enroll today to gain access to this course
Enroll in this course
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2025 OpenCourser