OpenCourser brand icon
Sign in
We may earn an affiliate commission when you visit our partners.

x64dbg Script Programming For Reverse Engineering

Paul Chin, PhD

Welcome to "x64dbg Scripting For Reverse Engineering and Memory Hacking." This comprehensive course is designed for both new and experienced students in the field of ethical hacking and reverse engineering. Whether you've taken my previous x64dbg courses or are familiar with the basics of reverse engineering, this course will take your skills to the next level by diving deep into the power of xdbg scripting.

Read more

Welcome to "x64dbg Scripting For Reverse Engineering and Memory Hacking." This comprehensive course is designed for both new and experienced students in the field of ethical hacking and reverse engineering. Whether you've taken my previous x64dbg courses or are familiar with the basics of reverse engineering, this course will take your skills to the next level by diving deep into the power of xdbg scripting.

In this course, you'll learn how to automate reverse engineering tasks using x64dbg scripting, a critical skill for any serious security researcher. We'll start with the fundamentals of scripting within x64dbg and gradually move on to more advanced topics, such as modifying process memory to alter control flow logic, patching memory without the need to unpack executables, and test complex packing protections and anti-tampering mechanisms.

The course is structured to provide you with hands-on experience through real-world scenarios and challenges, allowing you to apply what you've learned in a practical, problem-solving context. If you're someone who enjoys solving puzzles and facing challenges head-on, this course will provide you with the tools and knowledge to do just that.

By the end of this course, you'll have the skills to automate and enhance your reverse engineering process, making you more efficient and effective in your work. Whether you're a security researcher looking to sharpen your skills or someone passionate about ethical hacking, this course is for you.

Join me in "x64dbg Scripting For Reverse Engineering and Memory Hacking" and start mastering the art of scripting in x64dbg today.

Sign up today and I will see you inside.

Enroll now

What's inside

Learning objectives

  • Script programming
  • Learn new reverse engineering techniques never taught before
  • Practice and hone your assembly language hacking skills
  • Hacking process memory
  • Patch memory of packed program without unpacking
  • Automating reverse engineering
  • Debugging packed program without unpacking
  • Evaluating, analyzing and testing upx, vmprotect, enigma and themida protectors
  • Automation using autoit
  • Using aob (array of bytes) scan to solve aslr (address space layout randomization) issues
  • Finding critical regions by tracing input strings in memory
  • Using handle proc algorithm to pause debugger to run scripts
  • Searching for critical regions even though there are no visible strings
  • Tracing input strings in the stack and parameter windows to find critical region
  • Enabling disabled buttons
  • Modifying message boxes with scripts and aob scans
  • Using instruction pointers eip and rip to modify flow control
  • Two step input validation and x64 calling conventions
  • Automatic rtu (run to user) to speed up search for critical regions
  • Creating keygens via scripting
  • Patching stack parameters on the fly
  • And more . . .
  • Show more
  • Show less

Syllabus

Introduction

Introduction and Welcome to the course.

Installing Oracle Virtual Box and Windows Virtual Machine, x64dbg and DiE (Detect It Easy)

Read more

Understand the need and benefits for script programming

The power of scripting to defeat ASLR protection

Installing Visual Studio Code, configuring syntax highlighting,  and Writing Your First Hello World Script

Create a Crackmes Project folder

Basic Introductory skills and knowledge

Where to go for help and documentation and also how to use the command input box in x64dbg

Where to go for help and documentation and how to use x64dbg's command input box

This tests your knowledge on x64dbg documentation and x64dbg Command Input Box

Use the MOV, INC, DEC, ADD, SUB and XOR instructions in a script

Use the MOV, INC, SUB, ADD, SUB and XOR instructions in a script

How to use scripting to push and pop

Use scripting to push registers to the stack and to pop the stack back to registers, understand significance of the order of pushes and pops.

Use TEST and do branching with JE and JNE instructions

Using the TEST instruction to do branching with JE and JNE instruction

Use the CMP instruction in scripting along with JE, JG and JL to branch

Use the CMP instruction in scripting along with JE, JG and JL to branch

Init and Attach

Init and attach to processes

Run and serun

run and serun and the differences between them

sto Stepover

Stepping over code using sto command

Stepping into calls

Simple single step into without parameters

Perform recursive Step Into

How to ignore exceptions when stepping over and stepping into

Ignore exceptions when stepping over or stepping into code

Use running to RET (rtr)

Run to RET for every function.

Skip Instructions

Skip instructions using the skip command

Undo Instructions

Undo instructions using the command InstrUndo

Understand User Modules and System Modules, Set BP on API calls and Run to User Code

Understand User Modules and System Modules, Set BP on API calls and Run to User Code

Step until system call and loops

Understand User and System Modules, Step Until System Call. Use Loops.

Understand how to dynamically get the Base Address and do Array of Bytes Scans

Get Base Address dynamically and do AOB scans.

Patching a Jump with NOPs using Memory Patching

AoB Scan followed by patching a jump with NOPs

Search For Strings On The Stack

Search For Strings On The Stack

Search For Strings In the Registers

Search For Strings In the Registers.  Useful for tracing where the password is about to be checked for either the correct length or to be compared with the real password.

Enable a disabled button by tracing breakpoint counts

Enable a disabled button by tracing breakpoint counts, then manually modify the SetWindowLong parameters.

Enable a disabled button by patching the SetWindowLong parameters using scripts

Enable a disabled button by patching the SetWindowLong parameters using scripts

Modify MessageBox by Tracing Memory Where Parameters Originally Load

Modify MessageBox by Tracing Memory Where Parameters Originally Load

Modify MessageBox With Scripts and AoB Scans

Modify MessageBox With Scripts and AoB Scans

Trace the Stack and Register for Input Strings (Username and Password)

Introduction to 04-Crypt0sBeginnerCrackMe - Preliminary Analysis

Trace Input Strings On The Stack and Understand The Differences Between ESP & EBP stack view

Tracing Password Input String On The Stack

Tracing Input Strings in Registers

Trace Input String on the Stack, Register and Patch Memory to to Reverse Jump Logic

Trace Input String on the Stack and Register

Patch Memory With NOPs to Reverse Jump Logic

Trace CMP Instructions

Trace CMP instructions for comparison between Input and Actual Password

Patch memory with NOP to reverse a jump logic

Handle-Proc algorithm and Two-in-one Patch for MessageBox and Reversing Jump

GUI App Basic Input String Search

Handle-Proc Algorithm for Searching For Critical Functions

Two-in-one Patch for MessageBox and Reversing Jump

Do Variations on the Handle-Proc Algorithm and Searching For UTF-16 Input Strings

Understand that the Handle-Proc algorithm can be varied and also that not all buttons can be used to pause the debugger

Using Various Different Proc To Pause The Debugger and Also How to Search For Unicode (UTF-16) Input Strings.

Patch memory to change the serial key to your own

Create Keygen From Results of Search For Input String

Introduction to Keygenthis and Handle-Proc algorithm

Handle-Proc Variations and Stack Search For Input String
Creating a Keygen From Results of Search For Input String
Create Keygen Where There Are No Error MessageBoxes

Create Keygen Where There Are No Error MessageBoxes

Use the Message Breakpoint Method, EBX Register Search and BP on GetDlgItemText

Preliminary Analysis of Trope crackme and trying Handle-Proc Algorithm on it

Use the Message Breakpoint Method to Locate the Critical Function

Fishing out the actual password by tracing the individual character comparison. Use the ZF flag to prevent early exit.

Set Breakpoint On GetDlgItemText API To Locate Critical Function

Understand  Concept Behind The Message Breakpoint Method For Finding The Critical Function

Deal with Dynamic Bytes in AOB and Modify EIP To Jump To Exit

Preliminary analysis on Patchme-Revamped

Avoiding Dynamic Bytes and Modifying EIP to Exit App

Identifying Dynamic Bytes In AOB and Replacing Them With Wildcard Placeholders

Understand Two-Step Input Validation and x64 Calling Conventions in Parameter Window

Understand the two-step input validation process

Understand x64 calling convention and parameter windows

Solve a UPX packed program - fishing for password and patching memory - without unpacking

Looking for a place to put bp in a packed program

Using VirtualProtect to locate unpacked region to insert breakpoint after input

Using scripts and AOB scans to set breakpoints in the unpacked region and to locate the Critical Region

Patching with NOPs to always show the Good Message

Solve UPX packed program - fishing for serial and patching - without unpacking

Save this course

Save x64dbg Script Programming For Reverse Engineering to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in x64dbg Script Programming For Reverse Engineering with these activities:
Review Assembly Language Fundamentals
Solidify your understanding of assembly language, which is crucial for reverse engineering and understanding how programs work at a low level.
Show steps
  • Study x86-64 assembly language syntax and instructions.
  • Practice writing simple assembly programs.
  • Debug assembly code using a debugger like GDB.
Read 'Reverse Engineering for Beginners'
Gain a solid foundation in reverse engineering principles before diving into x64dbg scripting.
View Alter Ego on Amazon
Show steps
  • Obtain a copy of 'Reverse Engineering for Beginners'.
  • Read the book, focusing on chapters related to debugging and disassembly.
  • Take notes on key concepts and techniques.
Practice x64dbg Scripting Basics
Reinforce your understanding of x64dbg scripting syntax and commands through repetitive exercises.
Show steps
  • Write scripts to perform basic operations like setting breakpoints and modifying registers.
  • Experiment with different x64dbg commands and observe their effects.
  • Debug your scripts to identify and fix errors.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Document Your Scripting Discoveries
Solidify your understanding by creating a personal knowledge base of x64dbg scripting techniques and solutions.
Show steps
  • Create a document or wiki to store your scripting notes.
  • Document each script you write, including its purpose, code, and any challenges you encountered.
  • Organize your notes for easy reference.
Automate a Simple Reverse Engineering Task
Apply your x64dbg scripting skills to automate a real-world reverse engineering task, such as patching a simple crackme.
Show steps
  • Choose a simple crackme or vulnerable program.
  • Identify a specific task to automate, such as bypassing a license check.
  • Write an x64dbg script to perform the task automatically.
  • Test your script thoroughly and refine it as needed.
Contribute to x64dbg Scripting Community
Share your knowledge and contribute to the x64dbg scripting community by creating and sharing scripts or documentation.
Show steps
  • Find an x64dbg scripting forum or repository.
  • Share your scripts and documentation with the community.
  • Provide feedback and assistance to other users.
Read 'Practical Reverse Engineering'
Expand your knowledge of reverse engineering techniques beyond the scope of the course.
View Practical Reverse Engineering on Amazon
Show steps
  • Obtain a copy of 'Practical Reverse Engineering'.
  • Read the book, focusing on chapters related to your areas of interest.
  • Experiment with the techniques described in the book.

Career center

Learners who complete x64dbg Script Programming For Reverse Engineering will develop knowledge and skills that may be useful to these careers:

Reading list

We've selected two books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in x64dbg Script Programming For Reverse Engineering.
Cover image
Practical Reverse Engineering
Save
Provides a comprehensive guide to reverse engineering, covering x86, x64, and ARM architectures, as well as the Windows kernel. It delves into reversing tools and obfuscation techniques, making it highly relevant for understanding the challenges and solutions presented in the x64dbg scripting course. It serves as a valuable reference for understanding the underlying principles and tools used in reverse engineering.
Practical Reverse Engineering
Paperback
Check
Practical Reverse Engineering
Kindle Edition
Check
Cover image
Alter Ego
Save
Provides a comprehensive introduction to reverse engineering concepts and techniques. It covers essential topics such as disassemblers, debuggers, and program analysis. It is particularly useful for beginners as it explains complex concepts in a clear and accessible manner. This book serves as a great reference for understanding the fundamentals before diving into x64dbg scripting.
Alter Ego
Paperback
Check
Alter Ego
Kindle Edition
Check

Share

Help others find this course page by sharing it with your friends and followers:
Facebook
LinkedIn
Reddit
X
Link
Email

Similar courses

Similar courses are unavailable at this time. Please try again later.
Effort
10 total hours
Via
Udemy
Instructor
Paul Chin, PhD
Language
English
Share and help others discover this course.
Facebook LinkedIn X Reddit Link Email
Don't miss out
Enroll today to gain access to this course
Enroll in this course
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2025 OpenCourser