x64 Assembly Language and Reverse Engineering Practicals

If you've been using debuggers to reverse-engineer programs and wish you had a better understanding of Assembly Language, or if you're just looking to learn Assembly Language in a fun and exciting way, then this course is for you. Embark on a journey to master the intricacies of x64 assembly language with this specialized course, designed for those aiming to elevate their reverse engineering skills. This is the second course in this series after the first one: Assembly Language Programming for Reverse Engineering. In that earlier course the focus was on x86 basics where we covered 32-bit assembly. In this course, we continue the journey by learning x64 (64-bit assembly). In the second half of this course, we will apply all the knowledge from the first course, as well as new insights from this course, to hands-on reverse-engineering exercises with executable binaries. Taking the first course is preferable but not mandatory. You could just jump straight into this x64 course and learn the differences with x86 as you go along.

While traditional assembly language courses focus on writing code from scratch using assemblers like NASM or FASM, they often leave a knowledge gap when it comes to applying that expertise to real-world reverse engineering. This course is tailored to fill this gap, providing you with the knowledge to modify and extend the functionality of existing 32-bit and 64-bit applications.

Utilizing the powerful x64dbg debugger, we'll bypass the traditional assemblers to teach you assembly language in the context it's most used in the field: directly within the debugging environment. This is also known as the hacker's perspective to assembly language. This hands-on approach ensures that you learn by doing, which is critical for effectively reverse engineering and manipulating software.

This black art is not widely taught and there are no existing courses elsewhere that put together coherently all the knowledge of assembly and reverse engineering in one place. More often than not, courses on Assembly and Reverse Engineering are offered separately. Many courses on Reverse Engineering don't cover the background assembly language from a software hacker's perspective. Moreover, almost all Assembly Language courses focus on writing programs from scratch rather than modifying existing programs to add new functionality with your own code using a debugger like x64dbg.

In this course, you will learn to:

- Navigate and utilize the x64dbg debugger
- Analyze and alter x64 executable files to inject custom code.
- Harness data, executable and memory segments to expand program capabilities.
- Develop new functions within existing applications for added functionality.
- Understand the x64 Microsoft Calling conventions and Stack Frames.
- Directly manipulate memory data segment
- Bypass string encryption
- Deep tracing to retrieve data and passwords
- Use Python to patch process memory
- Modify packed programs without unpacking
and more . . .

Features of this course:

This course is oriented towards practical applications
No lengthy, dull theoretical lectures
First half of this course: Learn x64 Assembly Language through the x64dbg debugger
Second half of this course: Practice reversing, tracing, extracting data, memory hacking, and modifying executable binaries

Learning Objectives:

By the end of this course, you will have a thorough understanding of x64 assembly language from a reverse engineer's perspective, a skill set that is rare and highly sought after in fields like cybersecurity, malware analysis, and software development.

Whether you're a security researcher, a malware analyst, a student of software security, or a programmer looking to deepen your understanding of software internals, this course is your stepping stone to becoming proficient in the 'black art' of assembly language and reverse engineering.

Enroll now to gain this competitive edge and take your skills to the next level. Let's unravel the complexities of x64 together. I look forward to guiding you through every step of this exciting journey. See you inside.

Enroll now

Or start a personal plan

And upskill your team with Udemy

Here's a deal for you

We found an offer that may be relevant to this course.

Save money when you learn. All coupon codes, vouchers, and discounts are applied automatically unless otherwise noted.

Use code at checkout. Valid until January 1, 2025

New customers offer

Visit Udemy from OpenCourser as a new customer and claim these savings on your first order.

What's inside

Learning objectives

X64 (64-bit) assembly language
Reverse engineering
X64dbg debugging
Modifying programs
Injecting code into 64-bit exe files
Hollowing out 64-bit exe files
64-bit registers
64-bit memory read and write access
X64 calling conventions
Creating x64 functions
Password phishing without strings
Creating keygens
Reversing program code logic
Trace highlighting and animation
Stack manipulation

Comment tracing debug technique
Hooking winapi debug technique
File patching
Enabling disabled buttons
Removing nag screens
Deep tracing to phish out passwords
Loop tracing techniques
Defeating anti-debugger protection
Reversing binary without strings
Using python to write loaders and memory patcher
Reversing software protected binary without unpacking
And more . . .
Show more
Show less

X64 (64-bit) assembly language
Reverse engineering
X64dbg debugging
Modifying programs
Injecting code into 64-bit exe files
Hollowing out 64-bit exe files
64-bit registers
64-bit memory read and write access
X64 calling conventions
Creating x64 functions
Password phishing without strings
Creating keygens
Reversing program code logic
Trace highlighting and animation
Stack manipulation
Comment tracing debug technique
Hooking winapi debug technique
File patching
Enabling disabled buttons
Removing nag screens
Deep tracing to phish out passwords
Loop tracing techniques
Defeating anti-debugger protection
Reversing binary without strings
Using python to write loaders and memory patcher
Reversing software protected binary without unpacking
And more . . .
Show more
Show less

Syllabus

Introduction

Welcome and introduction to the course

Installing Virtual Machine and x64dbg

The importance of using virtual machines

What is the x64 64-bit architecture

Why using a debugger to learn assembly language rather than common assemblers like NASM, FASM or MASM

Intel CPU compatibility mode (32-bit) and 64-bit mode, addressing 64-bit registers and parts thereof

Learn the basic MOV instructions

Write simple move instructions.

This is to test your understanding and ability to write your first assembly instructions from within x64dbg.

How to read from and write to memory

How write to memory and read from it.

Test your understanding of initialized data and uninitialized data segments of memory

How to create strings in memory and access it.

Using the BSS segment to create variables

Learn how to use the ADD instructions

Using XOR to zero out a register

Learn the ADD instructions

Learn how ADD Instructions work in assembly

MOV using partial registers

Create a new hollowed out template for use in this course.

MOV using partial registers

How to access parts of a register

Basic PUSH and POP instructions

Basic PUSH and POP

An exercise on PUSH and POP

PUSH and POP exercise

More MOV Instructions

Intro to MOV Instructions

Practical on MOV

Learn the XCHG Instructions

XCHG Instructions

XCHG Memory

How to use the INC, DEC, NEG, ADD and SUB instructions

Learn how to use the INC and DEC on registers and memory

Learn how to use the NEG (Negate) Instruction

How to use ADD and SUB instructions

Learn about the CF, OF, SF and ZF flags

A quick conceptual introduction to the Carry Flag (CF), Overflow Flag (OF), Sign Flag (SF), and Zero Flag (ZF) in x64 assembly programming

How to interpret the CF flag

How the OF flag works with signed operations.

How to interpret and use the SF flag register.

Learn how the ZF flag works.

Understand what are Bitwise Operations and be able to use them

Understand bitwise logical operations

Understand how AND Operations works and how to use them

Understand how OR Operations work and how to use them

Understand how XOR operation works and the significance of XOR

Learn how the Logical Operations affect the SF and ZF registers

Practical on NOT Operation

How branching is implemented using jumps and the types of jumps available

Introduction to the two types of jumps: Unconditional and Conditional Jumps

Learn how JMP works and its 3 forms, address, register and memory.

How the TEST instruction works and its significance in testing RAX values after function calls.

How the CMP instruction how and its use in testing for register values.

Learn about the most common conditional jumps in x64dbg, i.e. JE/JNE and JB/JAE and their significance.

Understanding how CMP is used to compare signed numbers and the use of JG, JL and their counterparts JGE and JLE

Learn how to implement the IF structure in assembly using TEST, CMP together with JE and JNE instructions

Learn how to implement IF-ELSE statements in assembly by using CMP and JG

How to implement multiple IF tests within assembly.

How to implement WHILE loops in assembly inside x64dbg

How to implement DO-WHILE loops in assembly inside x64dbg.

How array works inside x64dbg

Introduction to how to insert an array directly into memory and access it.

How to loop through an array use base address and offsets.

Learn how LEA is used to calculate effective addresses and also how to use it to do simple arithmetic.

Learn how multiplication and division can be implemented inside x64dbg

Learn how to implement MUL in x64dbg

Learn how DIV instructions are implemented inside x64dbg

Learn how to implement IMUL inside x64dbg

Using stack frames, CALL and RET to create functions

How to implement PUSH and POP inside x64dbg

What are stackframes and how to use them to create functions

Understanding Microsoft x64 Calling Conventions

A practical on how to implement a 4 arguments function call with no local variables

A practical hands-on on how to implement a 4 arguments function call with with 4 local variables

A hands-on on how to implement a 5 argument function call

Learn how to implement a Function Call with 6 arguments

Practical on implementing a 7 argument function call.

Apply the knowledge and skills in practical reverse engineering projects

Introduction to the practicals and some tips as well as installing DiE

Preliminary basic skills including how to check the exe file type

Learn how to reverse engineer an exe and use String Search method to phish for passwords

Use the String Search method to phish for passwords.

How to reverse a jump and patch the file

How to reverse a jump and do a simple surface patch.

How to use the Comment Tracing method to analyze register values and control flow logic

How to directly access memory to modify password and patch the exe file

How to enable a greyed out button, remove nag message box, accept any serial key and modify message box

How to enable a greyed-out disable Register button

How to remove a Nag Message Box

Patching to accept any serial key and also to modify the message box

Recursively trace into inner function calls to phish out username and password

Deep trace into inner function calls to identify username

Do deep tracing into inner function calls to uncover the password

Step into function calls to trace EAX values that is being returned by the function

Understand the significance of the EAX register in relation to function call returns and step into function calls to trace it.

Identify the correct jump to patch in order to reverse the logic so that the right message is shown

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Covers reverse engineering, a skill set that is highly sought after in fields like cybersecurity, malware analysis, and software development

Taught by Paul Chin, who is an experienced instructor in Assembly Language Programming for Reverse Engineering and a skilled practitioner in Assembly Language

Designed for those aiming to elevate their reverse engineering skills, who have an interest in Assembly Language, and who are reverse-engineering programs

Involves hands-on reverse-engineering exercises with executable binaries, applying knowledge of Assembly Language and techniques from the first course in this series

Uses practical applications and avoids lengthy, dull theoretical lectures, making the learning process more engaging and applicable

Uses the powerful x64dbg debugger to teach assembly language in the context it's most used in the field for reverse engineering, providing a hacker's perspective

Save this course

Save x64 Assembly Language and Reverse Engineering Practicals to your list so you can find it easily later:

Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in x64 Assembly Language and Reverse Engineering Practicals with these activities:

Review x64dbg debugger

Show steps

Using x64dbg debugger is a key component of this course. Doing this refresher will help you get familiar with the tool and its features.

Show steps

Download and install x64dbg on your computer
Open x64dbg and familiarize yourself with the interface
Load a simple executable file into x64dbg and practice debugging it

Follow online tutorials on x64 Assembly Language

Show steps

There are many tutorials available online that can help you learn assembly language. Following a tutorial can deepen your understanding of some concepts.

Browse courses on Assembly Language

Show steps