We may earn an affiliate commission when you visit our partners.
Course image
Paul Chin

If you've been using debuggers to reverse-engineer programs and wish you had a better understanding of Assembly Language, or if you're just looking to learn Assembly Language in a fun and exciting way, then this course is for you. Embark on a journey to master the intricacies of x64 assembly language with this specialized course, designed for those aiming to elevate their reverse engineering skills. This is the second course in this series after the first one: Assembly Language Programming for Reverse Engineering. In that earlier course the focus was on x86 basics where we covered 32-bit assembly. In this course, we continue the journey by learning x64 (64-bit assembly).  In the second half of this course, we will apply all the knowledge from the first course, as well as new insights from this course, to hands-on reverse-engineering exercises with executable binaries. Taking the first course is preferable but not mandatory. You could just jump straight into this x64 course and learn the differences with x86 as you go along.

Read more

If you've been using debuggers to reverse-engineer programs and wish you had a better understanding of Assembly Language, or if you're just looking to learn Assembly Language in a fun and exciting way, then this course is for you. Embark on a journey to master the intricacies of x64 assembly language with this specialized course, designed for those aiming to elevate their reverse engineering skills. This is the second course in this series after the first one: Assembly Language Programming for Reverse Engineering. In that earlier course the focus was on x86 basics where we covered 32-bit assembly. In this course, we continue the journey by learning x64 (64-bit assembly).  In the second half of this course, we will apply all the knowledge from the first course, as well as new insights from this course, to hands-on reverse-engineering exercises with executable binaries. Taking the first course is preferable but not mandatory. You could just jump straight into this x64 course and learn the differences with x86 as you go along.

While traditional assembly language courses focus on writing code from scratch using assemblers like NASM or FASM, they often leave a knowledge gap when it comes to applying that expertise to real-world reverse engineering. This course is tailored to fill this gap, providing you with the knowledge to modify and extend the functionality of existing  32-bit and 64-bit applications.

Utilizing the powerful x64dbg debugger, we'll bypass the traditional assemblers to teach you assembly language in the context it's most used in the field: directly within the debugging environment. This is also known as the hacker's perspective to assembly language. This hands-on approach ensures that you learn by doing, which is critical for effectively reverse engineering and manipulating software.

This black art is not widely taught and there are no existing courses elsewhere that put together coherently all the knowledge of assembly and reverse engineering in one place. More often than not, courses on Assembly and Reverse Engineering are offered separately. Many courses on Reverse Engineering don't cover the background assembly language from a software hacker's perspective. Moreover, almost all Assembly Language courses focus on writing programs from scratch rather than modifying existing programs to add new functionality with your own code using a debugger like x64dbg.

In this course, you will learn to:

  1. - Navigate and utilize the x64dbg debugger

  2. - Analyze and alter x64 executable files to inject custom code.

  3. - Harness data, executable and memory segments to expand program capabilities.

  4. - Develop new functions within existing applications for added functionality.

  5. - Understand the x64 Microsoft Calling conventions and Stack Frames.

  6. - Directly manipulate memory data segment

  7. - Bypass string encryption

  8. - Deep tracing to retrieve data and passwords

  9. - Use Python to patch process memory

  10. - Modify packed programs without unpacking

  11. and more . . .

Features of this course:

  1. This course is oriented towards practical applications

  2. No lengthy, dull theoretical lectures

  3. First half of this course: Learn x64 Assembly Language through the x64dbg debugger

  4. Second half of this course: Practice reversing, tracing, extracting data, memory hacking, and modifying executable binaries

Learning Objectives:

By the end of this course, you will have a thorough understanding of x64 assembly language from a reverse engineer's perspective, a skill set that is rare and highly sought after in fields like cybersecurity, malware analysis, and software development.

Whether you're a security researcher, a malware analyst, a student of software security, or a programmer looking to deepen your understanding of software internals, this course is your stepping stone to becoming proficient in the 'black art' of assembly language and reverse engineering.

Enroll now to gain this competitive edge and take your skills to the next level. Let's unravel the complexities of x64 together. I look forward to guiding you through every step of this exciting journey. See you inside.

Enroll now

What's inside

Learning objectives

  • X64 (64-bit) assembly language
  • Reverse engineering
  • X64dbg debugging
  • Modifying programs
  • Injecting code into 64-bit exe files
  • Hollowing out 64-bit exe files
  • 64-bit registers
  • 64-bit memory read and write access
  • X64 calling conventions
  • Creating x64 functions
  • Password phishing without strings
  • Creating keygens
  • Reversing program code logic
  • Trace highlighting and animation
  • Stack manipulation
  • Comment tracing debug technique
  • Hooking winapi debug technique
  • File patching
  • Enabling disabled buttons
  • Removing nag screens
  • Deep tracing to phish out passwords
  • Loop tracing techniques
  • Defeating anti-debugger protection
  • Reversing binary without strings
  • Using python to write loaders and memory patcher
  • Reversing software protected binary without unpacking
  • And more . . .
  • Show more
  • Show less

Syllabus

Introduction

Welcome and introduction to the course

Installing Virtual Machine and x64dbg

The importance of using virtual machines

Read more
What is the x64 64-bit architecture

Why using a debugger to learn assembly language rather than common assemblers like NASM, FASM or MASM

Intel CPU compatibility mode (32-bit) and 64-bit mode, addressing 64-bit registers and parts thereof

Learn the basic MOV instructions

Write simple move instructions.

This is to test your understanding and ability to write your first assembly instructions from within x64dbg.

How to read from and write to memory

How write to memory and read from it.

Test your understanding of initialized data and uninitialized data segments of memory

How to create strings in memory and access it.

Using the BSS segment to create variables

Learn how to use the ADD instructions

Using XOR to zero out a register

Learn the ADD instructions

Learn how ADD Instructions work in assembly

MOV using partial registers

Create a new hollowed out template for use in this course.

MOV using partial registers

How to access parts of a register

Basic PUSH and POP instructions

Basic PUSH and POP

An exercise on PUSH and POP

PUSH and POP exercise

More MOV Instructions
Intro to MOV Instructions

Practical on MOV

Learn the XCHG Instructions

XCHG Instructions

XCHG Memory

How to use the INC, DEC, NEG, ADD and SUB instructions

Learn how to use the INC and DEC on registers and memory

Learn how to use the NEG (Negate) Instruction

How to use ADD and SUB instructions

Learn about the CF, OF, SF and ZF flags

A quick conceptual introduction to the Carry Flag (CF), Overflow Flag (OF), Sign Flag (SF), and Zero Flag (ZF) in x64 assembly programming

How to interpret the CF flag

How the OF flag works with signed operations.

How to interpret and use the SF flag register.

Learn how the ZF flag works.

Understand what are Bitwise Operations and be able to use them

Understand bitwise logical operations

Understand how AND Operations works and how to use them

Understand how OR Operations work and how to use them

Understand how XOR operation works and the significance of XOR

Learn how the Logical Operations affect the SF and ZF registers

Practical on NOT Operation

How branching is implemented using jumps and the types of jumps available

Introduction to the two types of jumps: Unconditional and Conditional Jumps

Learn how JMP works and its 3 forms, address, register and memory.

How the TEST instruction works and its significance in testing RAX values after function calls.

How the CMP instruction how and its use in testing for register values.

Learn about the most common conditional jumps in x64dbg, i.e. JE/JNE and JB/JAE and their significance.

Understanding how CMP is used to compare signed numbers and the use of JG, JL and their counterparts JGE and JLE

Learn how to implement the IF structure in assembly using TEST, CMP  together with JE and JNE instructions

Learn how to implement IF-ELSE statements in assembly by using CMP and JG

How to implement multiple IF tests within assembly.

How to implement WHILE loops in assembly inside x64dbg

How to implement DO-WHILE loops in assembly inside x64dbg.

How array works inside x64dbg

Introduction to how to insert an array directly into memory and access it.

How to loop through an array use base address and offsets.

Learn how LEA is used to calculate effective addresses and also how to use it to do simple arithmetic.

Learn how multiplication and division can be implemented inside x64dbg

Learn how to implement MUL in x64dbg

Learn how DIV instructions are implemented inside x64dbg

Learn how to implement IMUL inside x64dbg

Using stack frames, CALL and RET to create functions

How to implement PUSH and POP inside x64dbg

What are stackframes and how to use them to create functions

Understanding Microsoft x64 Calling Conventions

A practical on how to implement a 4 arguments function call with no local variables

A practical hands-on on how to implement a 4 arguments function call with with 4 local variables

A hands-on on how to implement a 5 argument function call

Learn how to implement a Function Call with 6 arguments

Practical on implementing a 7 argument function call.

Apply the knowledge and skills in practical reverse engineering projects

Introduction to the practicals and some tips as well as installing DiE

Preliminary basic skills including how to check the exe file type

Learn how to reverse engineer an exe and use String Search method to phish for passwords

Use the String Search method to phish for passwords.

How to reverse a jump and patch the file

How to reverse a jump and do a simple surface patch.

How to use the Comment Tracing method to analyze register values and control flow logic

How to use the Comment Tracing method to analyze register values and control flow logic

How to directly access memory to modify password and patch the exe file

How to directly access memory to modify password and patch the exe file

How to enable a greyed out button, remove nag message box, accept any serial key and modify message box

How to enable a greyed-out disable Register button

How to remove a Nag Message Box

Patching to accept any serial key and also to modify the message box

Recursively trace into inner function calls to phish out username and password

Deep trace into inner function calls to identify username

Do deep tracing into inner function calls to uncover the password

Step into function calls to trace EAX values that is being returned by the function

Understand the significance of the EAX register in relation to function call returns and step into function calls to trace it.

Identify the correct jump to patch in order to reverse the logic so that the right message is shown

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Covers reverse engineering, a skill set that is highly sought after in fields like cybersecurity, malware analysis, and software development
Taught by Paul Chin, who is an experienced instructor in Assembly Language Programming for Reverse Engineering and a skilled practitioner in Assembly Language
Designed for those aiming to elevate their reverse engineering skills, who have an interest in Assembly Language, and who are reverse-engineering programs
Involves hands-on reverse-engineering exercises with executable binaries, applying knowledge of Assembly Language and techniques from the first course in this series
Uses practical applications and avoids lengthy, dull theoretical lectures, making the learning process more engaging and applicable
Uses the powerful x64dbg debugger to teach assembly language in the context it's most used in the field for reverse engineering, providing a hacker's perspective

Save this course

Save x64 Assembly Language and Reverse Engineering Practicals to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in x64 Assembly Language and Reverse Engineering Practicals with these activities:
Review x64dbg debugger
Using x64dbg debugger is a key component of this course. Doing this refresher will help you get familiar with the tool and its features.
Show steps
  • Download and install x64dbg on your computer
  • Open x64dbg and familiarize yourself with the interface
  • Load a simple executable file into x64dbg and practice debugging it
Follow online tutorials on x64 Assembly Language
There are many tutorials available online that can help you learn assembly language. Following a tutorial can deepen your understanding of some concepts.
Browse courses on Assembly Language
Show steps
  • Find a reputable online tutorial on x64 Assembly Language
  • Follow the tutorial and complete all the exercises
  • Apply what you have learned from the tutorial to your projects
Practice x64 Assembly instructions
Regular practice writing x64 assembly instructions in x64dbg will solidify your understanding.
Browse courses on Assembly Language
Show steps
  • Practice writing simple assembly instructions in x64dbg
  • Practice reading and understanding x64 assembly instructions
  • Practice writing and reading more complex assembly instructions
Five other activities
Expand to see all activities and additional details
Show all eight activities
Read "Practical Reverse Engineering" by Bruce Dang
This book provides a comprehensive overview of reverse engineering techniques.
Show steps
  • Read the book and complete the exercises
  • Apply what you have learned to your own projects
Volunteer as an x64 Assembly mentor
Mentoring others can help you reinforce your own understanding of assembly language.
Browse courses on Assembly Language
Show steps
  • Find a mentoring opportunity or organization
  • Meet with your mentee and provide guidance and support
  • Reflect on your mentoring experience and improve your skills
Create an x64 Assembly program
Creating your own assembly program will give you the opportunity to apply what you have learned and test your skills.
Browse courses on Assembly Language
Show steps
  • Design and plan your program
  • Code your program in x64 Assembly Language
  • Compile and debug your program
  • Test your program and fix any errors
  • Document your program
Contribute to an open-source x64 Assembly project
Contributing to an open source project is a great way to learn from others and improve your skills.
Browse courses on Assembly Language
Show steps
  • Find an open-source x64 Assembly project to contribute to
  • Read the project documentation and familiarize yourself with the codebase
  • Make a change to the project and submit a pull request
Reverse engineer an x64 executable
Reverse engineering an x64 executable is a great way to apply your skills and learn more about how assembly language works.
Browse courses on Assembly Language
Show steps
  • Choose an x64 executable to reverse engineer
  • Load the executable into x64dbg and begin analyzing it
  • Identify the key functions and data structures in the executable
  • Write a report documenting your findings

Career center

Learners who complete x64 Assembly Language and Reverse Engineering Practicals will develop knowledge and skills that may be useful to these careers:

Reading list

We haven't picked any books for this reading list yet.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to x64 Assembly Language and Reverse Engineering Practicals.
Assembly Language Programming for Reverse Engineering
Most relevant
Malware Analysis and Introduction to Assembly Language
Most relevant
Malware Analysis and Assembly Language Introduction
Most relevant
Getting Started with Reverse Engineering
Most relevant
64-Bit Assembly & Shellcoding for Ethical Hackers
Most relevant
Specialized Exploits: Windows and Linux Shellcode
Most relevant
Design a CPU 2
Ghidra Concepts and Basic Functionality
Assembler Language
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser