We may earn an affiliate commission when you visit our partners.
Course image
Hussein Nasser

I used to think that performance of the backend depends on the application logic itself. However, there are many other factors that play a role in overall quality and performance of the application. Networking, connection establishment, security, backend communication, protocol serialization, intermediaries and much more.

Read more

I used to think that performance of the backend depends on the application logic itself. However, there are many other factors that play a role in overall quality and performance of the application. Networking, connection establishment, security, backend communication, protocol serialization, intermediaries and much more.

Often debugging the app if you have the source code allows the developer to zone in to the problem and identify it, However most of the time as an engineer you either don’t have access to the source code or its time consuming to debug a complex app. That is why in this course I present you with some tools I use to analyze the backend application performance and provide a good guess and what might be the problem without stepping in the code. Often known as black box testing.

If your application is a web application that is consumable through a browser, devtools allow us to pretty much inspect all traffic going out from the app and can tell us so much about the app. If the app is not available in the browser we will then demonstrate MITM proxy which is a proxy that intercepts HTTP traffic and log it, this way we can inspect requests and see which of those are the culprit. Finally, if the app uses a protocol that isn’t HTTP intercepting it with a proxy becomes little tricky, so we will use both tcpdump and Wireshark to capture low level packets and see our requests this way.

This course is designed for developers and engineers who have built backend and frontend applications and would like to take their skills further. This course is intermediate to advanced and it is recommended that students have a background in networking and backend fundamentals both of which I have courses for.

Enroll now

What's inside

Learning objectives

  • Identify bottlenecks in backend applications
  • Find where latencies live
  • Intercept slow requests from mobile and web apps
  • Using devtools networking tap to its full potentional
  • Using man in the middle proxy mitm to intercept http requests
  • Using wireshark for packet inspections
  • Real-world performance analysis of backend applications

Syllabus

Introduction
Welcome

For Backend and full stack engineers. 

We will start with analyzing web application running on the browser using DevTools networking. With that we can identify DNS issues, connection and TLS issues and then requests that take the most time.


Because not all apps run on the browser we will use a more advanced tooling to intercept HTTP the raffic between two running applications (a client and as server), MITM proxy is perfect for the job. We will intercept a mobile application and see its requests. We will also learn how to intercept traffic between to services without setting the proxy feature.


Finally when we need to go deeper than HTTP , we will use Wireshark to intercept low level TCP packets and any protocol such as mysql or postgres.


I call these the three tiers levels of analysis



In the final sections I will give an example of a slow running application and how to use all three tools to identify exactly where the bottleneck is.

Read more

In this lecture we will setup the example of a simple C HTTP WebServer that we will use to run our analysis through all tiers. I chose C because it doesn't hide implementation. And we need to know what our backend is doing, intercept , insert break points so we can stop execution in certain points.


The source code attached.


The Kernel Accept, Send and Receive queues
Course downloadable resources
Supplementary Lectures (Fundamentals)
Supplementary Lectures
OSI Model
Internet Protocol
TCP
Proxy vs Reverse Proxy
Here we learn the first level of analysis using DevTools. Learn Chrome DevTools and use it to identify performance issues in Applications
Tier 1 Analysis
Anatomy of a Request/Response

We dive into details about the DevTools architecture and components especially the networking tab, we learn the different columns, the waterfall and what each latency metric means.

DevTools Networking - Part 2 - Waterfall

Backends process request differently depending on the nature of the request, some backends might take time to process a request. Some backends might not have enough workers to process the request and might queue them, or there might be enough workers to process the request but the host is too busy and the architecture was built in a non-scalable manner where many threads running and processes are starving for CPU and ram.


We can have a C web server accept the connection but doesn't write anything to the client until 5 seconds. and close

The server might process and serialize the response very quickly but downloading the response and deserializing it on the client might be what's taking time. Slow network or client side logic or just client might be busy might be the reason.


We can use the web server to accept the connection, immediately write content-length 10 bytes but only write 9. Then sleep for 5 seconds then write the final byte.



The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.


We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.

There are other factors that slows down requests during connection establishment, first we need to perform a DNS to find the IP address of the host. Second after creating the connection we may need to encrypt it with TLS. I will demonstrate examples of when this can be slow. for this I'll use a VPN in Hong Kong to increase latency as devtools simulated latency (alas) does not apply on connection establishment. 

We take a deep look at everything Twitter has to offer through the lens of DevTools

Here we graduate to a more sophisticated analysis that will capture not only the traffic we send but all HTTP traffic to backend, we will use Man-In-The-Middle Proxy (MITM Proxy for that)
Tier 2 Analysis

There are cases where the app is not supported on the browser and is a desktop or mobile app even if the protocol is HTTP. For that we can't use DevTools and we need a slightly different way to sniff requests and connection establishment. We will be adding a proxy layer where the app is instructed to use the proxy to direct all HTTP traffic to the proxy and the proxy will turn around and send the request to the true backend after it captures valuable information. MITM is one of the popular HTTP debugging tools. I also use Fiddler for Windows which is my favorite when I'm on a Windows machine.


In this lecture we will learn all about MITM proxy, setup , configuration, certificate generation, traffic interception etc.

MITM Proxy - Part 2 - Intercepting SSL

Using curl -x to specify the proxy and the backend as our c web server.

Most Mobile app use HTTP as a transport protocol, so while true we can't use DevTools obviously to intercept and example the requests we can still intercept HTTP traffic through MITM proxy. In this lecture we will learn how to direct mobile traffic to MITM proxy instance running on a machine to capture the traffic coming from the App. This can be applied for any HTTP application even desktop ones.


It is important to note that some apps uses certificate pinning so we can't intercept everything.

One of the tricks in debugging micro services or any two services or servers that talk to each other is intercepting the traffic. Sometimes you can set a proxy on one server and capture the traffic as it goes out, however some application simply don't obey proxy rules and override them. In this case we use another trick.


There are many ways to do this but this is just one trick. Say service A  talks to service B , we can pretend MITM proxy is Service B by having it listen on the same port and host as Service B (after stopping Service B). Then we spin up Service B with a different port. We later configure MITM to forward traffic to new Service B port.

This is effectively making MITM as a reverse proxy

This is tier 3 analysis the most verbose and detailed but indeed show us everything that is going on. Analyze any TCP/IP protocol not just HTTP using Wireshark
Tier 3 Analysis
Introduction to Wireshark - Part 1 - Traffic Analysis
Introduction to Wireshark - Part 2 - Decrypting HTTPS/TLS

The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.


We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.


With Wireshark we can  watch how slow the TCP handshake is.

Identify Slow Reading Backends (TCP Zero Window)
Wiresharking Postgres
Identify and pinpoint exactly where the perform is (web app to backend app to a database)

1) When the client app is doing local process (Client-Backend-DB)

The javascript is returning all products then doing a client side filtering to populate the product types in a drop down.

/products/

the fix is to call

/productTypes/


2) When the backend application is doing local processing (Client-Backend-DB)

The client is correctly calling /producTypes but the backend is doing local processing..


Select name from product types then doing a distinct at app side.

The app displays the product types in a dropdown and populate it on start up. /productTypes

The fix is to do a select distinct type from products;



3) Add a load balancer,   sometimes page load is  slow sometimes it's fast.

We fixed one version of the backend to add the distinct, but the others still use the client side filtering.

how: doing a mitm proxy between at the load balancer.

then once we know the bad backend we go to the backend and do Wireshark at that backend to see its missing distinct.


4) Replica database missing index, product details page

we fetch a product by its id , the id index is missing on the replica but exists on the master

how: then once we know the bad backend we go to the backend and do Wireshark at it, we see the slow response at Wireshark to the database ,which tells us its the database and not the backend fault..

Source code attached.

Example 1 - Frontend Performance
Example 2 - Backend Performance
Example 3 - Backend Performance with LoadBalancer
Example 4 - Database Performance
Course Summary
Other essential tools
What we learned
Answering Your questions
How does the Kernel manage backend connections?
Extras
Analyzing ChatGPT

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Explores backend performance analysis, a skill increasing in relevance to industry
Teaches methods of intercepting HTTP requests through MITM proxy and Wireshark, tools highly relevant to industry
Teaches use of DevTools Networking Tap to analyze performance issues in applications, a tool used in industry
Taught by Hussein Nasser, a backend and full-stack engineer with several years of experience in the field
Requires that students have a background in networking and backend fundamentals, potentially creating a barrier to entry for novice learners

Save this course

Save Troubleshooting Backend Performance to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Troubleshooting Backend Performance with these activities:
Review computer networking fundamentals
Strengthen your understanding of computer networks, which are essential for analyzing backend performance.
Browse courses on Networking
Show steps
  • Review the OSI model and its layers
  • Learn about different types of networks, including LANs, WANs, and MANs
  • Understand how IP addresses and MAC addresses work
  • Configure a basic network using a router and switch
Follow tutorials on using DevTools for performance analysis
Gain hands-on experience using DevTools, a powerful tool for analyzing backend performance in web applications.
Show steps
  • Find tutorials on using DevTools for performance analysis
  • Follow the tutorials to learn how to use DevTools to identify performance bottlenecks
  • Practice using DevTools to analyze the performance of your own web applications
Practice identifying performance bottlenecks in backend applications
Develop your skills in identifying performance bottlenecks in backend applications through repetitive exercises.
Show steps
  • Set up a backend application with known performance bottlenecks
  • Use the techniques learned in the course to identify the bottlenecks
  • Repeat the process with different backend applications to improve your skills
Show all three activities

Career center

Learners who complete Troubleshooting Backend Performance will develop knowledge and skills that may be useful to these careers:

Reading list

We haven't picked any books for this reading list yet.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Troubleshooting Backend Performance.
Leveraging Load Balancing Options on the GCP
Most relevant
web2py 2 Fundamentals
The Startup Founder's Guide to Bubble Programming
API Testing a real web application via Postman
MERN Stack Front To Back: Full Stack React, Redux &...
Debugging, Testing, and Publishing a Flutter 2 App
Modernizing Applications with Apigee X
Angular & NodeJS - The MEAN Stack Guide [2024 Edition]
React 18 Debugging Playbook
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser