I used to think that performance of the backend depends on the application logic itself. However, there are many other factors that play a role in overall quality and performance of the application. Networking, connection establishment, security, backend communication, protocol serialization, intermediaries and much more.
I used to think that performance of the backend depends on the application logic itself. However, there are many other factors that play a role in overall quality and performance of the application. Networking, connection establishment, security, backend communication, protocol serialization, intermediaries and much more.
Often debugging the app if you have the source code allows the developer to zone in to the problem and identify it, However most of the time as an engineer you either don’t have access to the source code or its time consuming to debug a complex app. That is why in this course I present you with some tools I use to analyze the backend application performance and provide a good guess and what might be the problem without stepping in the code. Often known as black box testing.
If your application is a web application that is consumable through a browser, devtools allow us to pretty much inspect all traffic going out from the app and can tell us so much about the app. If the app is not available in the browser we will then demonstrate MITM proxy which is a proxy that intercepts HTTP traffic and log it, this way we can inspect requests and see which of those are the culprit. Finally, if the app uses a protocol that isn’t HTTP intercepting it with a proxy becomes little tricky, so we will use both tcpdump and Wireshark to capture low level packets and see our requests this way.
This course is designed for developers and engineers who have built backend and frontend applications and would like to take their skills further. This course is intermediate to advanced and it is recommended that students have a background in networking and backend fundamentals both of which I have courses for.
For Backend and full stack engineers.
We will start with analyzing web application running on the browser using DevTools networking. With that we can identify DNS issues, connection and TLS issues and then requests that take the most time.
Because not all apps run on the browser we will use a more advanced tooling to intercept HTTP the raffic between two running applications (a client and as server), MITM proxy is perfect for the job. We will intercept a mobile application and see its requests. We will also learn how to intercept traffic between to services without setting the proxy feature.
Finally when we need to go deeper than HTTP , we will use Wireshark to intercept low level TCP packets and any protocol such as mysql or postgres.
I call these the three tiers levels of analysis
In the final sections I will give an example of a slow running application and how to use all three tools to identify exactly where the bottleneck is.
In this lecture we will setup the example of a simple C HTTP WebServer that we will use to run our analysis through all tiers. I chose C because it doesn't hide implementation. And we need to know what our backend is doing, intercept , insert break points so we can stop execution in certain points.
The source code attached.
We dive into details about the DevTools architecture and components especially the networking tab, we learn the different columns, the waterfall and what each latency metric means.
Backends process request differently depending on the nature of the request, some backends might take time to process a request. Some backends might not have enough workers to process the request and might queue them, or there might be enough workers to process the request but the host is too busy and the architecture was built in a non-scalable manner where many threads running and processes are starving for CPU and ram.
We can have a C web server accept the connection but doesn't write anything to the client until 5 seconds. and close
The server might process and serialize the response very quickly but downloading the response and deserializing it on the client might be what's taking time. Slow network or client side logic or just client might be busy might be the reason.
We can use the web server to accept the connection, immediately write content-length 10 bytes but only write 9. Then sleep for 5 seconds then write the final byte.
The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.
We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.
There are other factors that slows down requests during connection establishment, first we need to perform a DNS to find the IP address of the host. Second after creating the connection we may need to encrypt it with TLS. I will demonstrate examples of when this can be slow. for this I'll use a VPN in Hong Kong to increase latency as devtools simulated latency (alas) does not apply on connection establishment.
We take a deep look at everything Twitter has to offer through the lens of DevTools
There are cases where the app is not supported on the browser and is a desktop or mobile app even if the protocol is HTTP. For that we can't use DevTools and we need a slightly different way to sniff requests and connection establishment. We will be adding a proxy layer where the app is instructed to use the proxy to direct all HTTP traffic to the proxy and the proxy will turn around and send the request to the true backend after it captures valuable information. MITM is one of the popular HTTP debugging tools. I also use Fiddler for Windows which is my favorite when I'm on a Windows machine.
In this lecture we will learn all about MITM proxy, setup , configuration, certificate generation, traffic interception etc.
Using curl -x to specify the proxy and the backend as our c web server.
Most Mobile app use HTTP as a transport protocol, so while true we can't use DevTools obviously to intercept and example the requests we can still intercept HTTP traffic through MITM proxy. In this lecture we will learn how to direct mobile traffic to MITM proxy instance running on a machine to capture the traffic coming from the App. This can be applied for any HTTP application even desktop ones.
It is important to note that some apps uses certificate pinning so we can't intercept everything.
One of the tricks in debugging micro services or any two services or servers that talk to each other is intercepting the traffic. Sometimes you can set a proxy on one server and capture the traffic as it goes out, however some application simply don't obey proxy rules and override them. In this case we use another trick.
There are many ways to do this but this is just one trick. Say service A talks to service B , we can pretend MITM proxy is Service B by having it listen on the same port and host as Service B (after stopping Service B). Then we spin up Service B with a different port. We later configure MITM to forward traffic to new Service B port.
This is effectively making MITM as a reverse proxy
The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.
We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.
With Wireshark we can watch how slow the TCP handshake is.
1) When the client app is doing local process (Client-Backend-DB)
The javascript is returning all products then doing a client side filtering to populate the product types in a drop down.
/products/
the fix is to call
/productTypes/
2) When the backend application is doing local processing (Client-Backend-DB)
The client is correctly calling /producTypes but the backend is doing local processing..
Select name from product types then doing a distinct at app side.
The app displays the product types in a dropdown and populate it on start up. /productTypes
The fix is to do a select distinct type from products;
3) Add a load balancer, sometimes page load is slow sometimes it's fast.
We fixed one version of the backend to add the distinct, but the others still use the client side filtering.
how: doing a mitm proxy between at the load balancer.
then once we know the bad backend we go to the backend and do Wireshark at that backend to see its missing distinct.
4) Replica database missing index, product details page
we fetch a product by its id , the id index is missing on the replica but exists on the master
how: then once we know the bad backend we go to the backend and do Wireshark at it, we see the slow response at Wireshark to the database ,which tells us its the database and not the backend fault..
Source code attached.
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.