We may earn an affiliate commission when you visit our partners.
Course image
Udemy logo

Distributed Tasks Demystified with Celery, SQS & Python

Timothy Mugayi

This course teaches beginners to industry professionals the fundamental concepts of Distributed Programming in the context of python & Django. We look at how to build applications that increase throughput and reduce latency. In this course, we will take a dive intially in the irst part of the course and build a strong foundation of asynchronous parallel tasks using python-celery a distributed task queue framework. We will explore AWS SQS for scaling our parallel tasks on the cloud. These fundamentals will aid you in building scalable Python solutions for virtually any python project. By the end of this course, you will have learnt how to use popular distributed programming frameworks for python and Django. Through concepts learnt, you will discover the world of distributed computing with Python and how easy it is to build distributed components into your python or Django projects.

Read more

This course teaches beginners to industry professionals the fundamental concepts of Distributed Programming in the context of python & Django. We look at how to build applications that increase throughput and reduce latency. In this course, we will take a dive intially in the irst part of the course and build a strong foundation of asynchronous parallel tasks using python-celery a distributed task queue framework. We will explore AWS SQS for scaling our parallel tasks on the cloud. These fundamentals will aid you in building scalable Python solutions for virtually any python project. By the end of this course, you will have learnt how to use popular distributed programming frameworks for python and Django. Through concepts learnt, you will discover the world of distributed computing with Python and how easy it is to build distributed components into your python or Django projects.

Why take this course?

  • As to be python developer or Django web application developer, Its important to learn how to build applications that are able to process long running jobs or tasks in a non blocking way e.g sending mass email, map reduce, running high computational functions. Building a chatting applications.
  • This course builds up your skills in distribute programming giving you the tools you need to scale your applications
  • This course consist of projects you can use to implement in your real world projects with the sole emphasis of allowing your applications to become distributed and have asynchronous components
  • During the course, you will have online access to the instructor and get individualized answers to your questions posted on forums.
  • As an enrolled student you will get lifetime access to over 30 lectures plus and counting including future series of lessons
  • This course comes with a 30 day money back guarantee. This means If you are not satisfied in any way, you'll get your money back. Though you may just miss out of the future existing lessons planned ahead and free source code for your own personal or business projects. So what are you waiting for?

"Come join me and lets build distributed Python apps in a way that will advance your career, enhance your knowledge and potiential to earn even more. The best part is this doesnt need to be hard it actually can be fun"

Enroll now

What's inside

Learning objectives

  • Fundamentals of multithreading in python
  • How to implement distributed tasks with python & django
  • Implement message passing communication between processes to build parallel applications
  • How to scale on the cloud with aws simple queue service (sqs)
  • Learn how to build a distributed social media data ingestor

Syllabus

Introduction

In this lesson we cover all the packages that we need to get started with the course and prep up our development workspace.

Read more
Getting started with Multithreading in Python
Blocking vs non blocking (part 1)
Blocking vs non blocking (part 2)
Concurrency Consumer & Producer problem a deep dive
Solving Consumer producers problem with Mutual Exlusion

In this lesson we will discusses thread synchronization mechanisms in Python and how to synchronize access to shared resources, We take the consumer & producer problem from a different angle with conditions 

Controlling threads with conditions (Part 2)
Controlling threads with conditions (Part 3)
Daemon threads by example (Part 4)
Consumer producer a thread safe FIFO queue
Core Celery Distributed Tasks
Getting started with Celery
Celery backends & Asyncresult by example
Python exception handling back to the basics
Exception handling in Celery Explained
Celery scheduled periodic tasks (Part 1)
Celery scheduled periodic tasks (Part 2)

In this lesson, we will touch base on a key important aspect of incorporating locks in distributed workers. Distributed locks are a useful primitive in many environments where different processes must operate on shared resources in a mutually exclusive way.

Celery scheduled periodic tasks solar schedules
Distributed tasks with AWS SQS
Introduction to distributed tasks with AWS SQS

In this lesson we learn how to configure AWS simple queue service for distributed applications

How to create a AWS SQS background worker in python (Part 1)

PENDING LAUNCH 28/10/2017

In this lesson we take a look at what are dead letter queues and how they fit in distributed services with sqs

In this lesson we see a live example of how to implement dead letter queues in AWS SQS

In this special video we take a look at how to bypass AWS SQS 256kb hard limit when sending large data via SQS queues. 

Distributed data ingestor Project #1

With the advent of numerous Social media platforms, the need has become of real importance to have the ability to tap into these platforms for their data.  Such data allows businesses to understand the wants and needs of customers. Data extraction poses a fantastic use case to explore how to build and orchestrate distributed processes to pull data from various social media platforms. In this series of data injestor project lessons, we explore how to build our own distributed data injestor that will allow us to aggregate raw data while taking all concepts learnt and implementing them into a real-world use case that has been implemented by large multi-million dollar cooperation’s. 

Real world examples of data ingestors
Creating a twitter developer application and Authentication Token
Building your first social ingestor twitter (Part 1)
Building your first social ingestor twitter (Part 2)

Most APIs are subject to a limit on how many calls can be made per second (or minute, or other short time period), This is in order to protect servers from being overloaded and maintain high quality of service for other clients wishing to use a service. In This lesson we explore twitter rate limits and how to overcome them

Building your first social ingestor twitter Handle (Part 4)
Building your first social ingestor twitter Handle (Part 5)
Distributed Email Workers Project #2

In this lesson we dive into the core basics of how SMTP works and look at what are your options for transactional email services

In this series of lessons we take a look at how we can craft out our email background worker. 

Creating your first background email worker (Part 2)
Creating your first background email worker (Part 3)
Learn about tools you can leverage on to enhance your overall development productivity and code quality when building distributed applications

In this video we take a walk through getting started with pycharm IDE

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Develops fundamental concepts of Parallel programming, Distributed tasks, Message passing communication, and Asynchronous programming
Prerequisite knowledge of Python programming is necessary to take this course
Teaches students to build applications that can process long running jobs or tasks in a non blocking way
Helps students build up their skills in distributed programming, giving them the tools they need to scale their applications
Provides students with projects that they can use to implement in their real world projects with the sole emphasis of allowing their applications to become distributed and have asynchronous components
Taught by Timothy Mugayi, a recognized instructor in Python and Django

Save this course

Save Distributed Tasks Demystified with Celery, SQS & Python to your list so you can find it easily later:
Save

Career center

Learners who complete Distributed Tasks Demystified with Celery, SQS & Python will develop knowledge and skills that may be useful to these careers:
Data Engineer
Data Engineers are in high demand, particularly given the rise of big data and the cloud. This course gives you all the tools to get started as a Data Engineer, and learning from an expert career advisor can ensure that you're fully prepared for the job market. You'll be building a social media data ingestors and a distributed email worker by the end of the course, which means you'll have some work to showcase when applying for your first Data Engineering role.
Software Engineer
Software Engineers plan, design, develop, test, and maintain computer software. This course will help you build a strong foundation in distributed programming, which is a skill that's in high demand for Software Engineers. By the end of the course, you'll have a portfolio of projects, including a distributed social media data ingestor, that you can show off when applying for roles.
Cloud Architect
Cloud Architects design, build, and manage cloud computing systems. This course will teach you how to scale on the cloud with AWS Simple Queue Service (SQS), which is a key skill for Cloud Architects. You'll also learn how to build a distributed social media data ingestor, which is a valuable project to add to your resume when applying for Cloud Architect roles.
Data Analyst
Data Analysts collect, clean, and analyze data to help businesses make informed decisions. This course will help you build a foundation in distributed programming, which is a skill that's increasingly important for Data Analysts. You'll also learn how to build a distributed social media data ingestor, which is a project that can help you stand out from other applicants when applying for Data Analyst roles.
Business Analyst
Business Analysts translate business requirements into technical specifications. This course can help you build a deeper understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Business Analysts, especially those who want to work in the tech industry.
Product Manager
Product Managers oversee the development and launch of products. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Product Managers, especially those who want to work in the tech industry.
Project Manager
Project Managers plan, execute, and close projects. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Project Managers, especially those who want to work in the tech industry.
Database Administrator
Database Administrators manage, maintain, and troubleshoot databases. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Database Administrators, especially those who want to work in the tech industry.
Systems Analyst
Systems Analysts analyze and design computer systems. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Systems Analysts, especially those who want to work in the tech industry.
Security Analyst
Security Analysts protect computer systems from unauthorized access and use. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Security Analysts, especially those who want to work in the tech industry.
Network Administrator
Network Administrators manage, maintain, and troubleshoot computer networks. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Network Administrators, especially those who want to work in the tech industry.
Computer Programmer
Computer Programmers write, test, and maintain computer programs. This course can help you build a stronger foundation in programming, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Computer Programmers, especially those who want to work in the tech industry.
Computer Support Specialist
Computer Support Specialists provide technical support to users of computers and computer systems. This course can help you build a better understanding of the technical side of things, particularly when it comes to distributed programming and cloud computing. This knowledge can be very valuable for Computer Support Specialists, especially those who want to work in the tech industry.
Web Developer
Web Developers design, develop, and maintain websites. This course may be of some use to you as a Web Developer, particularly if you're interested in learning more about distributed programming and cloud computing. However, it's important to note that this course is not specifically tailored to Web Developers.
UX Designer
UX Designers design the user experience for websites and other digital products. This course may be of some use to you as a UX Designer, particularly if you're interested in learning more about distributed programming and cloud computing. However, it's important to note that this course is not specifically tailored to UX Designers.

Reading list

We've selected eight books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Distributed Tasks Demystified with Celery, SQS & Python.
A textbook that provides a comprehensive overview of modern operating systems, including concurrency and distributed systems. Provides a solid theoretical foundation for understanding distributed systems.
A practical guide to designing and building data-intensive applications that are reliable, scalable, and efficient.
Covers the fundamentals of cloud-native development using Python, including best practices and tools for building and deploying scalable, resilient applications.
A practical guide to performance engineering for distributed systems, covering techniques for performance measurement, analysis, and optimization.
Presents patterns and techniques for writing concurrent programs that are correct, efficient, and maintainable.
Provides a comprehensive overview of DevOps, covering principles, practices, and tools. Useful as additional reading for those interested in the broader context of DevOps.
A business novel that explores the challenges of managing and delivering software projects effectively. Provides insights into DevOps and agile methodologies.

Share

Help others find this course page by sharing it with your friends and followers:
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser