Django & Celery – Easy async task processing

So, while developing a web application, there comes a time when we need to process some of the tasks in the background, perhaps asynchronously. For example, your user would upload photos and the app would post them to multiple social networks. We would definitely want to offload the uploading task to some background workers.

Django and Celery makes background task processing a breeze. In this article, we shall see how we can setup Django and Celery to start processing our background tasks. We would use Redis to maintain our task queue.

How does it work?

  • We define some tasks in our application. These tasks are expected to run for a pretty long time.
  • We run the celery workers. Celery knows how to find and load these tasks. The workers keep waiting on us.
  • We add some jobs to the workers queue from our web app. The workers now have something to work on. So they start taking the jobs from the queue and start processing them.
  • We can query the status of the jobs from our web app to know whats happening.
  • The easy to use Python API makes it really simple to use. You don’t need any specialisation or anything in Redis.

Setting Up

Let’s first install the Redis server:

The version that comes from Ubuntu official repo is quite old. You can install the latest version from 3rd party PPAs.

Install Celery with Redis support:

And then install django-celery package:

Configuration

Add “djcelery” to your installed apps list:

Modify your main app’s settings.py file to add the celery specific settings:

Now, inside your main application directory (the directory in which settings.py is located), create a file named “celery.py” with these contents:

The above codes do a few things:

  • It creates our own Celery instance.
  • We ask the celery instance to load necessary configs from our project’s settings file.
  • We make the instance auto discover tasks from our INSTALLED_APPS.

Also let’s modify the “__init__.py” file in the same directory to make the celery app available more easily:

This would allow us to use the same app instance for shared tasks across reusable django apps.

Defining Tasks

Now let’s create a tasks.py file in one of our INSTALLED_APPS and add these contents:

Now we have defined our own celery app, we have our tasks. It’s now time to launch the workers and start adding tasks.

Processing Tasks

Before we can start processing tasks, we have to launch the celery daemon first. This is how we do it:

Here, we tell celery to use the celery instance we defined and configured earlier. Here “project” is the main app, the package that contains our settings.py along with celery.py. The “app” the variable name which holds the celery instance.

Now let’s use the Django shell to add and query jobs:

So as we can see, out task was processed by celery. And we could easily query the status. We would generally use the meta data to store any task related information.

Object Oriented Programming in Python: Bangla Screencast

Video is available in HD

সংযুক্তি – ১

স্ক্রীনকাস্টে প্রাইভেট ভ্যারিয়েবল বা ডাটা হাইডিং এর উদাহরন দেখানোর সময় ক্লাস লেভেল ভ্যারিয়েবল ব্যবহার করা হয়েছে । ইনস্ট্যান্স ভ্যারিয়েবল এর ক্ষেত্রেও একই নিয়ম প্রযোজ্য । এখানে কোড স্যাম্পল দেওয়া হলো:

সংযুক্তি – ২

ক্লাস নির্ভর মেটাক্লাস ইম্প্লিমেন্টেশন এবং একটি কাল্পনিক প্রয়োগ দেখার জন্য এই ভিডিও টি দেখতে পারেন – Python – Metaprogramming

Installing IPython Notebook on OS X

If you do not know what is IPython Notebook, please checkout the link here – http://ipython.org/notebook.html. IPython is an awesome tool that allows you to combine Python codes with rich text to create meaningful & interactive contents.

Here’s a quick demonstration:

In this post, I am going to share what I did to setup IPython Notebook on my OS X. These are the commands I used. Homebrew was used to install libraries/dependencies. Pip was used to install Python packages.

If you use the same commands and everything works fine, you will have a working installation of IPython Notebook. You can launch the notebook web server using the following command:

This will launch the server and open your default web browser :) Please note that the notebook will use your current directory as the root while displaying the files and directories. I created a dedicated directory inside my Google Drive to store my notebooks. So I created an alias for always starting the notebook server pointed to that directory. I put this in my .zshrc:

Now from any place, I can issue this command to run the server:

I am currently playing with both IPython and the IPython Notebook. Both are awesome :)

Tessel: Review

Tessel is a micro controller that runs JavaScript. It is NodeJS compatible – that means you can take advantages of npm and all the NodeJS awesomeness with it. The key attraction for me was the built in Wifi, makes it very easy to connect to the internet. Tessel has a good collection of modules which you can connect to the board to extend it’s functionality. The modules are listed here: https://tessel.io/modules. Each module has a dedicated npm package. You just have to install the package and require it to interact with the module. Cool, no?

I ordered a Tessel from their website on June 14. I got the board at hand today, July 20. They shipped via DHL global mail.

Here’s the tessel main board:

Tessel Main Board

I also purchased a GPRS module to interact with mobile networks. Here’s a photo of the module connected to the main board:

Tessel with GPRS Module

Setting It Up

Setting up was easy. I already has NodeJS installed. I installed the tessel command line tool.

After tessel is installed, I tried listing the devices:

Then I copied the example blinking codes and tried them out. All worked fine. So my tessel was working! Then I tried listing the available wifi networks:

I connected to my home wifi:

It connected to the Wifi just like that! Awesome!

Then I tested out the instructions on the “Tweet” section which uses the recently connected WiFi to send out a tweet.

The WiFi is working excellent!

The GPRS Module

Now I connected the GPRS module and tested out the sample codes. I used a spare Banglalink SIM to send out the SMS. It turns out that you don’t need to type the number in international format. The 017******** format works too :)

tesla

Just a quick note: If you follow the official example (it sends a text on gprs.ready event), and then kill the process by hitting Ctrl + C, you might have issues controlling the GPRS module without restarting it. So I think for testing, we can use the Tessel REPL from the command line: