So to be hip, I am going to write one of these multipart blog things that describe building a simple project from start to finish. Are you ready? O.K. here we go.
What is a Queue Server?
A queue server is a server that processes a queue, which is why it is so aptly named. You send it a list of things to do and it does them over time based on your parameters, be it “as fast as possible” or “when system resources allow” or “once an hour” it is your call.
Over the course of the next week or so I am going to demonstrate building a simple queue server that will be able to fork itself off as a background daemon, accept input through a named pipe, and process things as fast as it can until it runs out of things to do – and when that happens it will sit there waiting with the upmost desire for you to provide it with more jobs.
Why we might need a Queue Service.
Take into consideration the following graphic.
This graphic visualizes a basic use case of a behind the scenes queue system. Starting from the top left we begin on a webpage that takes some sort of user input, and through some sort of API, perhaps AJAX perhaps not, takes that user input and dumps it on top of a job queue that is already busy processing other requests.
Once the API has dropped whatever data it needs on the job queue, it sends us to a new page on which we can see the status of the current job. While the job is pending, every 10 seconds or so we can run an AJAX request (or something) to check another API to see if the job is done. If not we stay on the page we are on, and check again another 10 seconds later. If the job is done, then the page is updated to reflect that and we stop checking the API.
One problem this overcomes is if the job takes a long time to process, the web server and script will not timeout waiting for results. By default PHP runs for a max of 30 seconds before it kills itself for running too long, so if the job is going to take two minutes to process you might see how that be bad. Second, locking the user on a blank white screen while PHP waits is very poor usability. Perhaps they would like to do other things on your site while they wait? Just an idea.
We are going to build a simple queue system to do a stupid simple job. In fact the job will be to sleep, just to simulate a process that does not finish instantly. The only real important note here is that I am assuming we are running on a Unix like OS, be it Linux, BSD, Solaris, whatever. Through the next set of posts we will be developing this application fairly procedurally, as the point of the series is to first highlight how to do specific things in PHP (queue, pipe, fork, etc) and the second, bring them all together in a single application. The very last post in the series I will take the final product and refactor it for object orientation just for fun.
- Part 1 – Understanding the project. <= you are here
- Part 2 – Building the queue structure.
- Part 3 – Accepting input from a Named Pipe.
- Part 4 – Run as a background process, a.k.a. forking.
- Part 5 – Refactoring for Object Orientation.
So check back soon for part 2, where we will build a really basic queue system and then evolve it into awesome over parts 3 and 4.