Adam Carlile
Queuing Sunspot indexing jobs with Resque

I’m currently working on a social networking platform(very beta!) for London, and we use Resque to handle all of our background jobs, sending outbound mail, image processing etc.

We recently implemented Sunspot and Solr for indexing our models to allow rapid full text searching. Now Sunspot is awesome, has a great DSL and makes indexing and searching super easy, it also has great built in support for DelayedJob, which is another awesome background processing framework.

However since we’ve already implemented Resque it seemed the logical choice to integrate it with Sunspot. But documentation seems to be patchy at best as to how to implement the queue with Resque.

Enter SunspotSessionProxy

SunspotSessionProxy is a class that sits between the Sunspot interface and the Solr server, allowing you to change the behaviour of certain actions that Sunspot will perform, such as indexing, searching etc.

So through this class we can modify the default behaviour of the index action to route it via a Resque worker.

This class delegates all of it’s methods to the Session class, apart from the index method, which we have rewritten to include a call to Resque to enqueue the job.

We also keep hold of the original session so that we can call it from inside the Resque worker, so we can actually run the original indexing method, instead of our modified index method

Work, Work, Work

Here’s the code for the Resque worker.

Very simple, just takes the arguments passed by the ResqueSessionProxy#index, gets all the AR objects, and then runs the original sessions index method from inside the worker thread

The Final Piece

Finally we have to tell Resque to use this new session proxy, so just a simple line an in initialiser will do it

That’s it! Hopefully that has helped some of you, as I’ve not seen any documentation on this subject

  1. adamcarlile posted this