Background Elastic Search indexing using Sidekiq

5 years ago

This article shows how we implemented background ActiveRecord model indexing using Sidekiq. By default, the SearchKick gem allows for 4 ways of automatically indexing a Rails model when it is saved to the backing database:

  1. Inline (the default, synchronously, anytime a record is inserted, updated, or deleted)
  2. Asynchronous (using ActiveJob)
  3. Asynchronous (using a Redis Queue)
  4. Manual (i.e. disabled automatic syncing)

Since we are using Sidekiq for our background jobs, we clearly needed a custom way to automatically index our models. Enter the Indexable concern:

#
# Adds support for calling a sidekiq job to index the model
# that includes this module.
#
module Indexable
extend ActiveSupport::Concern
  included do
def index_elasticsearch
ElasticSearchWorker.perform_async(self.class.to_s, id)
self
end
end
end

Indexing the model when it updates then becomes relatively generic and straightforward:

class Vehicle < ActiveRecord::Base
include Indexable
  searchkick callbacks: false
after_commit :index_elasticsearch
  def search_data
{
brand: self.brand,
licence_plate: self.licence_plate,
(...)
}
end
  (...)
end

Notice that:

  1. We include the concern in the model
  2. We instruct SearchKick to deactivate automatic indexing (by setting callbacks = false)
  3. We tell Rails to run the Indexable concern’s index_elasticsearch method whenever there is a database commit on this model
  4. And finally we just define the Hash we want to index regarding this model (in our example just the brand and licence plate)

There’s only one missing piece of the puzzle, which is the Sidekiq worker to index any models we mark as needing reindexing:

class ElasticSearchWorker
include Sidekiq::Worker
sidekiq_options queue: :elastic_search, retry: 10
def perform(clazz, id)
Rails.logger.debug("Indexing #{clazz} id #{id}")
clazz.constantize.find(id).reindex
end
end

In this file we basically just define which queue name we want for these jobs and call SearchKick’s reindex method on the model we are indexing.

As long as you have sidekiq up and running, its workers should pickup these jobs and index your data.

Happy indexing!


Background Elastic Search indexing using Sidekiq was originally published in Drover Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.