Ruby on Rails

Ignoring Sidekiq jobs in development

Dennis Paagman
Name
Twitter
@djfpaagman

Sidekiq is well-known and fantastic background processing library for Ruby. I use it in almost every project that needs to run code that can be slow and is not real-time sensitive (the user does not need to see the result on it’s screen immediately), like sending emails or calling outside APIs.

This means that a lot of unnecessary jobs can be queued while you’re testing in your development environment. Most of the time it’s not required to run compute-heavy jobs on your development machine.

One solution is to just not run the Sidekiq process and clean out the queue ocassionally, only start it when you need to actually test the output of some jobs.

You can also prevent certain jobs from running by returning early with an environment check, but this adds unnecessary code and is easy to forget. It also makes debugging harder, as you need to read the code to see why a certain job didn’t do anything.

I wrote a small piece of middleware that allows you set up an ignore list for your development environment. This way you can just run the Sidekiq process and not worry about jobs you don’t care about. At Springest we also use it to make sure certain jobs are not run in our staging environment.

Stick the following code in lib/sidekiq/middleware/ignorer.rb:

module Sidekiq
  module Middleware
    class Ignorer
      # Sidekiq middleware to ignore jobs on an per environment basis. The main
      # use case is to not run heavy data munching jobs on local dev environments.
      #
      # More info on Sidekiq middlewares: https://github.com/mperham/sidekiq/wiki/Middleware
      def call(*args)
        # Sidekiq::Client calls middleware with more arguments than
        # Sidekiq::Server. For that reason we deconstruct the arguments inside
        # the method, so that the original arguments are properly yielded to the
        # next middleware.
        _worker, job, _queue, _redis_pool = args

        if config[::Rails.env]&.include?(job["class"])
          Sidekiq.logger.info "Not queuing #{job["class"]}, because it's on the ignore list for #{::Rails.env}. See config/sidekiq_ignorer.yml."

          false
        else
          yield
        end
      end

      def config
        @config ||= begin
          # Making sure we use ::Rails, otherwise it will us Sidekiq::Rails, where
          # Sidekiq::Rails.root resolves to its gem directory.
          path = ::Rails.root.join("config", "sidekiq_ignorer.yml")
          YAML.load_file(path)
        end
      end
    end
  end
end

Now configure the Sidekiq client and server to use the middleware, you probably already have config/initializers/sidekiq.rb set up, so add it there:

Sidekiq.client_middleware do |chain|
  chain.add Sidekiq::Middleware::Ignorer
end

Sidekiq.server_middleware do |chain|
  # Because jobs can also queue jobs, we need this middleware on the server as well
  chain.add Sidekiq::Middleware::Ignorer
end

Note that we also need to include the middleware on the server, as jobs can queue other jobs as well, and we want those to be ignored as well.

Finally, set up your ignore list in config/sidekiq_ignorer.yml, for example:

# See lib/sidekiq/middleware/ignorer.rb for more info.
#
# If you need to run one of these jobs, temporarily remove it from this list
# and restart sidekiq and your web server.
development:
  - FilterUpdaterWorker

Now you’ll get a nice log message when a job from the list is queued and is ignored.