HACKER Q&A
📣 iamwil

What are you guys using for background workers nowadays?


Back when I was working with Rails about 4-10 years ago, I used background workers (sidekiq, faktory, delayed job, resque) to offload processing that can happen outside of the request/response cycle, to keep response times down. Examples might be sending email or making different sizes of uploaded images.

What do people of all stacks use nowadays? Do JAM stack devs use web workers and service workers? Or are people using AWS lambda for this stuff (or beanstalk? blah)? Or are you using Kubernete's job runners? Or do you offload it to Zapier? Or do you use big data tools like Apache Spark, because it's specific to the job, and a generic runner won't do? Or are there mature services for the specific offline processing that you need that you can just punt it to a 3rd party?

For me, we've co-opted event handlers in Hasura as a poor man's background job runner, and I wish there was something that gave me more visibility into job throughput and performance, easier debugging, chainability between jobs, and integration with 3rd party services. Anyone have a job running framework/platform that they love?


  👤 Nextgrid Accepted Answer ✓
Celery works fine for my needs and happens to be provider-agnostic and very easy to understand and run locally. I haven't had experience with the Rails-specific solutions you mention but is there any problem with them?