Ask HN: How would you queue and process 10K+ long running jobs
9 by bballer | 24 comments on Hacker News.
Hey guys wanted to ask a question about what technologies & methodologies you would architect together if you needed to constantly be queuing up 10K+ jobs, distributing the work out and then reporting that it was completed. It would be required that you should never be able to schedule a duplicate job when one is queued/running, and you need to ensure that each job only gets picked up by 1 worker and run once. These jobs could last anywhere (in run time) from 30 seconds to 1 hour. I've tried googling but my fu is failing me. Would love to hear the thoughts from people who have maybe solved similar problems. Thanks!
Hey guys wanted to ask a question about what technologies & methodologies you would architect together if you needed to constantly be queuing up 10K+ jobs, distributing the work out and then reporting that it was completed. It would be required that you should never be able to schedule a duplicate job when one is queued/running, and you need to ensure that each job only gets picked up by 1 worker and run once. These jobs could last anywhere (in run time) from 30 seconds to 1 hour. I've tried googling but my fu is failing me. Would love to hear the thoughts from people who have maybe solved similar problems. Thanks!
No comments:
Post a Comment