Get the latest tech news
Opsqueue: Lightweight batch processing queue for heavy loads – now open-source
We are happy to announce the open-source release of opsqueue, our opinionated queueing system. Opsqueue is a lightweight batch processing queue, optimized for high-throughput situations. It is very operationally simple and flexible to use, with a novel prioritization approach where decisions can be made in the middle of ongoing work.
One example would be message brokers like RabbitMQ(or hosted versions like Google's PubSub or Amazon's SQS), another, if you want the same streams to be handled in different ways by multiple services, would be distributed stream-processing platforms like Apache Kafka. But flipping that around: if individual operations take longer, you can get away with a smaller chunk size, which means that you can still get the benefits of higher parallelism and quick recovery from failures. : What is 'specialized' is that in the normal producer-consumer, communication happens in one direction, whereas in 'generate-execute' the Producer is interested in the outcome of the Consumer(s) as well, to check for errors and/or do further processing.
Or read this on Hacker News