The tag has no usage guidance.

learn more… | top users | synonyms

34
votes
2answers
19k views

When to use Advanced Message Queuing Protocol like RabbitMQ? [closed]

Can somebody explain me in what use-cases I should consider using AMQP like e.g. RabbitMQ? What are the pros and cons?
24
votes
1answer
12k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
18
votes
4answers
399 views

Using flat files vs database/API as a transport between a frontend and backend

I've got an application which has generated a rather heated discussion between a couple of the developers. Basically, it's split into a web layer and a backend layer. The web layer collects ...
18
votes
1answer
7k views

What are the solutions to the Distributed Queue problem?

I am trying to learn more about the various ways that the problem of a Distributed Queue may be solved. So I would like know what products, services, implementations and research papers that are ...
14
votes
1answer
8k views

Why database as queue so bad? [closed]

I've just read this article, and I'm confused. Let's imagine 1 webapp and 1 distinct application acting as "worker", both sharing the same database. Oh, I said "sharing"..but what does the article ...
11
votes
1answer
5k views

Designing an scalable message queue architecture

I have recently started learning the nuances of scalable and enterprise computer architecture, and one of the central components is a messaging queue. In order to learn the most I can from any ...
8
votes
2answers
595 views

Product agnostic book on Message Queues? [closed]

Are there any recommendations for book(s) that are largely agnostic (with examples of MQ implementations is fine) on vendor but go to great details on the architecture, management, nomenclature, ...
6
votes
2answers
674 views

Implementation of communication between packages (Java)

I'm making a project with 5 packages. The packages can communicate with each other by sending Messages to a central MessageQueue in the Main package and that MessageQueue takes care of all the ...
6
votes
1answer
1k views

Does Akka obsolesce JMS/AMQP message brokers? [closed]

I spent the last week deep diving into the Akka docs and finally understand what actor systems are, and the problems that they solve. My understanding (and experience with) traditional JMS/AMQP ...
5
votes
3answers
2k views

NoSQL Document Database as a Message Queue

I am considering using a NoSQL Document database as a messaging queue. Here is why: I want a client to post a message (some serialized object) to the server and not have to wait for a synchronous ...
5
votes
2answers
7k views

REST or a message queue in a multi-tier heterogeneous system?

I'm designing a REST API for a three-tier system like: Client application -> Front-end API cloud server -> user's home API server (Home). Home is a home device, and is supposed to maintain connection ...
5
votes
1answer
93 views

How to Recover from Inconsistent Job State without Database Polling

I'm working on scaling an application which is currently polling a mySQL database to send async jobs to a background processing system. A state machine is being used to keep track of the entities ...
5
votes
1answer
3k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
4
votes
3answers
519 views

Web Service and Message Queue

We're looking at creating a web services/REST API layer that will be consumed by web and mobile clients. To make the solution more robust I was considering putting the commands from PUT, POST, and ...
4
votes
2answers
1k views

Where does Windows Workflow Foundation fit into the messaging oriented middleware architecture?

I've been trying to find information online about where WWF 4.5 fits with in an architecture based around an enterprise message queue using central broker(s) (RabbitMQ in particular). It seems like ...
4
votes
1answer
202 views

Message queue vs database for delayed tasks

I need to build a system that can handle a fairly high amount of delayed tasks (e.g. scheduled emails). For non-delayed tasks I would go for something like RabbitMQ. But, is it ok to let tasks ...
4
votes
2answers
244 views

Understanding the Single-Writer Principle

The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
4
votes
1answer
154 views

Azure ServiceBus Queue: Consumer Application

I am running a .net MVC4 web application. I want to be able to hit an action on a controller, create an email and send it to my service bus queue to be processed (sent). What are my options as far ...
4
votes
4answers
763 views

Architecting Python application consisting of many small scripts

I am building an application which, at the moment, consists of many small Python scripts. Each Python script processes items from one Amazon SQS queue. Emails come into an initial queue and are ...
4
votes
1answer
210 views

Is it always the best way to go to queue emails instead of sending them real-time? [duplicate]

If you had to decide between queueing emails and sending them real-time, is it always the best way to go putting them in a queue and sending them one by one instead of instantly as soon as a user ...
3
votes
3answers
3k views

UML: Queue processor in a sequence diagram

Which is the appropriate way of representing a queue processor in a UML sequence diagram? I want to represent in the same diagram two systems that are loosely coupled through a queue. The first ...
3
votes
1answer
478 views

How to get JMS to front end

I currently have a web application and I would like to add a messaging feature to it. In order to do that, I use JMS(actually OpenMQ, the implementation provided with Glassfish 3). The problem is that ...
3
votes
1answer
381 views

Passing notifications between programs written in different programming languages

I work in a team of people with experience in varied programming languages. Recently we were required to build a report when an order is placed in our system. Our ecommerce store runs on PHP and the ...
3
votes
1answer
79 views

How to use Messagequeing in a distributed web application

I am curious for solutions to this problem: Assume we are going to build an online-store. To better scale parts of the architecture, it is broken into independend subsystems. The typical scenario ...
3
votes
1answer
388 views

When does one NOT need to use queues in an application?

I've been developing an application using the Laravel Framework & have been exposed to the idea of queues for the first time. There are a multitude of tutorials & articles about using queues ...
3
votes
1answer
71 views

Simple messaging pattern to replace shared DB?

There are two modules A and B using the same DB. A imports B as a JAR. I read that messaging patterns can replace the tightly coupled solution of using a shared DB but I am not sure which message ...
3
votes
2answers
474 views

Is it worth it to use a message-queue framework in this case?

We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a ...
3
votes
2answers
1k views

How do I document a communication protocol on top of message queues and channels?

I'm working on a large project at the moment and I'm trying to document the communication protocol that sits on top of a message queue middleware. The documentation is intended for another ...
3
votes
1answer
827 views

Is ZeroMQ a good choice to make a Python app and a C# managed assembly work together?

I have a task that involves talking to a .NET-based API (namely AutoCAD) to retrieve data, send commands, and react to events. I want to separate the API operations and the proper program logic ...
3
votes
1answer
693 views

Push-Based Events in a Services Oriented Architecture

I have come to a point, in building a services oriented architecture (on top of Thrift), that I need to expose events and allow listeners. My initial thought was, "create an EventService" to handle ...
3
votes
2answers
324 views

How to solve an event-queue problem in a computer emulator in Java

I'm writing a low-level emulator (in Java) of the IBM 1620, a late-50s computer. By "low level," I mean that the lowest observable unit of work is the 20-microsecond "trigger" -- machine instructions ...
2
votes
2answers
382 views

Write own messaging system vs. utilize existing ones

We are trying to have our own startup, with a middleware application to glue small applications with enterprise legacy systems. for such middle-ware to function properly, we will need some sort of ...
2
votes
3answers
470 views

Design a Queuing Solution with Clustering and Multiple Consumers

It is a Design Problem which I am listing out here. I have different set of business operations that are carried out for different business entities. Operations: Operation A Operation B Operation ...
2
votes
1answer
107 views

What is an appropriate pattern for handling a queue of network events in C++?

I'm writing an application that sends and receives messages over the network and pushes them into a std::deque queue. and I'm looking for the appropriate programming pattern to handle all of the ...
2
votes
3answers
189 views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
2
votes
2answers
159 views

Building workflow driven applications [closed]

Quick primer: I am a happy developer on the Microsoft platform. Up to now I never had to design any applications which contained long running processes (with or without human interaction).In the past ...
2
votes
1answer
133 views

How do I handle long running third party calls from backing up my message queue?

Here is my exact scenario. I must make requests to a third party service The service takes ~15 seconds to respond It also has no webhooks or any call back mechanisms These requests are not made ...
2
votes
1answer
750 views

Is having a bi directional message queue a design smell?

I have an architecture with 1 backend server and multiple frontend servers. The frontend servers are connected in a bi-directional (web sockets) connection to clients so they can send messages to ...
2
votes
1answer
64 views

Using short-lived REST requests to obtain messages from a message queue for clients

I would like to put together a system where clients obtain messages from a REST API, but with the producer being able to use the semantics of a message queue: e.g. ordering of messages, a destination ...
2
votes
1answer
114 views

Should I parse an email template before or after placing it in the queue?

I have a web application that sends out transactional (triggered) emails to users. We've setup a message queue for the web application to write to when it needs to send out an email (best practices ...
2
votes
0answers
403 views

Architecture design for Java servlet that triggers work (for sending push notifications)

I'm designing a push notification server (for pushing notification via APNS and Google servers) and have come up with the following design for a Java based web application server. I would really like ...
2
votes
0answers
230 views

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed ...
1
vote
2answers
609 views

Access JSON object in different Languages

I have a Java code which is writing some data in string format to Messaging queue. Now any one can subscribe to this queue and fetch this data. Here the client fetching this data can be written in any ...
1
vote
1answer
143 views

How many Topics (queues) to practically use in an ESB

In all the Service Bus examples and tutorials I've seen, they cover publishing a single message to a single Topic that is picked up by a single listening Subscriber. Whilst this is good for a simple ...
1
vote
1answer
50 views

Sequencing distributed tasks with a distributed DB

Background I have two separate processes, WriteIt() and ReadIt(). One creates records, and the other processes the records in a DB cluster. Once WriteIt() creates a record, it queues a ReadIt() ...
1
vote
1answer
137 views

Options for repeated distributed task

Scenario My project has a need to periodically scan the database and notify administrators for particular issues. For example: Any errors in the logs since last update? Do values match expected ...
1
vote
2answers
2k views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...
1
vote
2answers
473 views

Distributed transactions with Kafka

I need to implement a transaction that spans over loosely coupled (SOA+MOM) components. When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
1
vote
3answers
236 views

Using message queue systems - forcing synchronous processing

I have a php process producer-a which places a job on RabbitMQ a - deferring processing to a third-party service. The 3rd party service completes processing a and places a response onto queue b. I ...
1
vote
1answer
158 views

Persist data downloaded by Celery workers

I'm working in a tool that downloads tweets from Twitter to process them later. For this purpose I'm using Celery with RabbitMQ, sending task with the keywords that must be tracked by the workers. My ...