The tag has no wiki summary.

learn more… | top users | synonyms

2
votes
2answers
147 views

Is it worth it to use a message-queue framework in this case?

We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a ...
0
votes
0answers
47 views

A layer of abstraction too far?

Using an enterprise service bus, a message translator pulls a message off the queue ready to be consumed by a service. The translator picks up the message in a canonical data model (CDM) form and ...
0
votes
0answers
21 views

How many Topics (queues) to practically use in an ESB

In all the Service Bus examples and tutorials I've seen, they cover publishing a single message to a single Topic that is picked up by a single listening Subscriber. Whilst this is good for a simple ...
0
votes
0answers
28 views

synchronized procesing of queued objects [migrated]

I have a queue, where new objects are inserted and processed. For processing I use singleton thread that can end and can be restarted. private Thread backgroundThread; private final ...
2
votes
0answers
74 views

Architecture design for Java servlet that triggers work (for sending push notifications)

I'm designing a push notification server (for pushing notification via APNS and Google servers) and have come up with the following design for a Java based web application server. I would really like ...
1
vote
1answer
74 views

Persist data downloaded by Celery workers

I'm working in a tool that downloads tweets from Twitter to process them later. For this purpose I'm using Celery with RabbitMQ, sending task with the keywords that must be tracked by the workers. My ...
1
vote
0answers
56 views

Improve communication between controller and trackers in a Twitter fetcher tool using RabbitMQ or Apache Flume

I've been working for a time with some researches developing a tool to fetch tweets from Twitter and process them in some way. The first prototype "worked" but became a pain as we used sockets to ...
-1
votes
1answer
183 views

Will a MQ topic persist messages when one of the consumers is down [closed]

If I have a websphere MQ topic and 3 listeners have subscribed to the Topic. One listener is down when a message comes in the Topic. Other listeners consume the message. When the listener comes back ...
0
votes
0answers
65 views

Distributed computing - web services vs message queues

I Need to decide on the architecture for a distributed computing implementation. In the server, there is a database, which holds many mathematical problems, that need to be solved by crunching ...
0
votes
2answers
170 views

Storm topology to handle “dating website”-like workloads

suppose im writing a dating website, similar to okcupid. there are profiles, and i need to compute the (N^2) "match" table - given every 2 profiles whats the match between them? I was thinking this ...
1
vote
0answers
34 views

Job queue dispatching based on resource constraints

We have a job-queue shaped hole in our architecture, but it isn't immediately obvious that there's a piece of existing software to fill it. There may be a reason for this, or there might not, so let ...
4
votes
3answers
97 views

Web Service and Message Queue

We're looking at creating a web services/REST API layer that will be consumed by web and mobile clients. To make the solution more robust I was considering putting the commands from PUT, POST, and ...
1
vote
1answer
32 views

Sequencing distributed tasks with a distributed DB

Background I have two separate processes, WriteIt() and ReadIt(). One creates records, and the other processes the records in a DB cluster. Once WriteIt() creates a record, it queues a ReadIt() ...
1
vote
1answer
61 views

Options for repeated distributed task

Scenario My project has a need to periodically scan the database and notify administrators for particular issues. For example: Any errors in the logs since last update? Do values match expected ...
1
vote
3answers
95 views

Using message queue systems - forcing synchronous processing

I have a php process producer-a which places a job on RabbitMQ a - deferring processing to a third-party service. The 3rd party service completes processing a and places a response onto queue b. I ...
0
votes
1answer
168 views

Is having a bi directional message queue a design smell?

I have an architecture with 1 backend server and multiple frontend servers. The frontend servers are connected in a bi-directional (web sockets) connection to clients so they can send messages to ...
2
votes
3answers
225 views

Design a Queuing Solution with Clustering and Multiple Consumers

It is a Design Problem which I am listing out here. I have different set of business operations that are carried out for different business entities. Operations: Operation A Operation B Operation ...
2
votes
3answers
1k views

UML: Queue processor in a sequence diagram

Which is the appropriate way of representing a queue processor in a UML sequence diagram? I want to represent in the same diagram two systems that are loosely coupled through a queue. The first ...
3
votes
1answer
514 views

Push-Based Events in a Services Oriented Architecture

I have come to a point, in building a services oriented architecture (on top of Thrift), that I need to expose events and allow listeners. My initial thought was, "create an EventService" to handle ...
1
vote
1answer
89 views

More than 20 Records Inserted in a Span of 1 Sec

Well, this is not specific to any programming languages. I came across this situation in Android, iOS, PHP and .NET. The problem is many times I come across a situation while building IoT apps, where ...
1
vote
2answers
897 views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...
0
votes
0answers
117 views

When to use a message queue as a web service database?

Does it make sense to implement a message queue system such as RabbitMQ or Kafka for the purpose of storing data in a queue? For example, this could be done in order to handle things such as ...
0
votes
0answers
21 views

Logging and analytics from multiople processes

I am setting up a system on OpenStack comprising of several independently running processes and need a logging and analytics process running which will receive very frequent input from the running ...
0
votes
1answer
68 views

Should I have a queue with SMS requests to picked up by a separate process or connect directly to the API?

I created an email queue where I put the requests for emails to be sent. If the SMTP server is unavailable, the requests are not lost. When the SMTP server gets back online, the background service ...
3
votes
0answers
54 views

How to use Messagequeing in a distributed web application

I am curious for solutions to this problem: Assume we are going to build an online-store. To better scale parts of the architecture, it is broken into independend subsystems. The typical scenario ...
2
votes
1answer
182 views

Passing notifications between programs written in different programming languages

I work in a team of people with experience in varied programming languages. Recently we were required to build a report when an order is placed in our system. Our ecommerce store runs on PHP and the ...
0
votes
1answer
103 views

Locking database records

What I have right now: A table with lot of records. About 200 users can select a record according to some rules based on the data in each record. There is a high chance that 2 or more users will ...
4
votes
4answers
605 views

Architecting Python application consisting of many small scripts

I am building an application which, at the moment, consists of many small Python scripts. Each Python script processes items from one Amazon SQS queue. Emails come into an initial queue and are ...
3
votes
1answer
355 views

How to get JMS to front end

I currently have a web application and I would like to add a messaging feature to it. In order to do that, I use JMS(actually OpenMQ, the implementation provided with Glassfish 3). The problem is that ...
4
votes
1answer
84 views

How to Recover from Inconsistent Job State without Database Polling

I'm working on scaling an application which is currently polling a mySQL database to send async jobs to a background processing system. A state machine is being used to keep track of the entities ...
2
votes
1answer
2k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
10
votes
1answer
3k views

Why database as queue so bad? [closed]

I've just read this article, and I'm confused. Let's imagine 1 webapp and 1 distinct application acting as "worker", both sharing the same database. Oh, I said "sharing"..but what does the article ...
1
vote
1answer
98 views

Estimating time remaining in a queue with multiple workers

I've been trying to figure out a way of calculating the estimated time remaining for an item in a queue when there are multiple workers processing items from the single queue. A client submits a job ...
0
votes
2answers
265 views

Create and delay task for 24 hours [closed]

I'm trying to come up with a sane architecture to delay a task for 24 hours. So far I've thought of: unix cron job, but this will not be able to take the load we give it, since we will have many ...
28
votes
2answers
15k views

When to use Advanced Message Queuing Protocol like RabbitMQ? [closed]

Can somebody explain me in what use-cases I should consider using AMQP like e.g. RabbitMQ? What are the pros and cons?
2
votes
0answers
157 views

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed ...
4
votes
2answers
1k views

Where does Windows Workflow Foundation fit into the messaging oriented middleware architecture?

I've been trying to find information online about where WWF 4.5 fits with in an architecture based around an enterprise message queue using central broker(s) (RabbitMQ in particular). It seems like ...
3
votes
1answer
334 views

When does one NOT need to use queues in an application?

I've been developing an application using the Laravel Framework & have been exposed to the idea of queues for the first time. There are a multitude of tutorials & articles about using queues ...
3
votes
1answer
632 views

Is ZeroMQ a good choice to make a Python app and a C# managed assembly work together?

I have a task that involves talking to a .NET-based API (namely AutoCAD) to retrieve data, send commands, and react to events. I want to separate the API operations and the proper program logic ...
3
votes
2answers
4k views

REST or a message queue in a multi-tier heterogeneous system?

I'm designing a REST API for a three-tier system like: Client application -> Front-end API cloud server -> user's home API server (Home). Home is a home device, and is supposed to maintain connection ...
1
vote
0answers
429 views

Apply filter in Kafka Queue

Is it possible to apply a filter on the Kafka stream of topic? What I want to do is filter out messages while consuming and then run some custom logic on top of it. It is a sort of thing that Storm ...
1
vote
2answers
392 views

Access JSON object in different Languages

I have a Java code which is writing some data in string format to Messaging queue. Now any one can subscribe to this queue and fetch this data. Here the client fetching this data can be written in any ...
19
votes
1answer
9k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
9
votes
1answer
3k views

Designing an scalable message queue architecture

I have recently started learning the nuances of scalable and enterprise computer architecture, and one of the central components is a messaging queue. In order to learn the most I can from any ...
8
votes
2answers
488 views

Product agnostic book on Message Queues? [closed]

Are there any recommendations for book(s) that are largely agnostic (with examples of MQ implementations is fine) on vendor but go to great details on the architecture, management, nomenclature, ...
2
votes
1answer
107 views

Should I parse an email template before or after placing it in the queue?

I have a web application that sends out transactional (triggered) emails to users. We've setup a message queue for the web application to write to when it needs to send out an email (best practices ...
0
votes
0answers
839 views

MSMQ Receive Timeout Even With Items in Queue

Tried posting this on main StackOverflow site, but no replies in 5 days. I am having ongoing problems trying to implement a very simple MSMQ solution: 2 PCs (not servers) running Win7 and Win8 ...
5
votes
3answers
2k views

NoSQL Document Database as a Message Queue

I am considering using a NoSQL Document database as a messaging queue. Here is why: I want a client to post a message (some serialized object) to the server and not have to wait for a synchronous ...
3
votes
2answers
938 views

How do I document a communication protocol on top of message queues and channels?

I'm working on a large project at the moment and I'm trying to document the communication protocol that sits on top of a message queue middleware. The documentation is intended for another ...
0
votes
1answer
595 views

Scheduling Thousands of *Recurring* Jobs

I'm looking for general approaches and ideally libraries/tools for scheduling many thousands of recurring jobs. A simple example is a feed aggregator, where I'd want to repeatedly poll thousands of ...