The tag has no wiki summary.

learn more… | top users | synonyms

2
votes
0answers
43 views

How to use Messagequeing in a distributed web application

I am curious for solutions to this problem: Assume we are going to build an online-store. To better scale parts of the architecture, it is broken into independend subsystems. The typical scenario ...
2
votes
0answers
140 views

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed ...
1
vote
0answers
24 views

Job queue dispatching based on resource constraints

We have a job-queue shaped hole in our architecture, but it isn't immediately obvious that there's a piece of existing software to fill it. There may be a reason for this, or there might not, so let ...
1
vote
0answers
44 views

Persist data downloaded by Celery workers

I'm working in a tool that downloads tweets from Twitter to process them later. For this purpose I'm using Celery with RabbitMQ, sending task with the keywords that must be tracked by the workers. My ...
1
vote
0answers
38 views

Improve communication between controller and trackers in a Twitter fetcher tool using RabbitMQ or Apache Flume

I've been working for a time with some researches developing a tool to fetch tweets from Twitter and process them in some way. The first prototype "worked" but became a pain as we used sockets to ...
1
vote
0answers
360 views

Apply filter in Kafka Queue

Is it possible to apply a filter on the Kafka stream of topic? What I want to do is filter out messages while consuming and then run some custom logic on top of it. It is a sort of thing that Storm ...
0
votes
0answers
37 views

Distributed computing - web services vs message queues

I Need to decide on the architecture for a distributed computing implementation. In the server, there is a database, which holds many mathematical problems, that need to be solved by crunching ...
0
votes
0answers
100 views

When to use a message queue as a web service database?

Does it make sense to implement a message queue system such as RabbitMQ or Kafka for the purpose of storing data in a queue? For example, this could be done in order to handle things such as ...
0
votes
0answers
21 views

Logging and analytics from multiople processes

I am setting up a system on OpenStack comprising of several independently running processes and need a logging and analytics process running which will receive very frequent input from the running ...
0
votes
0answers
97 views

Message Queue: Which one is the best scenario?

I write a web crawler. The crawler has 2 steps: get a html page then parse the page I want to use message queue to improve performance and throughput. I think 2 scenarios: scenario 1: ...
0
votes
0answers
742 views

MSMQ Receive Timeout Even With Items in Queue

Tried posting this on main StackOverflow site, but no replies in 5 days. I am having ongoing problems trying to implement a very simple MSMQ solution: 2 PCs (not servers) running Win7 and Win8 ...