Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

There is an endless data stream of XML messages (and "heartbeats"), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel.

I'm still pondering. What is the best/most elegant solution to process the XML messages without losing any data, without redundance and with a (nearly) constant processing time? A never-ending process/script? Writing the stream in file(s) and processing it/them periodically step by step? Or something completely different?

The messages usually come every few seconds. Sometimes every second. Sometimes maybe every 10 seconds. It differs but not a lot. One XML message within the stream contains 45 rows. The messages should be stored afterwards.

Note: The concrete structure of the XML messages and the infrastructure of the participating systems are negligible in my opinion.

share|improve this question
1  
You might want to look at some XMPP implementations, because XMPP is basically the same: an endless XML stream. –  Joachim Sauer Mar 21 '13 at 14:05
1  
Sounds like the Observer/Listener pattern might be helpful. More information about your problem might help in suggesting a solution. How quickly/how often is the data coming in? How large are the xml messages? Do you need to store the message content somewhere? What do you do if there's an exception (the xml is incorrect)? –  Asaf Mar 21 '13 at 14:08
    
@Asaf: I've edited my question to share you a few more information –  derphil Mar 21 '13 at 15:10

1 Answer 1

up vote 0 down vote accepted

Event stream based is your best bet! in C I will use Expat. In Java, I will use SAX parser.

share|improve this answer
2  
Short answers like this are always better received if they are accompanied by a paragraph or two explaining why you are recommending this as a solution. –  Robert Harvey Mar 21 '13 at 15:58
    
well the reason why is because both are stream based –  Armando Mar 21 '13 at 21:49
1  
That explains why you didn't pick something else that's not stream-based, but it doesn't really say why you are picking Expat specifically (what features or characteristics make it ideal), or why an Event-based stream is better in this scenario. –  Robert Harvey Mar 21 '13 at 21:53
    
@Armando: Ok, I've took a look at SAX. So, just to ensure, that this is a good solution: Do you have any experience of using endless input streams (via telnet) with the the Java SAX parser? Or do you only suggest SAX because it's event based? –  derphil Mar 22 '13 at 9:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.