Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I need to write an application which consumes a forum's content via a REST API and stores threads and posts. The application will act as a bridge layer between the forum and write data to a third application periodically - as close to 'real-time'as possible. The platform is PHP 5.3 / MySQL and probably Symfony with the Zend_Rest client.

My question is what would be an appropriate / performant architecture be for the bridge layer? I imagine I will need to do an initial import of the forum data which will be slow (may take hours). The bridge application will also have a front-end for selectively adding the forum messages to the third application and adding further meta data e.g sentiment (was the message positive or negative in tone). I realise the data import / export could be done with procedural scripts and cron jobs but am wondering if there is a better way.

Many thanks,

share|improve this question
1  
Not really many choices. A web client is a web client; the HTTP rules are simple and strict. Why don't you build it, measure the performance, and then ask specific questions on Stack Overflow? –  S.Lott Apr 6 '11 at 10:13
    
I'd like to second S.Lott's sentiment. –  dirk Apr 11 '11 at 3:29
add comment

1 Answer

There's a variety of challenges when it comes to synchronizing API's like this. Although there isn't a silver bullet that I can offer that will make everything work; I can offer a few things to keep in mind.

Take advantage of any type of temporal meta-data available, Last-Updated and similar are your best friend when dealing with the time-oriented nature of forums. Seeing as it is REST, the bridge layer will likely need to run as a cron job. Since you won't be able to register any type of callback to receive new forum posts as they come in, everything will have to be after the fact and rely solely on temporal meta-data. This type of stuff is tricky, and sadly I can't offer a magic solution.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.