Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm an amateur programmer. I'd like to develop a software application (like Tradestation), to analyse real-time market data. Please teach me if the following approach is correct, ie the procedures, knowledge or software needed etc:

  1. Use a DB to read the real-time feed from data provider: what should be the right DB to use? I know it should be a time serious one. Can I use SQL, Mysql, or others? What database can receive real-time data feed? Do I need to configure the DB to do this?

  2. If the real-time data is in ASCII form, how can it be converted to those that can be read by the DB and my application? Should I have to write codes or just use some add-ins? What kind of add-in are needed?

  3. How should I code the program to retrieve the changing data from the DB so that the analysis software screen data can also change asynchronously? (like the RTD in excel)

  4. Which aspects of programming do I need to learn to develop the above?

Are there web resources/ books I can refer to for more information?

share|improve this question

closed as too broad by Jim G., gnat, GlenH7, MichaelT, Bart van Ingen Schenau Jun 23 '14 at 12:38

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

Good quality real-time data costs A LOT, particularly the exotic stuff; sometimes it is cost-prohibitive for a small start-up. I believe that your bottle-neck is economical, not technical. Think broader and keep in mind that financial services is a crowded market. It is also a connections-based business. Most deals are still signed while inside of strip clubs, like it or not. Little can be done without prior experience and lots of connections. You just might have to scrape Google finance or Yahoo finance instead. – Job Jun 11 '11 at 17:20
One more thing: a very common error that people make is they stick real-time data into a database. Why??? What advantages (and some associated overhead) of ACID do you get when storing data which will no longer be relevant 1 second from now? – Job Jun 11 '11 at 17:38
@job - Some people (most I thought) chart stock prices, so storing the data as it arrives will ensure you have that data when you chart it. Charting only the current price is rather useless. :-) – John MacIntyre Jun 11 '11 at 19:54
up vote 0 down vote accepted

Per se, a database stores data, but it doesn't read it. In practice, most database systems come with lots of tools, so maybe one of those tools allow you to read the feed; some database systems like Oracle have powerful built-in languages so you could place the reading program as a stored procedure in the database.

But I don't think that this is the right approach to go. You should better use a language like Java, C# etc. to write a program that fetches and parses the realtime feed and uses (N)Hibernate to write it into the database; other parts of the program could do the calculation and yet other parts create the screen output. A typical beginners mistake would be to create a big god class that does all that stuff; good programmers avoid that, they separate different concerns into different classes. In your case, the model-view-controller pattern seems like a good start.

share|improve this answer
Thank you for your advice. I'll start by learning the information about the data source like the format, structure, the ports used etc. Then I'll try to learn how to capture, parse, filter the real-time feed. There's a long way to go ! – Sam Jun 13 '11 at 1:00
Thanks. I think data capture is one of the crucial area I have to solve. If I don't use software outside, I'll try to write my own to get the data streams/ feed. I know ticker plant is used for this. Such software is usually tailor-made and there seems to be no such products free of charge. But I think it's only for large companies. It's a software one can write, though not easy. – Sam Jun 13 '11 at 6:05

research event-stream processing (ESP) and complex event processing (CEP), and know that if you have to hit the disk in a real-time stream, you're through

share|improve this answer

In case you already did not know, there are 100s of developers financial services providers hire to develop something like this, so keep your expectations realistic.

Typically exchanges like NYSE provide API that helps you check what the current value of the security is. You may also use the feed from your broker -- typically XML.

You need to understand how the feeds work, the format should be well documented.

You could use any programming language of your choice, typically C or C++ are used for real time, but C# is gaining ground. Haven't seen much Java here.

You would need a good database -- so learning SQL is required.

share|improve this answer
There's good amounts of Java on the other side of the Atlantic. Some lisp, too. – Denis de Bernardy Jun 11 '11 at 17:34

I am also working on a similar project to learn the basics of programming. To answer your question.

  1. You can use mySQL to store certain trading patterns or interesting observation. There is no point to store data that you don't need. Again, real live data are super expensive. Your best bet is to data scrap off yahoo and there are posts stackoverflow that shows you how to do it. You can look up one of my question and someone showed me how to do it in PYTHON.

  2. My data are in Unicode and parsing it to float is possible. Ask a more detailed question for that.

  3. Regarding how to deal with the dynamically changing data, it all depends on your trading method. For example, when I was a professional day trader, I wrote a simple program that just gives me an alert when I see a current bid price is higher than the current ask price. Another simple strategy can involve a trending stock crossing the high or the low. You can set variables that save as: open, high, low, moving average, %retracement...

share|improve this answer

If I were you, I would focus on real-time data retrieval last.

I'd probably approach it in the following steps:

  1. Basic charting functionality (because it's most interesting, harded code data for now)
  2. Database & data access to populate the charts
  3. Data retrieval & storage to the database from the data provider
  4. Updating the charts in real time based on data provider input events
  5. Revise & refactor for the rest of your life
share|improve this answer
Your advice is useful. Thanks. I think the data capture is the part I have to deal with besides hardcoding the chart data. – Sam Jun 13 '11 at 5:58

The database is the least important decision. MS SQL Server, PostgreSQL or MySQL will work fine.

Here's the important decisions:

  1. Platform - Is this going to run on your computer or on a server and delivered through a web page?
  2. Data provider. This is the hardest part to research and solve. I have had good experiences with MB Trading. You can find more answers on Quant SE or perhaps Stack Overflow. You will need to create a prototype to test the API.
  3. Language - From my own experience doing something similar, I suggest C# if it will run on your computer. Not C++ or C, as they are a lot more work with little benefit for this application. Java might be a good choice. Rails might be good if it running on a server.

You may be able to query the data provider once every second, and update the screen. This will depend on how the API works.

Here's some advice. Working with a data provider API and drawing charts is not very easy for a beginner. You will probably need to learn quite a bit about the language you are working in during the process. You can find great resources on Amazon, and ask here or on Stack Overflow for the best.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.