Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Introduction

Preamble

I was hesitant to post this, since it's an application whose intricate details are defined elsewhere, and answers may not be helpful to others. Within the past few weeks (I was actually going to write a blog post about this after I finished) I've discovered that the barrier I'm encountering is one that's actually quite common for newer developers.

This question is not so much about a specific thing as it is about piecing those things together.

I've searched the internet far and wide, and found many tutorials on how to create applications that are kind of similar to what I'm looking for. I've also looked at hiring another, more experienced, developer to help me along, but all I've gotten are unqualified candidates that don't have the experience necessary and won't take care of the client or project like I will. I'd rather have the project never transpire than to release a solution that is half-baked.

I've asked professors at my school, but they've not turned up answers to my question. I'm an experienced developer, and I've written many applications that are -- very abstractly -- close to what I'm doing, but my experiences from those applications aren't giving me enough leverage to solve this particular problem.

I just hope that posting this article isn't a mistake for me to write.

Project Description

I have a project I'm working on for a client that is a rewrite of an application, originally written in Foxpro 2.6 by someone before me, that performs some analysis (which, sadly, I'm not allowed to disclose as per of my employment contract) on financial data.

One day, after a long talk between the client and I -- where he intimately described his frustrations with all the bugs I've been hacking out of this code for 6 months now -- he told me to just rewrite it and gave me a month to write a good 1/8 of this 65k LOC Foxpro monstrosity. this 65k line of code foxpro monstrosity. It'll take me a good 3 - 6 months to rewrite this software (I know things the original programmer did not, like inheritance) going as I am right now, but I'm quickly discovering that I'm going to need to use databases.

Prior to this contract I didn't even know about foxpro, and so I've had to learn foxpro on the fly, write procedures and make modifications to the database. I've actually come to like it, and this project would be rewritten in Foxpro if it were still a supported language, because over the past few months, I've come to like the features of Foxpro that make it so easy to develop data-driven applications.

I once perfomed an experiment, comparing C# to Foxpro. What took me 45 minutes in C# took me two in Foxpro, and I knew C# prior to Foxpro.

I was hoping to leverage the power of C#, but it intimidates me that in foxpro, you can have one line of code and be using a database.

Prior to this, I have never written any serious database development from scratch. All the applications that I've written are in a different league. They are either completely data-naive or data-naive enough that I can get away with not using a database through serialization or by designing algorithms that work with the data in a manner that is stateless, so there is no need to worry about databases.

I've come to realize, very quickly, that serialization and my efficacy with data structures has been my crutch all these years that's prevented me from adventuring into databases, and has consequently hindered my success in real-world programming.

Sure, I've written some database stuff in Perl and Python, and I've done forms and worked with relational databases and tables, I'm a wizard in Access and Excel (seriously) and can do just about anything, but it just feels unnatural writing SQL code in another language...

I don't mind writing SQL, and I don't It's that bridge between the database and the program code that drives me absolutely bonkers. I hope I'm not the only one to think this, but it bothers me that I have to create statements like the following

string sSql =  "SELECT * from tablename"

When there's really no reason for that kind of unchecked language binding between two languages and two API's. Don't get my wrong, SQL is great, but I don't like the idea that, when executing commands on a SQL database, that one must intermix database and application software, and there's no database independence, which means that different versions of different databases can break code. This isn't very nice.

The nicest thing about Foxpro is the cohesiveness between programming language and database. It's so easy, and Foxpro makes it easy, because the tool just fits the task. I can see why so many developers have created a career with this language, because it lowered the barrier of entry to data-driven applications that so many businesses need. It was wonderful.

For my purposes today, with the demands and need for community support, extensibility, and language features, Foxpro isn't a solution that I feel would be the right tool for the job.

I'm also worried about working too heavy with the database, because I've seen data-driven .NET applications have issues with database caches, running out of memory, and objects in the database not being collected. (Memory leaks)

And OH the queries.


Which one, how, and why?

There are a plethora of different ways that a database can be setup, I think I counted 5 or 6 different kinds of database applications alone that I can chose from. That is a great mountain for me to climb when I don't even know where to begin when it comes to writing data-driven applications.

The problem isn't that I don't know SQL or that I don't know C#. I know both and have worked with both extensively. It's making them work together that's the problem, and it's something I've never done in C# before.

Reports

The client likes paper. The data needs to be printed out in a format that is extensible, layered, and easy to use. I have never done reporting before, and so this is a bit of a problem. From the data source comes crystal reports, and so there's a dependency on the database, from what I understand.

Code reuse

A large part of the design decision that I've gone through so far is to break the task of writing a piece of this software into routines and modular DLL's and so forth such that much of the code can be reused.

For example, when I setup this database, I want to be able to reuse the same database code over and over again.

I also want to make sure that when the day comes that another developer is here, that he/she will be able to pick up just where I left off.

The quicker I develop these applications, the better off I am.

Tasks & Goals

In my project, I need to write routines that apply algorithms and look for predefined patterns in financial data. Additionally, I need to simulate trading based on predefined algorithms and data. Then I need to prepare reports on that data.

Additionally, I need to have a way to change the code base for this application quickly and effectively, without hacking together some band-aid solution for a problem that really needs a trauma ward.

Special Considerations

The solution must be fast, run quickly on existing hardware, and not be too much of a pain to maintain and write. I understand that anything I write I'm married to -- I'm responsible for the things that I write because my reputation and livelihood is dependent on it.

Do I really need a database?

What about performance? Performance was such a big issue that I hand wrote a data structure that is capable of performing 2 billion operations, using a total of 4 gigs of memory in under 1/4 of a second using the standard core two duo processor. I could not find a similar, pre-written data structure in C# to perform this task.

What setup do I use in terms of database?

What about reporting? I'd prefer to have PDF's generated, but I'd like to be able to visually sketch those reports and then just have a ReportFactory of some sort, that when I pass some variables in, it just does that data.

About Me

I'm a lone developer for a small business in this area. This is the first time I've done this and I've never had the breadth and depth of my knowledge tested.

I'm incredibly frustrated with this project because I feel incredibly overwhelmed with the task at hand.

I'm looking for that entry level point where I can draw a line and say "this is what I need to do"

Conclusion

I may have not been clear enough on my post. I'm still new to this whole thing, and I've been doing my best to contribute back to the community that I've leached so much knowledge from. I'd be glad to edit my post and add more information if possible.

I'm looking for a big-picture solution or design process that helps me get off the ground in this world of data-driven applications, because I have a feeling that it's going to be concentric to my entire career as a programmer for some time.

Specifically, if you didn't get it from the rest of the post (I may not have been clear enough) I really need some guidance as to where to go in terms of the design decisions for this project. Some things that'll be useful will be a pro/con list for the different kinds of database projects available in VS2010. I've tried, but generating that list has been as hard as solving the problem itself...

If you could walk a developer writing a data-driven application for the first time in C#, how would you do that? Where would you point them to?

share|improve this question
    
"It's that bridge between the database and the program code that drives me absolutely bonkers." I'm sure there are thousands of developers that have same thoughts. And I'm sure there are thousands of ideas how to overcome this problem. ORM being one. –  Euphoric Sep 9 '12 at 20:59
    
Visual FoxPro is a supported language until 2015, but you're right in taking the plunge now to something more modern and possibly cross-platform. You could look at Visual Studio Lightswitch perhaps, although it might be too much toward the MS Access level of sophistication to suit your needs. –  Alan B Sep 10 '12 at 7:46

2 Answers 2

up vote 5 down vote accepted

Phew! That's a hum-dinger of a question.

One suggestion I'd make is that, should you decide to move to Microsoft SQL Server as your database, then you should really consider using the SO-used library called Dapper.

It's a really easy-to-use interface between C# code and SQL Server, and should at least address your statement:

It's that bridge between the database and the program code that drives me absolutely bonkers.

share|improve this answer
    
Your answer was really useful and Dapper was exactly what I was looking for. Thank you so much. –  alvonellos Sep 9 '12 at 23:40

The main thing that stands out for me in your entire post is that you see C# and SQL as two completely different languages that can't (easily) work together. This is understandable when you've been using a language which integrates the two and you see examples all over the Internet where people are hard coding SQL in their application code.

The easiest thing you could do that would help you overcome this barrier is to look into using an ORM such as Fluent NHibernate. This will allow you to treat your database as just another layer of code. You create classes to represent your database entities and a mapping class that tells the code how to map each column (eg: "the PersonID column is an int and is an incrementing value). This means that you are then always dealing with nice objects in your code.

The next step would be to look at introducing the repository pattern which you get for free with NHibernate (the ISession property is what you want to look in to). This prevents you having to write SQL in your code as the framework can take lambda expressions and make use of LINQ to (under the hood) generate the SQL that you would otherwise have to hand write.

The biggest thing I can suggest is to make use of the brilliant frameworks that people have written to make our lives easier.

Good luck!

Edit: Apologies for any typos, I typed this on my iPhone while waiting for a flight at the airport!

share|improve this answer
    
Novel post, Stu. Thanks for taking the time to write this. I really like linq, and I've been using that with lambda expressions in my code quite well. I've even done some experimentation as to whether or not linq or plinq is appropriate for my code. I just love the idea that a layer of abstraction is there that isolates out SQL from the application code. It makes a lot of sense. I'll take a look at NHibernate. Thanks for your reply. –  alvonellos Sep 10 '12 at 9:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.