Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

My new boss has been working on this project for many years. I've only been here a few weeks, but I am not sure it's possible. He would like to design a system that is "100% data driven".

So if we put in enough data, we can define and generate any application. I've managed to at least get him to concede some things like users, or apps should have predefined values, but he likes the concept of the structure of the system, the user interface and the logic all being stored as data.

There are some demos of simple things and he's basically rediscovered some simple ideas of object oriented programming and your basic template systems, but I think overall that this goal might be actually impossible.

I don't know how you can define logic using data without the system becoming so complex that you are doing actual programming anyway.

I think theoretically it isn't because the thing that interprets the data ends up needing to become turing complete to describe the application so you've just shifted the problem one level higher to no net benefit.

Is such a 100% Data Driven Application possible?

share|improve this question
3  
Only if you write your own programming language. If you really have the need to write a lot of applications that are that similar, you may need better libraries, better architecture, or in an extreme case a Domain Specific Language (DSL). –  Michael K Mar 7 at 19:21
5  
I think you need to define what you mean by 'data driven' in a more specific way. –  GrandmasterB Mar 7 at 19:22
7  
In some languages, like Lisp, there's no clear line between code and data. That might result in database tables or columns that contain instructions to act on the data that lives alongside it, but I'm not sure if that's cheating. –  Rob Y Mar 7 at 19:22
18  
Of course you can do it! The data is stored as Java source files on the file system. We just compile & deploy and there you go. 100% flexibility, 100% data-driven. –  Jeremy Stein Mar 7 at 21:31
6  
@JeremyStein beat me to it. I was going to say my data was stored in Subversion, and changes to my 'configuration' are applied through the continuous integration system, and other deployment processes. –  Mr.Mindor Mar 7 at 22:21

7 Answers 7

Your boss should read this piece: Bad Carma: The "Vision" project, a cautionary tale about inner platform effect or second system effect.

Abstract

Those of us who work in Information Technology (IT) have all been on a project where something important is just not right. We know it, most everyone knows it, but nobody is quite able to put his or her finger on the problem in a convincing way.

This story is about such an IT project, the most spectacular failure I have ever experienced. It resulted in the complete dismissal of a medium-sized IT department, and eventually led to the destruction of a growing company in a growing industry. The company, which we'll call "Upstart," was a successful and profitable subscription television business.

The project occurred in the early 1990s, and it was a custom-built order-entry and customer-service application, closely resembling what is now referred to as Customer-Relationship Management or CRM. The core functionality of the system included:

  • Order entry and inventory
  • Customer service, help desk
  • General ledger, accounts receivable, billing, and accounts payable

The application was called "Vision" and the name was both its officially stated promise for Upstart as well as a self-aggrandizing nod to its architect. The application was innovative, in that it was built to be flexible enough to accommodate any future changes to the business. Not just any foreseeable future changes to the business, but absolutely any changes to the business, in any form. It was quite a remarkable claim, but Vision was intended to be the last application ever built. It achieved this utter flexibility by being completely data-driven, providing limitless abstraction, and using object-oriented programming techniques that were cutting-edge at the time.

Like many such projects that set out to create a mission-critical application, the development effort spanned two years, about a year longer than originally projected. But that was acceptable, because this was the application that would last forever, adapting to any future requirements, providing unlimited Return On Investment (ROI). When the application finally went "live," almost everybody in the company had invested so much in it that literally the fate of the company hinged on its success.

However, in the event of total project malfunction, mission-critical applications running the core business of multinational corporations are not permitted the luxury of the type of fast flameout demonstrated by thousands of "dot-com" companies in the era of the Internet bubble. Within a month of Vision going "live," it was apparent to all but those most heavily vested in its construction that it was a failure.

See Also

http://en.wikipedia.org/wiki/Inner-platform_effect

share|improve this answer
3  
+1 inner-platform effect. I think this TDWTF sums it up nicely: thedailywtf.com/Articles/The_Inner-Platform_Effect.aspx –  Snowman Mar 7 at 19:26
4  
It is funny when people fail to see the cost of writing a little bit of code is far less than building an entire platform. –  brianfeucht Mar 7 at 20:14
9  
@brianfeucht: The idea of the infinitely configurable platform is seductive. –  Robert Harvey Mar 7 at 20:34
1  
The inner platform effect reminds me of Google libraries such as Guava, where instead of using if statements, the code is filled with tons of Predicate instances. Thats just horrible. –  lukasz1985 Mar 7 at 20:57
3  
@RobertHarvey and fun to build. Just as long as I do not have to support the end users ;) –  brianfeucht Mar 7 at 22:24

The answer is yes, it is possible to create a fully data-driven system and yes, it's usually a really bad idea.

A fully data driven program is one in which all logic and configuration is handled by values stored in such a way that in another context they would be regarded as data. There were many 4GL products produced in the 1980s that provided the capability to generate reports, forms, tables and logic using data items entered into a multiplicity of forms, stored in tables and accessible through reports. I used to refer to such systems as "paint by numbers" but I see it has now come to be known as the "inner system" effect. Good name.

People who create these systems are trying (whether they know it or not) to create a new programming language. Since they do not have the skills, they do it badly. From the viewpoint of the JVM/CLR, a compiled Java/C# program is simply data. In this case it has been done well. In either case, programmers are needed to use the language, whatever it is.

There is one specific way to make this work, that I know of. You build the skeleton of each of the components you need: form, report, table, etc. You provide a mechanism to configure various parts of these components by setting data items. For a chosen set of features you make the decisions and freeze them into the system, and specifically deny the ability to configure those features.

You also implement a language that has the ability to code logical operations. My recommendation is to use an existing language such as lua or maybe Python. You embed pieces of this code wherever logical operations are needed.

By doing this you substantially reduce the amount of writing required to implement each form, report, table and so on. The system seems to be data-driven, but only to a point.

At this point you have just implemented a new 4GL. If you happen do do this successfully, please let me know. Most people fail dismally. I will be the first to congratulate you on your achievement.

share|improve this answer
2  
Nice writeup. S.A.P. (the ERP system) is the classical example of such a system. You don't program in it, you "configure" it. Which is so bloody complex to get anything significant done, that it has created an entire consultancy industry around it. –  Tonny Mar 8 at 16:11
    
@Tonny: Thanks. I have no firsthand experience with SAP, but I understand that SAP/R3 and ABAP come close to this description, and are a major generator of war-stories: what wrong and by how many times the budget blew out. Still makes the company heaps of money. –  david.pfx Mar 8 at 23:33
    
As someone who's had first hand experience of S.A.P i'd just like to comment... (Can some one show me the way to the asylum now please?) –  shawty Jul 9 at 21:08

I think you're basically correct. A language runtime is already a fully flexible data-driven system. It takes one piece of data (the program) and uses it to determine how it should act on other data. It might even have a multi-user scheme to store code for re-use by other programs (ranging from an include path to proper install management).

A "scripting language", roughly speaking, is a language runtime where this code input is human-readable. A compiler places an extra step between the user and the runtime. "Joke" languages like Malbolge and APL needn't be human-readable in any form. But it's all the same thing at one level, and anyway human-readable does not mean that all potential users have the skills to read or write it, or can be expected to develop them.

There are good reasons why you don't normally expose a language runtime directly to end-users. The main one being that removing flexibility increases convenience.

If I want to type a SO post, I just want to type it. I'm perfectly capable of instead writing a C++ program to output it, but I would not use a web browser that exposed a C++ program editor instead of a regular text box. People who don't know C++ not only wouldn't use the browser, they couldn't.

If I want to configure certain business parameters then I don't necessarily want to do that using a Turing-complete specification language, and even if I did this is probably not distinguishable from "hard-coding" those same business parameters in any other programming language. You still need to consider whether what you're writing means what you want it to mean. You still need to test that changes are correct. That is, you still need programming skills for any tasks that are non-trivial and not anticipated by someone who does have programming skills who prepared a specialised sub-system ("application") for you to configure ("use").

So if you're about to embark on a 100% data-driven system, that can do anything given the right data, you have two questions to ask yourself:

  1. Are we in the business of inventing programming languages, or should we be?
  2. Will our new programming language be better (for our purposes) than the ones we already have and will we support and develop it as it needs?

Sometimes the answers are yes, and you write a domain-specific language of some kind. Or even a real general-purpose programming language if you're Sun/Microsoft/Stroustrup/van Rossum/many others. Sometimes the answers are no and you have the "inner platform" effect -- after much effort and trial and error you end up with something. If you're lucky it's only slightly inferior to the programming language you wrote it in, and no easier to use.

Some languages are harder or easier to use than others, in particular if they are specialised to a purpose like R then some users will find them much easier. What you probably won't do, is make general applications programming fundamentally easier. At any one time there are probably several people/organizations in the world with the potential to do that, but your boss/company has to honestly consider whether or not that includes him/you.

There's a trick often used for games, which is to expose Lua bindings to the game engine. This allows designers to program in a relatively easy language, but still engage a "real" programmer where necessary for performance or to access particular functionality of the engine or the platform. The resulting Lua scripts are "data" as far as the engine is concerned. They don't all need to include much of what you'd call "logic" as opposed to config data, and often they pretty much define all the plot and environment, but not the whole gameplay. This is not 100% data-driven and it certainly isn't 100% error-free, but it's an interesting practical compromise.

share|improve this answer
    
Well put. The closest thing to a 100% data-driven is system is a programming language. And we already have those, so now our job is to provide one of them with the actual data, in the form of textual statements, to get it to deliver the actual functionality that we currently need. –  RBarryYoung Mar 8 at 18:50

You mean your boss wants you to write this:

[
  {
    "statement": "Assignment ",
    "variable": "App",
    "value": {
      "type": "Function",
      "arguments": [],
      "function-body": [
        {}
      ]
    }
  },
  {
    "statement": "Assignment",
    "variable": "App.prototype.action",
    "value": {
      "type": "Function",
      "arguments": [
        "data"
      ],
      "function-body": [
        {
          "statement": "Call",
          "function-name": "console.log",
          "arguments": [
            "data"
          ]
        }
      ]
    }
  }
]

To generate this:

var App = function () {};
App.prototype.action = function ( data ) {
    console.log( data );
}

First one is JSON and the second one is JavaScript.

Clearification

I think theoretically it isn't because the thing that interprets the data ends up needing to become turing complete to describe the application so you've just shifted the problem one level higher to no net benefit.

Is such a 100% Data Driven Application possible?

This is where I just started. With my answer I'm trying to agree with the original post that: It is possible, but you're correct, it will just shift the problem one level higher for no [obvious] benefit.

share|improve this answer
    
Programmers is tour conceptual questions and answers are expected to explain things. Throwing code dumps instead of explanation is like copying code from IDE to whiteboard: it may look familiar and even sometimes be understandable, but it feels weird... just weird. Whiteboard doesn't have compiler –  gnat Mar 14 at 10:55
    
@gnat Thanks for the comment; I've updated my answer, trying to make it more clear. Please let me know if it still doesn't seem to be clear enough. –  Mahdi Mar 14 at 11:25

I worked at a company where this was the goal. Snippets of SQL were stored in database tables, read at runtime, and executed. Performance was terrible, as you can imagine, and bugs were frequent. It was also impossible to debug, with no stack traces or anything else that makes life easy.

"Data-driven programming" results from a fundamental lack of understanding of what we're doing, as programmers; any data that is capable of making an algorithm happen is actually "programming," even if you've somehow managed to mingle (mangle?) the two ideas in the user interface. Now, this doesn't mean that you can't combine the two ideas from the other direction, so that all code is data; that's the premise behind lisp, which is enabled by its homoiconicity and exploited by its macro system. Yes, these concepts sound similar, but their implications and applications are very different in practice.

Also, this may be editorializing, but the places I've encountered that want "completely data-driven" programming really don't value their programmers. They think of the code as a cost center, something to be outsourced, or ignored.

share|improve this answer

You could argue convincingly, I think, that any web browser application could be considered 100% data driven1.

Of course, that doesn't make it simpler or easier to build applications on the web, in fact it makes them much harder.

Tell your boss that he's reinventing the web browser, and he will eventually have to reinvent JavaScript to build anything reasonably complex.

1 Well, if you ignore plugins, JavaScript and HTML5.

share|improve this answer

Yes. As far as I know, a system like Mathematica, which is a so-called powerful programming language but essentially is a shell, is built upon the similar idea that your boss has expected. Wolfram Mathematica is now becoming complex enough such that many computational tasks can be easily done by it.

Data driven is a concept. If we programmers are going to manipulate data in a simple way, we need a shell that is easy for us to play with data. Try to understand that once we start talking about learning a programming language based on syntax, we are actually learning the application interface or simply its shell. If we understand the shell, we can drive the programs.

As for 100 % data driven, if the compiler or the interpreter can understand the shell, computation is driven. If the data has the same underlying structure as the shell or interface that has, the data can be driven by compiler or interpreter too. I think Mathematica is a good explanation for why I respond you with a yes.

share|improve this answer
1  
this post is rather hard to read (wall of text). Would you mind editing it into a better shape? –  gnat Mar 8 at 5:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.