Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This seems like the sort of problem that must have been solved elegantly long ago, but I haven't the foggiest how to google it and find it.

Suppose you're maintaining a large legacy system, which has a large collection of data (tens of GB) of various formats, including XML and two different internal configuration formats. Suppose further that there are abstract rules governing the values these files may or may not contain.

EXAMPLE: File A defines the raw, mathematical data pertaining to the aerodynamics of a car for consumption of the physics component of the system. File B contains certain values from File A in an easily accessible, XML hierarchy for consumption of a different component of the system. There exists, therefore, an abstract rule (or constraint) such that the values from File B must match the values from File A.

This is probably the simplest constraint that can be specified, but in practice, the constraints between files can become very complicated indeed.

What is the best method for managing these constraints between files of arbitrary formats, short of migrating it over to an RDBMS (which simply isn't feasible for the foreseeable future)? Has this problem been solved already?

To be more specific, I would expect the solution to at least produce notifications of violated constraints; the solution need not resolve the constraints.

==============================
Sample file structures

File A (JeepWrangler2011.emv):

MODEL JeepWrangler2011
{
    EsotericMathValueX 11.1
    EsotericMathValueY 22.2
    EsotericMathValueZ 33.3
}

File B (JeepWrangler2011.xml):

<model name="JeepWrangler2011">
  <!--These values must correspond File A's EsotericMathValues-->
  <modelExtent x="11.1" y="22.2" z="33.3"/>
  [...]
</model>

EDIT: Reflecting on this issue, it seems to me that a software solution would require something like the following:

  1. Develop an abstract language to describe the structure of arbitrary formats.
  2. Develop an abstract language to describe constraints between files.
  3. Develop a tool that consumes format-structure-descriptions (item 1), constraint-declarations (item 2), and a file that maps various files (pointed to by file paths) to a format-structure-declaration and produces a list of violated constraints.

It's a fascinating, non-trivial problem.

share|improve this question
    
have you thought about using XML Transformation? it might work for what you are doing. it's still a little unclear. can you give a detailed example of the two files? and what needs to match? this might be over my head a little bit... –  Malachi Oct 19 '12 at 16:41
1  
How was file B created? From A (in which case the logic is straightforward), or by an independent process (in which case you have a variant on Multiple Masters and my deepest sympathies)? –  Peter Rowell Oct 19 '12 at 16:47
1  
@ChristopherBerman: Multiple Masters occurs in any situation where data that should be the same is created/modified on 2 (or more) independent copies. It could be two people editing the same Word document on different machines, or two copies of a database being updated in slightly different ways. It's a situation you want to avoid if at all possible because you will ultimately have to merge those 2 masters together and the collision resolution can get ugly. Your situation means that A and B should have corresponding data, but very well might not. –  Peter Rowell Oct 19 '12 at 20:00
2  
Yes it is a fascinating non-trivial problem, that's why a bunch of people studied it a lot, they used the term "relational data" and created an abstract language (SQL) for dealing with and defining the structures and constraints along with huge persistence systems (MSSQL, Oracle, MySQL, ...) to allow this data to be stored and parsed quickly ;) –  Jimmy Hoffa Oct 19 '12 at 20:08
1  
@ChristopherBerman Think of it this way, is it easier to write a SQL like DSL or just use some ETL tools and get your data into an RDBMS? You could even just have a consistent ETL process everytime you want to validate these constraints while maintaining the data outside of the DB so they can maintain their current format. –  Jimmy Hoffa Oct 19 '12 at 20:20
show 6 more comments

1 Answer

up vote 3 down vote accepted

A database is the proper solution to this problem, anything else is just a hacky reinvention of existing database tools. You should reevaluate why you think this isn't a feasible solution, its not going to be more effort than a custom system to ensure you constraints are met. Databases are designed to create one version of the truth and serve it up any way you need it.

share|improve this answer
1  
I think the process that need be studied here is ETL as the poster may not know that getting data from format X into databases in a structured useful format is a well documented and tooled problem domain. –  Jimmy Hoffa Oct 19 '12 at 20:11
    
I am unfamiliar with ETL. Another topic to google/wiki. Thanks Jimmy! –  Christopher Berman Oct 19 '12 at 20:20
    
@Ryathal: On my side of the fence (theoretical familiarity with databases, but no practical implementation experience), I guess a database is the way to go. The projected solution in my question sort of IS a database defined on existing documents (which would be a foolish thing to recreate if it already exists). –  Christopher Berman Oct 19 '12 at 20:24
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.