Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

It happened frequently that I have to provide access to a bunch of files organized in a directory tree according to some (sometimes loosely specified) rules. My standard pattern is to provide a Database class which is initialized with the root directory. This class provides getX()-like (example: getStructure()) methods to extract data from the database. These methods normally return semantically meaningful objects (Structure) with proper methods returning data (eg. structure.getPoints()). I am not completely happy with this design, for two main reasons.

The first problem is that the mapping between in-application objects and files may not be 1:1, that is, to create the Structure object I may have to open different files in the "database", and the mapping may not be perfect. In this case, I call the Structure a "Thick object". An alternative ("thin objects") is to stick to objects that are fully represented in a file, even if poor in high-level domain meaning (that is, if I have two files to define a structure, one file containing points, and the other containing connections between these points, I just provide a Points object and a Connections object, and let someone else "connect the dots" outside Database).

The other problem I have is the following: who should perform the actual file parsing? I envision two strategies: either the objects are able to deserialize themselves from file ("pickle-like" in python parlance), or they are just dumb data containers filled by stateless parser objects (one per file), or even by the database object itself.

When you do ORM, there are clear, well defined rules on how the object is represented in the database, and the process is extremely well defined in terms of interface and behavior. This is not necessarily verified with an arbitrary bunch of files some people call "database".

I would really enjoy your points on this regard. How should I perform proper deserialization in this case of "raw mapping"? Thin or thick objects ? Smart or dumb ?

Note that I don't really have any control at all when it comes to the files I have to access. I get the files from external sources, and I have to convert them into some kind of domain objects, generally only for reading, but sometimes also for writing. Readonly is by far the most frequent case though.

share|improve this question
1  
"process is extremely well defined in terms of interface and behavior". Unsurprising. SQL enforces this. Are you asking for the file-system equivalent of SQL? –  S.Lott Feb 23 '12 at 19:38
    
@S.Lott: Not really. Unfortunately, when the "database" is a bunch of text files (hence the quotes) with very little structure or format specification, the best I can do is to hope that the parser does not blow up, let alone hope that any constraints are enforced or respected. –  Stefano Borini Feb 23 '12 at 20:20
    
In that case, the ORM comparison seems off-base. If there is "very little structure" then the ORM comparison seems to be misleading. –  S.Lott Feb 23 '12 at 20:44
    
@S.Lott: You and I never really understand each other, eh? I said that ORM works in a well defined way, and what I have is nothing like that. –  Stefano Borini Feb 23 '12 at 21:50
    
And I said that ORM depends on SQL, which the file system utterly lacks, making it an unfair comparison. I don't think it's logically possible to have anything like that given the nature of the problem. I find it misleading except if you're looking for some SQL-like layer to put tight constraints on the file system. Since you don't want to impose those constraints, the very idea of ORM doesn't seem like a good design principle. It might confuse the possible answers by being part of the question. Are you going to repeat your position again? –  S.Lott Feb 24 '12 at 10:40
add comment

3 Answers

The key here is to have layers with well-defined interfaces. The lowest layer deals strictly with physical access to the files. It handles things like caching, concurrent access, opening, flushing, etc. Its interface allows higher layers to retrieve one line at a time without having to worry about things like if the file is already open.

The second layer provides dumb data structures to work with. Its interface would basically consist of a bunch of records with columns of simple data types, all of which are contained within a single file. Rows of the database, if you will.

The third layer provides smart objects to work with. It aggregates the database rows from multiple tables to make objects with friendly interfaces.

share|improve this answer
add comment

I call the Structure a "Thick object". An alternative ("thin objects") is to stick to objects that are fully represented in a file, even if poor in high-level domain meaning

These are not exclusive. You often need to do both.

You have physical format and logical layout issues separate from the problem domain. Three layers.

objects are able to deserialize themselves from file ("pickle-like" in python parlance),

Doesn't work out well in the long run. The class definition becomes cluttered with representation issues.

Python objects don't pickle themselves. Pickling is handled separately. Sometimes a class must provide codec hints to the pickle algorithm.

dumb data containers filled by stateless parser objects (one per file), or even by the database object itself.

"Stateless" parser objects don't always work out well.

Parser (or factory or builder) designs do work out well. Just leave off "stateless." Think of lexical scanning and parsing as separate from the physical, logical and final problem domain representations that may be required.

How should I perform proper deserialization in this case of "raw mapping"?

Physical Format classes may be necessary. Often, these are already provided in a language library. Don't reinvent the wheel.

Thin or thick objects ?

Not an exclusive choice. You have logical layout and the proper domain-level classes independent of any physical representation.

Smart or dumb ?

Outside the problem domain, classes should be focused on representation. I don't know of that makes them dumb or not.

Keep the concerns separate.

  • Physical format is for CSV, ZIP, XML, JSON, etc.

  • Logical layout handles variations in column names, data types, and structure.

  • Problem domain is the real objects you wanted to work with.

share|improve this answer
    
Impressive answer. Thank you –  Stefano Borini Feb 23 '12 at 22:21
    
Already wrote significant parts of it. It may, however, fail to fit your criteria of no SQL-like enforced structure. sourceforge.net/projects/stingrayreader/?source=directory –  S.Lott Feb 24 '12 at 10:42
add comment

I do this by serializing and deserializing objects using XML. This allows you to define your rules by how you structure your classes.

Some simple ToXml() and FromXML() helper methods do all of the heavy lifting.

So, for example:

public class MyRecord
{
    [XmlAttribute]
    public string Foo { get; set; }

    [XmlAttribute]
    public int Bar { get; set }

    [XmlAttribute]
    public double Baz { get; set; }
}

results in XML that looks more or less like this:

<MyRecord Foo="ABC" Bar="123" Baz="123.45">

You can make this as elaborate as you want, using composition to nest objects within other objects, and so forth, creating tree-like structures, or even collections.

public class MyRecordTree
{
    public MyRecordTree Node { get; set; }
    public List<MyRecordTree> Children { get; set; }
}
share|improve this answer
    
Well, the point is that I don't really have any control at all when it comes to the files. I get the files from external sources, and I have to convert them into some kind of domain objects, generally only for reading, but sometimes also for writing. –  Stefano Borini Feb 23 '12 at 19:08
    
I don't have a good answer for that. You are right to sequester that functionality in its own classes, but there's no generalized solution, other than elbow grease. Many have tried, but those solutions tend to suffer from "Inner Platform Effect;" their implementation is often more difficult than just applying the elbow grease. The best you can do is maintain some sort of consistency, as illustrated here. –  Robert Harvey Feb 23 '12 at 19:12
    
Other than that, tools like FileHelpers can provide ideas for ways to configure things so that you can more easily hook them together. See also Extract, Transform and Load –  Robert Harvey Feb 23 '12 at 19:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.