Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Make is a standard tools for building software. But

make decides whether a target needs to be regenerated by comparing file modification times.

Are there any proven, preferably small tools that handle builds not for software but for data? Something that regenerates targets not only on mod times but on certain other properties (e.g. completeness). (Or alternatively some paper that describes such a tool.)

As illustration: I'd like to automate the following process:

  • get data (e.g. a tarball) from some regularly updated source
  • copy somewhere if it's not there (based e.g. on some filename-scheme)
  • convert the files to different format (but only if there aren't successfully converted ones there - e.g. from a previous attempt - custom comparison routine)
  • for each file find a certain data element and fetch some additional file from say an URL, but only if that hasn't been downloaded yet (decide on existence of file and file "freshness")
  • finally compute something (e.g. word count for something identifiable and store it in the database, but only if the DB does not have an entry for that exact ID yet)

Observations:

  • there are different stages
  • each stage is usually simple to compute or implement in isolation
  • each stage may be simple, but the data volume may be large
  • each stage may produce a few errors
  • each stage may have different signals, on when (re)processing is needed

Requirements:

  • builds should be interruptable and idempotent (== robust)
  • when interrupted, already processed objects should be reused to speedup the next run
  • data paths should be easy to adjust (simple syntax, nothing new to learn, internal dsl would be ok)
  • some form of dependency graph, that describes the process would be nice for later visualizations
  • should leverage existing programs, if possible

I've done some research on make alternatives like rake and have worked a lot with ant and maven in the past. All these tools naturally focus on code and software build, not on data builds. A system we have in place now for a task similar to the above is pretty much just shell scripts, which are compact (and are a ok glue for a variety of other programs written in other languages), so I wonder if worse is better?

share|improve this question

closed as off-topic by Martijn Pieters, MichaelT, Bart van Ingen Schenau, Kilian Foth, GlenH7 Feb 25 at 23:53

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – Martijn Pieters, MichaelT, Bart van Ingen Schenau, Kilian Foth, GlenH7
If this question can be reworded to fit the rules in the help center, please edit the question.

5  
I do similar things with plain make, it's definitely possible. I wouldn't say make was focused on software, anything you can describe with dependency rules can be run with it. The only very software-specific thing in make are the implicit rules, but if you don't use them they are never in your way. –  Benjamin Bannier Nov 23 '12 at 8:08
    
@honk - You should make an answer instead of a comment. –  mouviciel Nov 23 '12 at 9:50
add comment

7 Answers 7

up vote 2 down vote accepted

So far I'm not seeing anything specific in your requirements that make can't handle, if you structure the makefile correctly. With make, you declare that file b is dependent on file a. You then tell it how to make file b from file a. If file b is missing, or older than file a, make will execute it's instructions to do what you want.

If your conversion process is multi-step, then you will want to have multiple rules (d depends on c, c depends on b, b depends on a), and each rule has it's own input and output files, and it's own build instructions. If the build is interrupted, then only some of the files will be present, and everything will pick up where it left off.

Since each 'how to make this from that' is just a bit of shell script, you can make it do anything you like.

You've got an implication in there that you want to use something other than file times, but you don't say what that is (completeness is mentioned, but that's easily handled by proper dependency setup). If you need to make it trigger on (for instance) database contents, then make is probably not the tool for you. But if it's just files on disk, then it doesn't matter if they are software or data, make can handle it.

share|improve this answer
add comment

Try out SCons. It gives you all the flexibility of the Python language, while keeping the focus on “building things”.

It has a good documentation, and you can write your own builders, which is what you would be looking for.

share|improve this answer
add comment

I'd suggest waf, another Python build system. I never used SCons, so I don't know how they compare, but I used waf for non-software builds and was very happy with it.

I must admit though, it has quite a learning curve - at least it did for me. The fact that I wasn't familiar with Python certainly had something to do with that, but waf itself is quite "meta". It allows for very powerful abstractions, but it's core concepts might take some time to grasp.

On the plus side, it's very mature and stable - it has been under development for over six years, and is actively maintained. Despite it's age, there is a certain state of the art feel to it - but that may be because I came across waf after some painful experiences with make. In addition, it has no dependencies and can be used with older Python versions - such as 2.3.

The waf book serves as the primary documentation; you can also ask questions on the waf Google group.

share|improve this answer
add comment

You could take again a look to rake.

It is a ruby-based make-tool, but it provides all possibilities of a programming language. And you can define your own .

For a test I extended rake to (re)generate statistic graphs based on changes in a data base. I used a timestamp field in a table similar to modification dates in files - if there where new data, the I regenerate the graph.

If you are interested in details, give me a comment and I will check if I find again my old coding with DB-based rake tasks.

share|improve this answer
add comment

Make was made to automatically build software, but it is hopefully not limited to this function.

Make is a pull tool: you declare what can be done, what is needed to do something and how to produce this thing on the basis of the requirements. Then you ask to make something. If the desired thing is already present, and all its requirements haven't been modified since the last build, then Make does nothing. If the desired thing (target) does not exist, or it exists but one of its dependencies has been updated, Make (re)builds it. The process is recursive.

In order to determine if "something changed" since the last build, the last change date is used.

You can easily imagine a Latex to pdf process: To produce the final pdf, you need a set of text files and some png files. Each time you modify your text, you must build your pdf again. The picture files are generated on the basis of raw data. If the raw data changes, you need to invoke gnuplot which "transforms" each raw data file into a nice png file.

So you can use Make to automate your process, simply by thinking data pull instead of data push. The single push step will be the first one: gathering the original data.

However, Make is showing its age, and you might prefer more modern alternatives.

share|improve this answer
add comment

After evaluating lots of tools and libraries I found luigi, which is

a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

It's a great tool, plus over the last year a small but very friendly community has emerged around it. For a short intro, have a look at this PyData talk: http://vimeo.com/63435580

share|improve this answer
add comment

Drake seems to be designed for exactly this purpose. I haven't tried it yet, but it looks relevant at least.

I use SCons for this kind of thing now, and it works ... OK. The capabilities are all there, but I find it gets ugly kind of quickly.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.