Put very simply, my company's product is a webapp with a database backend. We have a stage environment and a production environment for each of roughly 100 clients. There is a separate database for each environment. The database contents come from a number of CSV files that are created and maintained by our (non-technical) data team. Data is client specific and must also be updated when dev adds new features to the product. However, not all clients use the same version of the product, so not only the contents but also the structure of the CSVs varies by client.
Currently, the CSV files are versioned in SVN. However, there is no separation between tested and untested data. My team is responsible for loading new data to customer environments, but we've gotten burned before by loading untested data to a production environment. The clear answer to this is to mimic a development setup and create a trunk, branches, and tags, where trunk is always stable (tested data), branches contain experimental/untested changes, and tags are created every time new data is loaded to an environment.
My question involves the dependency between data and code...when dev makes a release that includes new features, the data has to be updated to accommodate those features before we can upgrade the code on an environment. How can we track whether the data for a particular client is compatible with a particular version of code?
My first instinct is to create a branch in the data repo every time a code upgrade is planned for a given customer. The branch name would correspond to the release number. However, because the code version on our production boxes usually lags significantly behind that on our stage environments, the data team would have to work in two branches simultaneously, ensuring that new features for the latest and greatest code only go in the newest branch, but fixes continue to go in the older branch as well. Of course, once the code on stage and production diverge, we cannot be 100% sure that the data fixes we test on stage will work on the older code installed in production. Does anyone have any ideas about how we can manage this process better?