Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

My company (let's call them Acme Technology) has a library of approximately one thousand source files that originally came from its Acme Labs research group, incubated in a development group for a couple years, and has more recently been provided to a handful of customers under non-disclosure. Acme is getting ready to release perhaps 75% of the code to the open source community. The other 25% would be released later, but for now, is either not ready for customer use or contains code related to future innovations they need to keep out of the hands of competitors.

The code is presently formatted with #ifdefs that permit the same code base to work with the pre-production platforms that will be available to university researchers and a much wider range of commercial customers once it goes to open source, while at the same time being available for experimentation and prototyping and forward compatibility testing with the future platform. Keeping a single code base is considered essential for the economics (and sanity) of my group who would have a tough time maintaining two copies in parallel.

Files in our current base look something like this:

> // Copyright 2012 (C) Acme Technology, All Rights Reserved.
> // Very large, often varied and restrictive copyright license in English and French,
> // sometimes also embedded in make files and shell scripts with varied 
> // comment styles. 
> 
> 
>   ... Usual header stuff...
>
> void initTechnologyLibrary() {
>     nuiInterface(on);
> #ifdef  UNDER_RESEARCH
>     holographicVisualization(on);
> #endif
> }

And we would like to convert them to something like:

> // GPL Copyright (C) Acme Technology Labs 2012, Some rights reserved.
> // Acme appreciates your interest in its technology, please contact xyz@acme.com 
> // for technical support, and www.acme.com/emergingTech for updates and RSS feed.
> 
>   ... Usual header stuff...
>
> void initTechnologyLibrary() {
>     nuiInterface(on);
> }

Is there a tool, parse library, or popular script that can replace the copyright and strip out not just #ifdefs, but variations like #if defined(UNDER_RESEARCH), etc.?

The code is presently in Git and would likely be hosted somewhere that uses Git. Would there be a way to safely link repositories together so we can efficiently reintegrate our improvements with the open source versions? Advice about other pitfalls is welcome.

share|improve this question
13  
This codebase is screaming for branches. –  Florian Margaine Nov 7 '12 at 7:19
    
An example of using branches for this purpose would be most welcome. –  DeveloperDon Nov 8 '12 at 3:31
add comment

5 Answers

up vote 6 down vote accepted

It seems like it wouldn't be too difficult to write a script to parse the preprocessors, compare them to a list of defined constants (UNDER_RESEARCH, FUTURE_DEVELOPMENT, etc.) and, if the directive can be evaluated to false given what's defined, remove everything up to the next #endif.

In Python, I'd do something like,

import os

src_dir = 'src/'
switches = {'UNDER_RESEARCH': True, 'OPEN_SOURCE': False}
new_header = """// GPL Copyright (C) Acme Technology Labs 2012, Some rights reserved.
// Acme appreciates your interest in its technology, please contact xyz@acme.com 
// for technical support, and www.acme.com/emergingTech for updates and RSS feed.
"""

filenames = os.listdir(src_dir)
for fn in filenames:
    contents = open(src_dir+fn, 'r').read().split('\n')
    outfile = open(src_dir+fn+'-open-source', 'w')
    in_header = True
    skipping = False
    for line in contents:
        # remove original header
        if in_header and (line.strip() == "" or line.strip().startswith('//')):
            continue
        elif in_header:
            in_header = False
            outfile.write(new_header)

        # skip between ifdef directives
        if skipping:
            if line.strip() == "#endif":
                skipping = False
            continue
        # check
        if line.strip().startswith("#ifdef"):
            # parse #ifdef (maybe should be more elegant)
            # this assumes a form of "#ifdef SWITCH" and nothing else
            if line.strip().split()[1] in switches.keys():
                skipping = True
                continue

        # checking for other forms of directives is left as an exercise

        # got this far, nothing special - echo the line
        outfile.write(line)
        outfile.write('\n')

I'm sure there are more elegant ways to do it, but this is quick and dirty and seems to get the job done.

share|improve this answer
    
Wow, thanks. There is a lot of logic potentially to make a good filter and I appreciate your example. I hope to find something for reuse, and my development machine is fast with a big memory so performance is not a very big concern to run separate filters for the copyright and the defines, or to run the define filter more than once. We actually have multiple defines related to keywords that designate multiple future projects and a couple past projects that will not be released open source, but still are used internally and by early adopting customers. –  DeveloperDon Nov 8 '12 at 3:19
add comment

I'm not going to show you a tool to convert your codebase, plenty of answers already did that. Rather, I'm answering your comment about how to handle branches for this.

You should have 2 branches:

  • Community (let's call the open source version like this)
  • Professional (let's call the closed source version like this)

The preprocessors shouldn't exist. You have two different versions. And a cleaner codebase overall.

You're afraid of maintaining two copies in parallel? Don't worry, you can merge!

If you're making modifications to the community branch, just merge them in the professional branch. Git handles this really well.

This way, you keep 2 maintained copies of your codebase. And releasing one for open source is easy as pie.

share|improve this answer
    
Thanks, I'll explore this further. –  DeveloperDon Nov 9 '12 at 11:42
add comment

I have a solution but it will require a little work

pypreprocessor is a library that provides a pure c-style preprocessor for python that can also be use as a GPP (General Purpose Pre-Processor) for other types of source code.

Here's a basic example:

from pypreprocessor import pypreprocessor

pypreprocessor.input = 'input_file.c'
pypreprocessor.output = 'output_file.c'
pypreprocessor.removeMeta = True
pypreprocessor.parse()

The preprocessor is extremely simple. It makes a pass through the source and conditionally comments out source based on what is defined.

Defines can be set either through #define statements in the source or by setting them in the pypreprocessor.defines list.

Setting the input/output parameters allow you to explicitly define which files are being opened/closed so a single preprocessor can be setup to batch process a large number of files if desired.

Setting the removeMeta parameter to True, the preprocessor should automatically extract any and all preprocessor statements leaving only the post-processed code.

Note: Usually this wouldn't need to be set explicitly because python removed commented code automatically during the compilation to bytecode.

I only see one edge case. Because you're looking to preprocess C source, you may want to set the processor defines explicitly (ie through pypreprocessor.defines) and tell it to ignore the #define statements in the source. That should keep it from accidentally removing any constants you may use in your project's source code. There currently is no parameter to set this functionality but it would be trivial to add.

Here's a trivial example:

from pypreprocessor import pypreprocessor

# run the script in 'production' mode
if 'commercial' in sys.argv:
    pypreprocessor.defines.append('commercial')

if 'open' in sys.argv:
    pypreprocessor.defines.append('open')

pypreprocessor.removeMeta = True
pypreprocessor.parse()

Then the source:

#ifdef commercial
// Copyright 2012 (C) Acme Technology, All Rights Reserved.
// Very large, often varied and restrictive copyright license in English and French,
// sometimes also embedded in make files and shell scripts with varied 
// comment styles.
#ifdef open
// GPL Copyright (C) Acme Technology Labs 2012, Some rights reserved.
// Acme appreciates your interest in its technology, please contact xyz@acme.com 
// for technical support, and www.acme.com/emergingTech for updates and RSS feed.
#endif

Note: Obviously, you'll need to sort out a way to set the input/output files but that shouldn't be too difficult.

Disclosure: I am the original author of pypreprocessor.


Aside: I originally wrote it as a solution to the dreaded python 2k/3x maintenance issue. My approach was, do 2 and 3 development in the same source files and just include/exclude the differences using preprocessor directives. Unfortunately, I discovered the hard way that it's impossible to write a true pure (ie doesn't require c) preprocessor in python because the lexer flags syntax errors in incompatible code before the preprocessor gets a chance to run. Either way, it's still useful under a wide range of circumstances including yours.

share|improve this answer
    
This rocks. If nothing else we could do some like of three way diff that processed the files with and without the code we wanted to exclude, took their diff, then removed the diffed lines from the original. –  DeveloperDon Nov 8 '12 at 3:30
    
@DeveloperDon Yep, that's the general idea. There are a few different ways to handle it, it depends on how you plan to manage the commit-release cycle. This piece just automates away a lot of the work that would otherwise be tedious and/or prone to error. –  Evan Plaice Nov 8 '12 at 4:16
add comment

I was thinking about passing your code through the preprocessor to only expand macros, thus outputting only the interesting part in the #ifdefs.

Something like this should work:

gcc -E yourfile.c

But:

  • You'll lose all comments. You can use -CC to (kind of) preserve them, but then you'll still have to strip off the old copyright notice
  • #includes are expanded too, so you'll end up with a big file containing all the content of the included header files
  • You'll lose "standard" macros.

There might be a way to limit which macros are expanded; however my suggestion here is to split up things, instead of doing (potentially hazardous) processing on the files (by the way, how would you plan to maintain them after? e.g. reintroduce code from the opensource version into your closed source?).

That is, try putting the code you want to opensource in in external libraries as much as possible, then use them as you would with any other library, integrating with other "custom" closed-source libraries.

It might take a bit longer at first to figure out how to restructure things, but it's definitely the right way to accomplish this.

share|improve this answer
    
I had considered whether there might be something that could be done with the preprocessor to selectively eliminate the blocks we won't release yet. The code is complex and we will likely need more comments rather than less, but your suggestion is certainly worth having in the brainstorm list. WRT questions about how do we plan to maintain the source and move code backward and forward to the community, there is more planning needed. Bringing code into the proprietary code raises some good questions. –  DeveloperDon Nov 8 '12 at 3:09
add comment

Probably it would be good idea to

1.add comment tags like :

> // *COPYRIGHT-BEGIN-TAG*
> // Copyright 2012 (C) Acme Technology, All Rights Reserved.
> // Very large, often varied and restrictive copyright license in English and French,
> // sometimes also embedded in make files and shell scripts with varied 
> // comment styles. 
> // *COPYRIGHT-ENG-TAG*
>   ... Usual header stuff...
>
> void initTechnologyLibrary() {
>     nuiInterface(on);
> #ifdef  UNDER_RESEARCH
>     holographicVisualization(on);
> #endif
> }

2. Write script for open source builder to go through all files and replace text between COPYRIGHT-BEGIN-TAG and COPYRIGHT-ENG-TAG tags

share|improve this answer
1  
Do I need the begin tag? So far all our source files start with the copyright in the first line, and our shell scripts start with the copyright in the second line. There are a lot of files, so I would like to do the very smallest amount of hand editing that is possible. –  DeveloperDon Nov 8 '12 at 3:00
    
I think some files may use Doxygen to delineate their function, parameter, and return value names. For those files not already setup that way, it could really be a lot of editing if we made a choice that took up further in that direction. –  DeveloperDon Nov 8 '12 at 3:02
    
At least you have to change it once. if your copyright policy changed you can manage it. –  Alex Hashimi Nov 8 '12 at 3:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.