This is a follow up to this question. There I was asking how to do unit testing when you have a library of scientific algorithms. I have a similar problem now but with a different project.
I'm working on a 3D graphics engine framework abstraction for DirectX, OpenGl, WebGl, Silverlight, WPF, and basically whatever 3D API is out there, on C#, called Rendering .NET. The scenario is the following, the project is being developed by a small team of colleagues of mine, all working in the same office. We are all computer scientists, not much into the software engineering community and practices. I even had a hard time to convince to switch from Subversion to Git, and to publish the project on Codeplex. The project is getting big (around 80K lines of C#), and it now covers most of DirectX and OpenGl up to Shader Model 5.0, so its not a one man's project like the one described in the linked question. We also want to encourage the open source community to collaborate.
So far we've been testing by writing small applications that involve initializing all devices and resources, setting up a scene, and drawing. The thing is, I don't know how to make it different, I mean, how to design small test cases that test specific features of the framework, like resource allocation, primitive tessellation, shader compilation, etc., without having to recreate the entire engine initialization. For some of these cases I can think of useful mocks, but in general, how can I test the correctness of a rendering algorithm when the output is visual (an image, or an animated scene)? Right now the most clever thing we have come up is to render the same scene to a software renderer, DirectX and OpenGl, and compare them pixel by pixel, but tiny variations of the APIs make the resulting images different enough to fail the tests without being actual bugs.
So, what is the "correct" testing approach here?