I'm currently working on a project where there is a requirement to dynamically composite multiple synchronised video sources onto a single canvas. The initial prototype I put together loaded the video into HTMLVideoElements and then used requestAnimFrame (via Paul Irish's shim) to draw these onto a single canvas and maintain sync between the different elements.
Although this is functional and the system can hold sync fairly well (+/- 80ms tolerance) it is somewhat inefficient to say the least. I've been thinking about optimisations and one approach that seems pretty straight forward is to combine all the media into a single larger 'video sprite' and serve this. This would allow it to be load into a single video element which could then be used extract and render the areas of interest client side using
drawImage() and remove the need to manage sync between sources.
Has anyone else implemented something similar in the past? If so what approach worked well for you?