It means the game capture is not working correctly from directx, just try with running the game in windowed mode then.
Look at this guide:
http://www.gamereplays.org/companyofheroes/portals.php?show=page&name=streaming-company-of-heroes-justin-tv-broadcast
We're currently working with Inverse on updating it but all the necessary should still be relevant.