Adding colorobox to link is not so hard. You need to follow module instructions and it will work, just be careful to add EMBED code to URL, not just plain link. For example when adding You Tube video do this
<a class="colorbox-load" href="http://www.youtube.com/embed/5fMR_VsMs6k?width=853&height=480&iframe=true&autoplay=1"><img alt="" src="/sites/default/files/comp-arrow.png" style="width: 92px; height: 92px; " /></a>and notice youtube link, it is not just plain link but a link you get when clicking on share and then embed buttons on You Tube page