Is there any difference how the css3 effects (for example animation) are rendered if the doctype is defined as
(html5)
or
There is almost no difference* between those two doctypes in terms of CSS rendering.
The choice of doctype only affects validation, and which "browser mode" is used out of "Quirks Mode", "Standards Mode" ("no quirks mode"), "Almost Standards Mode" ("limited quirks mode").
See: http://hsivonen.iki.fi/doctype/
So, XHTML 1.0 Transitional
gives "Almost Standards Mode", whereas gives "Standards Mode".
*One difference is a minor and easily fixable adjustment concerning tables. Another is the treatment of heading elements nested inside certain HTML5 elements.