If for instance, we set a -vendor-transform: rotate(40deg) css attribute on a rectangle
It is not bug in jQuery. Simply it is not supported. If you check jQuery UI source code you will figure out that it doesn't use transformation matrix to calculate difference between transformed object and page.
Your example, and probably every jQ UI drag implementation suffer from this issue cause of 2 methods in JQ UI source code (around 314 line of jquery.ui.draggable.js file v1.8.23 ). Calculated offset do not matter about change in offset since rotation is done over center of element.
You have to calculate what is that change. Here is workaround, quick and dirty. The idea is to check what is difference in bounding box of transformed element.
Check sample here http://jsfiddle.net/mjaric/9Nqrh/
Ignore part with first two rotations, they are just done to minimize lines of code. Third involves translation of coordinate system for calculated difference. It will offset left and top after translation is performed (note it is first in filter).
If you want to avoid first two rotation filters, You could make code using formula for 2D rotation:
x' = x cos f - y sin f
y' = y cos f + x sin f
where f is angle of rotation, but it's not that simple and also includes more lines of code where you have to calculate what is diagonal angle of original bounding box since you need initial angle of top left corner which x and y coords are comparing to x axis (positive part). Then calculate change in x-x' and y-y'. But I'm predicting some issues with sign of change and coding/debugging would take more time then I have right now. Sorry cause of that but I'm sure you can figure out what to do after reading this post.