According to http://dougscripts.com/ setting shuffle and repeat modes via applescript is broken in iTunes 11.
According to this stackoverflow answer shuffle is now a
It looks like many of these scripts are broken in the newest iTunes. Here are two the work:
tell application "System Events" to tell UI element "iTunes" of list 1 of process "Dock"
if not (exists) then return
perform action "AXShowMenu"
click menu item "Shuffle" of menu 1
end tell
That one toggles shuffle via the Dock. You can watch the Dock menu animate when you use it.
This one toggles shuffle via the menu, invisibly:
tell application "System Events"
tell application process "iTunes"
tell menu 1 of menu item "Shuffle" of menu "Controls" of menu bar 1
if (value of attribute "AXMenuItemMarkChar" of item 1 of menu items as string) = "✓" then
click menu item 2
else
click menu item 1
end if
end tell
end tell
end tell
Both of these will work even with iTunes is in the background.