According to http://dougscripts.com/ setting shuffle and repeat modes via applescript is broken in iTunes 11.
According to this stackoverflow answer shuffle is now a
I was optimistic when I saw the AppleScript property current playlist of the iTunes application, but it doesn't work well. It's able to get and set the current playlist's name, but it can do neither for the properties shuffle or song repeat. It errors when trying to set either property, and it always returns 'false' for shuffle and 'off' for song repeat.
I think your only option is UI Scripting. Here's how to toggle shuffle through the menu bar:
tell application "System Events" to perform action "AXPress" of (first menu item of process "iTunes"'s menu bar 1's menu bar item "Controls"'s menu 1's menu item "Shuffle"'s menu 1 whose name ends with "Shuffle")
And here's how to set repeat:
tell application "System Events" to tell process "iTunes"'s menu bar 1's menu bar item "Controls"'s menu 1's menu item "Repeat"'s menu 1
perform action "AXPress" of menu item "Off"
perform action "AXPress" of menu item "All"
perform action "AXPress" of menu item "One"
end tell