According to the man page for pbpaste,
-Prefer {txt | rtf | ps}
tells pbpaste what type of data to look for in the
I can't see any way to do it from inside AppleScript, but since you're working in the shell anyway, I'd just post-process it: the "hex-encoded crap" is the RTF data you want. The simplest script I can think of is
perl -ne 'print chr foreach unpack("C*",pack("H*",substr($_,11,-3)))'
An explanation: substr($_,11,-3) strips off the «data RTF and »\n bits (each of the guillemets is two bytes); pack("H*", ...) packs hex-encoded data into a bytestream; unpack("C*", ...) unpacks a bytestream into an array of character values; print chr foreach ... converts each integer in the array to its corresponding character and prints it; and the -ne options evaluate the script given for each line, with that line implicitly stored in $_. (If you want that script in its own file, just make sure the shebang line is #!/usr/bin/perl -ne.) Then, running
osascript -e 'the clipboard as «class RTF »' | \
perl -ne 'print chr foreach unpack("C*",pack("H*",substr($_,11,-3)))'
will give you raw RTF output.