I have a lot of text files that look like this:
>ALGKAHOLAGGATACCATAGATGGCACGCCCT >BLGKAHOLAGGATACCATAGATGGCACGCCCT >HLGKAHOLAGGATACCATAGATGGCACGCCC
Yes, but I wouldn't. I would use shuf or sort -R (neither POSIX) to randomize the file and then select the first n lines using head.
shuf
sort -R
n
head
If you really want to use awk for this, you would need to use the rand function, as Mark Setchell points out.
awk
rand