How can I randomize the lines in a file using standard tools on Red Hat Linux?

问题:

How can I randomize the lines in a file using standard tools on Red Hat Linux?

I don't have the shuf command, so I am looking for something like a perl or awk one-liner that accomplishes the same task.

回答1:

And a Perl one-liner you get!

perl -MList::Util -e 'print List::Util::shuffle <>'

It uses a module, but the module is part of the Perl code distribution. If that's not good enough, you may consider rolling your own.

I tried using this with the -i flag ("edit-in-place") to have it edit the file. The documentation suggests it should work, but it doesn't. It still displays the shuffled file to stdout, but this time it deletes the original. I suggest you don't use it.

Consider a shell script:

#!/bin/sh  if [[ $# -eq 0 ]] then   echo "Usage: $0 [file ...]"   exit 1 fi  for i in "$@" do   perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new   if [[ `wc -c $i` -eq `wc -c $i.new` ]]   then     mv $i.new $i   else     echo "Error for file $i!"   fi done

Untested, but hopefully works.

回答2:

Um, lets not forget

sort --random-sort

回答3:

"shuf" is the best way

"sort -R" is painfully slow. I just tried to sort 5GB file. I gave up after 2.5 hours. Then shuf sorted it in a minute.

回答4:

cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7-

Read the file, prepend every line with a random number, sort the file on those random prefixes, cut the prefixes afterwards. One-liner which should work in any semi-modern shell.

EDIT: incorporated Richard Hansen's remarks.

回答5:

A one-liner for python:

python -c "import random, sys; lines = open(sys.argv[1]).readlines(); random.shuffle(lines); print ''.join(lines)," myFile

And for printing just a single random line:

python -c "import random, sys; print random.choice(open(sys.argv[1]).readlines())," myFile

But see this post for the drawbacks of python's random.shuffle(). It won't work well with many (more than 2080) elements.

回答6:

Related to Jim's answer:

My ~/.bashrc contains the following:

unsort () {     LC_ALL=C sort -R "$@" }

With GNU coreutils's sort, -R = --random-sort, which generates a random hash of each line and sorts by it. The randomized hash wouldn't actually be used in some locales in some older (buggy) versions, causing it to return normal sorted output, which is why I set LC_ALL=C.

---

Related to Chris's answer:

perl -MList::Util=shuffle -e'print shuffle<>'

is a slightly shorter one-liner. (-Mmodule=a,b,c is shorthand for -e 'use module qw(a b c);'.)

The reason giving it a simple -i doesn't work for shuffling in-place is because Perl expects that the print happens in the same loop the file is being read, and print shuffle <> doesn't output until after all input files have been read and closed.

As a shorter workaround,

perl -MList::Util=shuffle -i -ne'BEGIN{undef$/}print shuffle split/^/m'

will shuffle files in-place. (-n means "wrap the code in a while (<>) {...} loop; BEGIN{undef$/} makes Perl operate on files-at-a-time instead of lines-at-a-time, and split/^/m is needed because $_=<> has been implicitly done with an entire file instead of lines.)

回答7:

When I install coreutils with homebrew

brew install coreutils

shuf becomes available as n.

回答8:

Mac OS X with DarwinPorts:

sudo port install unsort cat $file | unsort | ...

回答9:

FreeBSD has its own random utility:

cat $file | random | ...

It's in /usr/games/random, so if you have not installed games, you are out of luck.

You could consider installing ports like textproc/rand or textproc/msort. These might well be available on Linux and/or Mac OS X, if portability is a concern.

回答10:

On OSX, grabbing latest from http://ftp.gnu.org/gnu/coreutils/ and something like

./configure make sudo make install

...should give you /usr/local/bin/sort --random-sort

without messing up /usr/bin/sort

回答11:

Or get it from MacPorts:

$ sudo port install coreutils

and/or

$ /opt/local//libexec/gnubin/sort --random-sort

转载请标明出处:How can I randomize the lines in a file using standard tools on Red Hat Linux?

文章来源: How can I randomize the lines in a file using standard tools on Red Hat Linux?

跳转到

• 问题:
• 回答1:
• 回答2:
• 回答3:
• 回答4:
• 回答5:
• 回答6:
• 回答7:
• 回答8:
• 回答9:
• 回答10:
• 回答11:

Close