how to trim file - remove the columns with the same value

Question

I would like your help on trimming a file by removing the columns with the same value.

# the file I have (tab-delimited, millions of columns)
jack 1 5 9
joh         


        

Answer 1

As far as I can tell, you'll need to make this a multi-pass program to meet your needs without blowing through memory. For starters, load a single line of the file into an array.

open FH,'datafile.txt' or die "$!";
my @mask;
my @first_line= split(/\s+/,);

Then you'll want to sequentially read in the other lines

while(my @next_line= split(/\s+/,)) {
/* compare each member of @first_line to @next_line
 * any match, make a mark in mask to true
 */

When you get to the bottom of the file, go back to the top and use mask to determine which colums to print.