awk

Combining two very large files ignoring the first sentence

一笑奈何 提交于 2020-01-15 07:16:07
问题 I want to combine two giant file each few hundred megabyte into a single file while ignoring the first line. I wanted to use awk as I thought it should be the most optimized way. the way I'm doing it only ignores the first line of second file. Any idea how to do make work or if there's a faster way to do it? awk 'FNR!=NR && FNR==1 {next} 1' 'FNR!=NR && FNR==1 {next} 2' s_mep_{1,2}.out >> s_mep.out 回答1: $ awk 'FNR>1' file{1,2} > file_12 回答2: With sed (sed '1d' file_1 ; sed '1d' file_2) > new

Find the maximum values in nth column for each distinct values in 1st column in bash

拟墨画扇 提交于 2020-01-15 07:12:26
问题 I have a 3 column file and I want to find the maximum value of the third column with rows with same first column and have also the second column in output. Input: 1 234 0.005 1 235 0.060 1 236 0.001 2 234 0.010 2 235 0.003 2 236 0.003 3 234 0.004 3 235 0.100 3 236 0.004 Desired output: 1 235 0.060 2 234 0.010 3 235 0.100 I found this hint from previous questions but I do not know how to have also the second column: !($1 in max) || $3>max[$1] { max[$1] = $3 } END { PROCINFO["sorted_in"] = "

Find the maximum values in nth column for each distinct values in 1st column in bash

删除回忆录丶 提交于 2020-01-15 07:10:49
问题 I have a 3 column file and I want to find the maximum value of the third column with rows with same first column and have also the second column in output. Input: 1 234 0.005 1 235 0.060 1 236 0.001 2 234 0.010 2 235 0.003 2 236 0.003 3 234 0.004 3 235 0.100 3 236 0.004 Desired output: 1 235 0.060 2 234 0.010 3 235 0.100 I found this hint from previous questions but I do not know how to have also the second column: !($1 in max) || $3>max[$1] { max[$1] = $3 } END { PROCINFO["sorted_in"] = "

How to merge separated fields in rows into one based on common fields in AWK / UNIX

笑着哭i 提交于 2020-01-15 06:38:28
问题 I am still just a new user to UNIX and especially to AWK. I am having the problem to merge rows based on first 2 columns values. My original data from a file as below: Original data content ======================== ID1 ID2 Field1 Field2 1 1 11F1 11F2 1 2 12F1 12F2 2 1 21F1 21F2 2 2 22F1 22F2 ID1 ID2 Field3 Field4 1 1 11F3 11F4 1 2 12F3 12F4 2 1 21F3 21F4 2 2 22F3 22F4 ID1 ID2 Field5 Field6 1 1 11F5 11F6 1 2 12F5 12F6 2 1 21F5 21F6 2 2 22F5 22F6 As you noticed, columns are split into different

parsing strings using either grep,awk or sed

笑着哭i 提交于 2020-01-15 05:33:27
问题 I have a file with lines like below 17:59:49.987 - JobID 864563: Found 7 clips from SeqID 862753 17:59:49.987 - Processing Job 864562 17:59:50.003 - JobID 864561: Location 14695 applied clip data successfully. Updating OCAMT_GM_Sent 17:59:50.003 - Processing Job 864563 17:59:50.003 - JobID 864564 17:59:50.018 - JobID 864565 17:59:50.034 - Processing Job 864565 17:59:50.034 - JobID 864566 17:59:50.034 - JobID 864562 17:59:50.034 - JobID 864563 17:59:50.034 - Processing Job 864566 17:59:50.049

Combine multiple awk output to print on one line

生来就可爱ヽ(ⅴ<●) 提交于 2020-01-15 03:27:06
问题 I have two different functions I wrote, each with their own AWK to search for a specific file in multiple directories and output the information I need. Both functions print the FILENAME and a specific field I need and work just fine on their own. I want to combine them together for more robust output. I am not a programmer. I wrote these while reading about awk as I went. Function 1 cver () { X="" case $1 in ("-b") X="bb";; ("-c") X="cpe";; ("-e") X="etech";; ("-k") X="core";; ("-o") X=

optimizing loop, passing parameters from external file, naming array arguments within awk

不羁的心 提交于 2020-01-15 03:27:06
问题 Am an awk newbie. Using Windows-GNU gawk in UNXUTILS. Have 2 kinds of records arranged sequentially in date and time order in my file(s), 30-field Order records (start with "O") where quantity is the 15th field, and 18-field Trade records (start with "T") where quantity is the 8th field. The underlying research data is historical-archival Indian stock market data spanning 15 days in April 2006, about 1000 firms, and comprising in all about 100 million separate order or trade records. My test

in place editing using awk

邮差的信 提交于 2020-01-14 13:25:28
问题 I want to add a line at top of file say f1 using awk. Is there a better way than the following? awk 'BEGIN{print "word"};{print $0}' f1 > aux;cp aux f1;\rm aux<br/> Does awk has something like -i option in sed? 回答1: Why not use sed - it would make the solution more straightforward $sed -i.bak '1i\ word ' <filename> 回答2: An alternate way to do this is: sed -i '1s:^: Word1\nWord2 :' file 来源: https://stackoverflow.com/questions/910121/in-place-editing-using-awk

Print a list of fields passed as a variable

南笙酒味 提交于 2020-01-14 08:46:12
问题 I have a file say a.txt and below are the contents of it: bob 1 100 lincoln 2 200 chris 3 300 The file contents are separated by space. Using awk, I could access each column. Using below command to print 1st and 3rd columns separated by comma: cat a.txt | awk ' { print $1","$3} ' and I was successful. Now I want to pass the criteria dynamically from another shell script. By saying criteria I mean - $1","$3 . I tried the below command but it didn't work. myvar="$1" awk -v a="$myvar" ' { print

How to get the validate the count with the group by data in unix

假装没事ソ 提交于 2020-01-14 06:05:28
问题 I have a list of records as following Source: a,yes a,yes b,No c,N/A c,N/A c,N/A d,xyz d,abc d,abc Output: a, Yes 2 b, No 1 c, N/A 3 d, xyz 1 d, abc 2 c, N/A "File is not correct" Here 'Yes' and 'No' are the acceptable words, If any other word count is greater than the 'Yes' or 'No' word count for an individual $1 value then we have issue a statement like "file is not good" I have tried the below script awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}' filetest.txt 回答1: If you are not