我需要提高我下面的脚本,这需要一个包含近百万独特线的输入文件。对每行,它在3查找文件不同的价值观,我打算在我的输出添加为逗号分隔值。
下面的脚本能正常工作,但它需要时间来完成这项工作。我在寻找一个真正的快速解决方案,也将在系统上少沉重。
#!/bin/bash
while read -r ONT
do
{
ONSTATUS=$(grep "$ONT," lookupfile1.csv | cut -d" " -f2)
CID=$(grep "$ONT." lookupfile3.csv | head -1 | cut -d, -f2)
line1=$(grep "$ONT.C2.P1," lookupfile2.csv | head -1 | cut -d"," -f2,7 | sed 's/ //')
line2=$(grep "$ONT.C2.P2," lookupfile2.csv | head -1 | cut -d"," -f2,7 | sed 's/ //')
echo "$ONT,$ONSTATUS,$CID,$line1,$line2" >> BUwithPO.csv
} &
done < inputfile.csv
inputfile.csv包含如下所示的行:
343OL5:LT1.PN1.ONT1
343OL5:LT1.PN1.ONT10
225OL0:LT1.PN1.ONT34
225OL0:LT1.PN1.ONT39
343OL5:LT1.PN1.ONT100
225OL0:LT1.PN1.ONT57
lookupfile1.csv包含:
343OL5:LT1.PN1.ONT100, Down,Locked,No
225OL0:LT1.PN1.ONT57, Up,Unlocked,Yes
343OL5:LT1.PN1.ONT1, Down,Unlocked,No
225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes
225OL0:LT1.PN1.ONT39, Up,Unlocked,Yes
lookupfile2.csv包含:
225OL0:LT1.PN1.ONT34.C2.P1, +123125302766,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
225OL0:LT1.PN1.ONT57.C2.P1, +123125334019,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
225OL0:LT1.PN1.ONT57.C2.P2, +123125334819,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
343OL5:LT1.PN1.ONT100.C2.P11, +123128994019,REG,DigitMap,Unlocked,_media_ANT,FD_BSFU.xml,
lookupfile3.csv包含:
343OL5:LT1.PON1.ONT100.SERV1,12-654-0330
343OL5:LT1.PON1.ONT100.C1.P1,12-654-0330
343OL5:LT7.PON8.ONT75.SERV1,12-664-1186
225OL0:LT1.PN1.ONT34.C1.P1.FLOW1,12-530-2766
225OL0:LT1.PN1.ONT57.C1.P1.FLOW1,12-533-4019
输出是:
225OL0:LT1.PN1.ONT57, Up,Unlocked,Yes,12-533-4019,+123125334019,FD_BSFU.xml,+123125334819,FD_BSFU.xml
225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes,12-530-2766,+123125302766,FD_BSFU.xml,
343OL5:LT1.PN1.ONT1, Down,Unlocked,No,,,
343OL5:LT1.PN1.ONT100, Down,Locked,No,,,
343OL5:LT1.PN1.ONT10,,,,
225OL0:LT1.PN1.ONT39, Up,Unlocked,Yes,,,
正如你所看到的,瓶颈将是循环多次内执行grep
。你可以通过关联数组创建一个查表提高效率。
如果awk
可用,请尝试以下方法:
[更新]
#!/bin/bash
awk '
FILENAME=="lookupfile1.csv" {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME=="lookupfile2.csv" {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME=="lookupfile3.csv" {
split($0, a, ",")
if (match(a[1], ".+\\.ONT[0-9]+")) {
ont = substr(a[1], RSTART, RLENGTH)
cid[ont] = a[2]
}
}
FILENAME=="inputfile.csv" {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' lookupfile1.csv lookupfile2.csv lookupfile3.csv inputfile.csv > BUwithPO.csv
{编辑]
如果你需要指定的文件的绝对路径,请尝试:
#!/bin/bash
awk '
FILENAME ~ /lookupfile1.csv$/ {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME ~ /lookupfile2.csv$/ {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME ~ /lookupfile3.csv$/ {
split($0, a, ",")
if (match(a[1], ".+\\.ONT[0-9]+")) {
ont = substr(a[1], RSTART, RLENGTH)
cid[ont] = a[2]
}
}
FILENAME ~ /inputfile.csv$/ {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' /path/to/lookupfile1.csv /path/to/lookupfile2.csv /path/to/lookupfile3.csv /path/to/inputfile.csv > /path/to/BUwithPO.csv
希望这可以帮助。
如果你在评论中已经指出,你不能使用而缺少gensub
由GNU awk
提供由于@tshiono提供的解决方案,你可以用两个调用替换gensub
用临时变量来sub
完成修整所需的后缀。
例:
awk '
FILENAME=="lookupfile1.csv" {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME=="lookupfile2.csv" {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME=="lookupfile3.csv" {
split($0, a, ",")
# ont = gensub("(\\.ONT[0-9]+).*", "\\1", 1, a[1])
sfx = a[1]
sub(/^.*[.]ONT[^.]*/, "", sfx)
sub(sfx, "", a[1])
# cid[ont] = a[2]
cid[a[1]] = a[2]
}
FILENAME=="inputfile.csv" {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' lookupfile1.csv lookupfile2.csv lookupfile3.csv inputfile.csv > BUwithPO.csv
我已注释到gensub
相关的部分中的使用FILENAME=="lookupfile3.csv"
的并与两个呼叫取代gensub
表达使用sub
(后缀)作为临时变量来sfx
。
给它一个尝试,让我知道,如果你能使用的。
下面的脚本类似于awk
解决办法,但用Perl编写的。将其保存为filter.pl
并使其可执行。
#!/usr/bin/env perl
use strict;
use warnings;
my %lookup1;
my %lookup2_1;
my %lookup2_2;
my %lookup3;
while( <> ) {
if ( $ARGV eq 'lookupfile1.csv' ) {
# 225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
if (/^([^,]+),\s*(.*)$/) {
$lookup1{$1} = $2;
}
} elsif ( $ARGV eq 'lookupfile2.csv' ) {
# 225OL0:LT1.PN1.ONT34.C2.P1, +123125302766,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^
if (/^(.+ONT\d+)\.C2\.P1,\s*([^,]+),(?:[^,]+,){4}([^,]+)/) {
$lookup2_1{$1} = "$2,$3";
} elsif (/^(.+ONT\d+)\.C2\.P2,\s*([^,]+),(?:[^,]+,){4}([^,]+)/) {
$lookup2_2{$1} = "$2,$3";
}
} elsif ( $ARGV eq 'lookupfile3.csv' ) {
# 225OL0:LT1.PN1.ONT34.C1.P1.FLOW1,12-530-2766
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^
if (/^(.+ONT\d+)[^,]+,\s*(.*)$/) {
$lookup3{$1} = $2;
}
} else { # assume 'inputfile.csv'
no warnings 'uninitialized'; # because not all keys ($_) have values in the lookup tables
# 225OL0:LT1.PN1.ONT34
chomp;
print "$_,$lookup1{$_},$lookup3{$_},$lookup2_1{$_},$lookup2_2{$_}\n";
}
}
执行它像这样:
./filter.pl lookupfile{1,2,3}.csv inputfile.csv > BUwithPO.csv
这是很重要的lookupfiles放在第一位(如awk
解决方案,顺便说一句。),因为他们建立了4名词典(Perl中的说法哈希)%lookup1
,%lookup2_1
等,然后从inputfile.csv
值是针对这些字典匹配。