有没有人有一个特别优雅的命令行(Linux和OS X)的方法来确定在给定的目录“文本上类似的”文件?
通过“文本上类似的”,我的意思是文件应该在N多行的唯一不同。
用awk
diff file1 file2 |awk '!/^<|^>|^-/{a=$0;lt[a]=0;gt[a]=0;next} # Use label (not start from <,>,---) and set the array lt and gt
/</{lt[a]++} # if has differ "<", sum it into array lt
/>/{gt[a]++} # if has differ ">", sum it into array gt
END{for (i in lt)
sum+=lt[i]>gt[i]?lt[i]:gt[i] # compare "<" or ">" lines, take the max and add in variable sum
printf "Files have differs in %d lines\n",sum # Do the print job.
if (sum<3) {print "So files are similar" }
else{print "So files are not similar"}
}'
如果有两行不同,“如果(和<3)”,我会认为这些文件是不相似的,你可以自己定义编号,例如,在我的命令。
测试结果。
$ cat file1
a
b
a
d
b
c
c
$ cat file2
a
b
d
b
d
c
d
f
$ diff file1 file2
3d2
< a
5a5
> d
7,8c7,8
< c
<
---
> d
> f
$ diff file1 file2 |awk '!/^<|^>|^-/{a=$0;lt[a]=0;gt[a]=0;next}/</{lt[a]++}/>/{gt[a]++}END{for (i in lt) sum+=lt[i]>gt[i]?lt[i]:gt[i];printf "Files have differs in %d lines\n",sum;if (sum<3) {print "So files are similar" }else{print "So files are not similar"}}'
Files have differs in 4 lines
So files are not similar
下面是使用统一diff
和wc
计算不同线路一个粗略的办法。 Grep
用于滤除DIFF上下文:
diff -U 0 file1 file2 | grep -v ^@ | grep -v ^--- | grep -v ^+++ | wc -l
也许PMD是什么你正在寻找:https://pmd.github.io
它的维护和使用也很简单。
你可能想复制的代码检测:https://pmd.github.io/pmd-5.5.5/usage/cpd-usage.html(如果您定位代码或简单的纯文本这不是你的问题清楚,但我不明白为什么它不应该在这两种情况下工作)。
使用Terraform意味着有很多从其他文件,只取得了一些变化复制的文件。这是很无奈,以找出其中一个文件是从当你想看看有什么特别的地方复制。我做了一个工具,我叫similarities.sh
帮我找出一个文件是多么相似的一组人的每个文件。
#!/bin/bash
fileA="$1"
shift
for fileB in "$@"; do
(
# diff once grep twice with the help of tee and stderr
diff $fileA $fileB | \
tee >(grep -cE '^< ' >&2) | \
grep -cE '^> ' >&2
# recapture stderr
) 2>&1 | (
read -d '' diffA diffB;
printf "The files %s and %s have %s:%s diffs out of %s:%s lines.\n" \
$fileA $fileB $diffA $diffB $(wc -l < $fileA) $(wc -l < $fileB)
)
done | column -t
这是在行动:
$ similarities.sh terraform.tfvars ../*/terraform.tfvars
The files terraform.tfvars and ../api_proxy/terraform.tfvars have 3:3 diffs out of 51:51 lines.
The files terraform.tfvars and ../cf-ip-location-lookup/terraform.tfvars have 4:12 diffs out of 51:59 lines.
The files terraform.tfvars and ../cf-region-cookie-setter/terraform.tfvars have 4:8 diffs out of 51:55 lines.
The files terraform.tfvars and ../cf-switch-region-origin/terraform.tfvars have 4:10 diffs out of 51:57 lines.
The files terraform.tfvars and ../reformat_devops_alerts/terraform.tfvars have 0:0 diffs out of 51:51 lines.
The files terraform.tfvars and ../restart_location/terraform.tfvars have 17:3 diffs out of 51:37 lines.
The files terraform.tfvars and ../warehouse-availability-etl/terraform.tfvars have 3:3 diffs out of 51:51 lines.