寻找类似的文本文件

问题描述 投票:3回答:4

有没有人有一个特别优雅的命令行(Linux和OS X)的方法来确定在给定的目录“文本上类似的”文件?

通过“文本上类似的”,我的意思是文件应该在N多行的唯一不同。

linux macos sed awk diff
4个回答
1
投票

用awk

diff file1 file2 |awk '!/^<|^>|^-/{a=$0;lt[a]=0;gt[a]=0;next}    # Use label (not start from <,>,---) and set the array lt and gt
     /</{lt[a]++}                                                # if has differ "<", sum it into array lt
     />/{gt[a]++}                                                # if has differ ">", sum it into array gt
END{for (i in lt) 
       sum+=lt[i]>gt[i]?lt[i]:gt[i]                              # compare "<" or ">" lines, take the max and add in variable sum
       printf "Files have differs in %d lines\n",sum             # Do the print job.
       if (sum<3) {print "So files are similar" }
       else{print "So files are not similar"}
    }'

如果有两行不同,“如果(和<3)”,我会认为这些文件是不相似的,你可以自己定义编号,例如,在我的命令。

测试结果。

$ cat file1
a
b
a
d
b
c
c

$ cat file2
a
b
d
b
d
c
d
f

$ diff file1 file2
3d2
< a
5a5
> d
7,8c7,8
< c
<
---
> d
> f

$  diff file1 file2 |awk '!/^<|^>|^-/{a=$0;lt[a]=0;gt[a]=0;next}/</{lt[a]++}/>/{gt[a]++}END{for (i in lt) sum+=lt[i]>gt[i]?lt[i]:gt[i];printf "Files have differs in %d lines\n",sum;if (sum<3) {print "So files are similar" }else{print "So files are not similar"}}'

Files have differs in 4 lines
So files are not similar

1
投票

下面是使用统一diffwc计算不同线路一个粗略的办法。 Grep用于滤除DIFF上下文:

diff -U 0  file1 file2  | grep -v ^@ | grep -v ^--- | grep -v ^+++ | wc -l

0
投票

也许PMD是什么你正在寻找:https://pmd.github.io

它的维护和使用也很简单。

你可能想复制的代码检测:https://pmd.github.io/pmd-5.5.5/usage/cpd-usage.html(如果您定位代码或简单的纯文本这不是你的问题清楚,但我不明白为什么它不应该在这两种情况下工作)。


0
投票

使用Terraform意味着有很多从其他文件,只取得了一些变化复制的文件。这是很无奈,以找出其中一个文件是从当你想看看有什么特别的地方复制。我做了一个工具,我叫similarities.sh帮我找出一个文件是多么相似的一组人的每个文件。

#!/bin/bash

fileA="$1"
shift
for fileB in "$@"; do
    (
        # diff once grep twice with the help of tee and stderr
        diff $fileA $fileB | \
            tee >(grep -cE '^< ' >&2) | \
                  grep -cE '^> ' >&2
    # recapture stderr
    ) 2>&1 | (
        read -d '' diffA diffB;
        printf "The files %s and %s have %s:%s diffs out of %s:%s lines.\n" \
            $fileA $fileB $diffA $diffB $(wc -l < $fileA) $(wc -l < $fileB)
    )
done | column -t

这是在行动:

$ similarities.sh terraform.tfvars ../*/terraform.tfvars
The  files  terraform.tfvars  and  ../api_proxy/terraform.tfvars                   have  3:3   diffs  out  of  51:51  lines.
The  files  terraform.tfvars  and  ../cf-ip-location-lookup/terraform.tfvars       have  4:12  diffs  out  of  51:59  lines.
The  files  terraform.tfvars  and  ../cf-region-cookie-setter/terraform.tfvars     have  4:8   diffs  out  of  51:55  lines.
The  files  terraform.tfvars  and  ../cf-switch-region-origin/terraform.tfvars     have  4:10  diffs  out  of  51:57  lines.
The  files  terraform.tfvars  and  ../reformat_devops_alerts/terraform.tfvars      have  0:0   diffs  out  of  51:51  lines.
The  files  terraform.tfvars  and  ../restart_location/terraform.tfvars            have  17:3  diffs  out  of  51:37  lines.
The  files  terraform.tfvars  and  ../warehouse-availability-etl/terraform.tfvars  have  3:3   diffs  out  of  51:51  lines.
© www.soinside.com 2019 - 2024. All rights reserved.