perl中的并行处理

Question

我使用以下代码来运行多处理作业。启动多处理后，我试图从结果文件grep“RUN COMPLETED”，但grep操作在它到达“RUN COMPLETED”行之前完成。因为我的grep输出不正确。但是，如果我使用睡眠（20）;在$ pm-> wait_all_children之后; grep输出是正确的。任何人都可以让我知道如何等到我的所有工作完成，这样我就可以摆脱睡眠（20）。我想我需要等到我的工作完全执行才能完全生成结果文件但无法理解如何操作。请建议。

pJobs:
foreach my $test(sort (@test_files)) {
    $pm->start and next pJobs;
    my $Exec = $output."/Runsets/".$test;
    system ("chmod +x $Exec");
    print $LOG `lsf_bsub -J $test -o $output/Runsets/output.file < $Exec`, "\n";
    $pm->finish;
    }
    $pm->wait_all_children;
my $DH = IO::Dir->new("$output/Runsets") || die "ERROR : Can not access $output/Runsets: ($!)";
while ( defined (my $file = $DH->read)) {
    if($file =~ /^testcase\.(\d+)/) {
        my $Result_File = IO::File->new ("$output/Runsets/$file", 'r') || die "ERROR while tryed to open $output/Runsets/$file:($!)";
        grep {
            my $line = $_;
            chomp $line;
            if($line =~ /RUN COMPLETED/) {
                print $OUT "FOUND"
            }elsif ($line =~ /^ERROR:/) {
                print "ERROR - NOT FOUND";
            } 
        } <$Result_File> ;
        $Result_File->close;
   }
}

Answer 1

据我所知，您在多处理代码中唯一要做的就是更改文件的权限以赋予它们执行权限 - 您实际上并没有运行测试。因此，wait_all_children只等待文件权限更改 - 而不是测试完成。

---

我无法使用wait_all_children()复制您的问题，但您可以使用将在子进程终止时执行的回调，这很好，因为您可以开始处理文件而无需等到所有子进程执行完毕：

use strict;
use warnings; 
use 5.020;
use autodie;
use Data::Dumper;
use Parallel::ForkManager;

my %prog_names = (
    'a.pl' => 'a.txt',
    'b.pl' => 'b.txt',
    'c.pl' => 'c.txt',
    'd.pl' => 'd.txt',
);

my $max_procs = 3;
my $pm = new Parallel::ForkManager($max_procs);

sub process_file {
    my ($pid, $exit_code, $ident) = @_;

    say "** $ident just got out of the pool ".
        "with PID $pid and exit code: $exit_code";

    open my $INFILE, '<', $prog_names{$ident};
    my @lines = grep {$_ =~ /hello/} <$INFILE>;
    close $INFILE;    
    say for @lines;
}

$pm->run_on_finish(\&process_file);

PROG_NAME:
for my $prog_name (keys %prog_names) {

    system("chmod a+x $prog_name");

    #The arg for start() becomes $ident in the callback function
    my $pid = $pm->start($prog_name) and next PROG_NAME;

    # This code is the child process:
    system("/usr/bin/env perl $prog_name");

    #A numeric arg specified for finish() becomes $exit_code in the
    #callback function.
    $pm->finish;
}

$pm->wait_all_children;

啊.评论:

use strict;
use warnings; 
use 5.020;
use autodie;
use Data::Dumper;

sleep int(rand(10));

open my $OUTFILE, '>', 'a.txt'; 
say {$OUTFILE} "hello a.txt world";
close $OUTFILE;

不.评论:

use strict;
use warnings; 
use 5.020;
use autodie;
use Data::Dumper;

sleep int(rand(10));

open my $OUTFILE, '>', 'b.txt'; 
say {$OUTFILE} "hello b.txt world";
close $OUTFILE;

c.pl和d.pl：

Similar to others.

输出：

$ rm [a-d].txt
remove a.txt? y
remove b.txt? y
remove c.txt? y
remove d.txt? y

$ perl 1.pl 
** c.pl just got out of the pool with PID 23293 and exit code: 0
hello c.txt world

** b.pl just got out of the pool with PID 23290 and exit code: 0
hello b.txt world

** a.pl just got out of the pool with PID 23296 and exit code: 0
hello a.txt world

** d.pl just got out of the pool with PID 23299 and exit code: 0
hello d.txt world

$ cat [a-d].txt
hello a.txt world
hello b.txt world
hello c.txt world
hello d.txt world