并行执行shell进程

问题描述 投票：42回答：5

是否有可用于在Windows批处理文件中并行执行多个进程的工具？我找到了一些有趣的Linux工具（parallel和PPSS），但是，我需要一个适用于Windows平台的工具。

额外奖励：如果该工具还允许在多台机器之间轻松分配流程，并远程运行流程，那就太棒了。

示例：我希望在以下for循环中使用它

PsExec

有限数量的processFile.exe实例并行运行以利用多核CPU。

windows batch-file shell parallel-processing multicore
5个回答
5
投票

Linux下的GNU xargs有一个“-P n”开关，可以并行启动“n”进程。

也许cygwin / mingw构建的xargs也支持这个？

然后你可以使用：

for %F in (*.*) do processFile.exe %F

但是，没有花哨的多节点进程产生。

51
投票

编辑 - 我修改了脚本以选择性地显示每个进程的输出

这是一个本机批处理解决方案，可以并行地可靠地运行命令列表，一次不会启动超过n个进程。

它甚至有一个内置的机制，通过PSEXEC将进程分发到特定的CPU或远程机器，但我还没有测试过这个功能。

使这项工作的技巧是通过CMD进程启动每个命令，该进程将stdout或未定义的句柄重定向到锁定文件。该进程将保持对文件的独占锁定，直到它终止。无论进程如何终止（正常退出，崩溃，终止进程），锁定都会立即释放。

主脚本可以通过尝试重定向到同一个锁文件来测试进程是否仍处于活动状态。如果进程仍处于活动状态，则重定向将失败，如果进程已终止，则重定向将成功。

默认情况下，脚本会忽略每个进程的输出。如果以xargs -P 4 processFile < fileList 选项作为第一个参数启动，则它显示每个进程的输出，而不进行交错。

我的演示将进程限制设置为4，并简单地运行一系列不同长度的PING命令。

我在XP，Vista和Windows 7上测试了这个。

/O

这是忽略过程输出的样本运行的输出

@echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: The list of commands could come from anywhere such as another file
:: or the output of another command. For this demo I will list the
:: commands within this script - Each command is prefixed with :::
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1

:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=4"

:: Optional - Define CPU targets in terms of PSEXEC specs
::           (everything but the command)
::
:: If a CPU is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \\server1 ...
:: set cpu2=psexec \\server1 ...
:: set cpu3=psexec \\server2 ...
:: etc.

:: For this demo force all CPU specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
:: Modify the IN () clause as needed to retrieve the list of commands
  set launch=1
  for /f "tokens=* delims=:" %%A in ('findstr /b ":::" "%~f0"') do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=%%A
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks^^!

如果使用显示过程输出的12:24:07.52 - proc1: starting ping /n 05 ::1 12:24:07.52 - proc2: starting ping /n 20 ::1 12:24:07.53 - proc3: starting ping /n 10 ::1 12:24:07.54 - proc4: starting ping /n 15 ::1 12:24:11.60 - proc1: finished ping /n 05 ::1 12:24:11.60 - proc1: starting ping /n 07 ::1 12:24:16.66 - proc3: finished ping /n 10 ::1 12:24:16.66 - proc3: starting ping /n 05 ::1 12:24:17.68 - proc1: finished ping /n 07 ::1 12:24:17.68 - proc1: starting ping /n 20 ::1 12:24:20.72 - proc3: finished ping /n 05 ::1 12:24:20.72 - proc3: starting ping /n 10 ::1 12:24:21.75 - proc4: finished ping /n 15 ::1 12:24:21.75 - proc4: starting ping /n 15 ::1 12:24:26.82 - proc2: finished ping /n 20 ::1 12:24:26.82 - proc2: starting ping /n 07 ::1 12:24:29.86 - proc3: finished ping /n 10 ::1 12:24:32.89 - proc2: finished ping /n 07 ::1 12:24:35.92 - proc4: finished ping /n 15 ::1 12:24:36.93 - proc1: finished ping /n 20 ::1 Thats all folks! 选项运行，则输出如下

/O
24
投票

试试------------------------------------------------------------------------------- 12:24:51.02 - proc1: starting ping /n 05 ::1 ------------------------------------------------------------------------------- 12:24:51.02 - proc2: starting ping /n 20 ::1 ------------------------------------------------------------------------------- 12:24:51.03 - proc3: starting ping /n 10 ::1 ------------------------------------------------------------------------------- 12:24:51.04 - proc4: starting ping /n 15 ::1 =============================================================================== 12:24:55.10 - proc1: finished ping /n 05 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 5, Received = 5, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:24:55.10 - proc1: starting ping /n 07 ::1 =============================================================================== 12:25:00.17 - proc3: finished ping /n 10 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:00.19 - proc3: starting ping /n 05 ::1 =============================================================================== 12:25:01.22 - proc1: finished ping /n 07 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:01.23 - proc1: starting ping /n 20 ::1 =============================================================================== 12:25:04.27 - proc3: finished ping /n 05 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 5, Received = 5, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:04.28 - proc3: starting ping /n 10 ::1 =============================================================================== 12:25:05.30 - proc4: finished ping /n 15 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 15, Received = 15, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:05.32 - proc4: starting ping /n 15 ::1 =============================================================================== 12:25:10.38 - proc2: finished ping /n 20 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 20, Received = 20, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:10.40 - proc2: starting ping /n 07 ::1 =============================================================================== 12:25:13.44 - proc3: finished ping /n 10 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:16.48 - proc2: finished ping /n 07 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:19.52 - proc4: finished ping /n 15 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 15, Received = 15, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:20.54 - proc1: finished ping /n 20 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 20, Received = 20, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== Thats all folks! ：

start

它打开一个带有给定标题的新窗口，并执行BAT，CMD或EXE文件。您还可以设置优先级，设置相同的环境等。

使用相关程序打开不可执行的文件。

进一步阅读：开始 - >运行

start "title of the process" "P:\ath\to.exe"

至少从WinME开始就可以使用Start。

祝好运！

19
投票

听起来更像是你想要使用Powershell 2.但是，你可以使用cmd /k start /? 生成新的cmd窗口（或其他进程），另请参阅start答案。虽然您可能必须使用其他一些工具和一些小技巧来创建类似“进程池”的东西（一次只能运行最多n个实例）。您可以通过使用this并计算已经有多少（tasklist /im loop或for，如果适用）来实现后者，并且只需等待（wc）并再次重新检查是否可以生成新进程。

为此，我拼凑了一小部分测试批次：

ping -n 2 ::1 >nul 2>&1

它最多会产生四个并行执行并最小化的新进程。等待时间可能需要调整，具体取决于每个进程的运行时间和运行时间。如果您正在执行其他操作，您可能还需要调整任务列表正在查找的进程名称。

但是，无法正确计算此批生成的进程。一种方法是在批处理开始时创建一个随机数（@echo off for /l %%i in (1,1,20) do call :loop %%i goto :eof :loop call :checkinstances if %INSTANCES% LSS 5 ( rem just a dummy program that waits instead of doing useful stuff rem but suffices for now echo Starting processing instance for %1 start /min wait.exe 5 sec goto :eof ) rem wait a second, can be adjusted with -w (-n 2 because the first ping returns immediately; rem otherwise just use an address that's unused and -n 1) echo Waiting for instances to close ... ping -n 2 ::1 >nul 2>&1 rem jump back to see whether we can spawn a new process now goto loop goto :eof :checkinstances rem this could probably be done better. But INSTANCES should contain the number of running instances afterwards. for /f "usebackq" %%t in (`tasklist /fo csv /fi "imagename eq wait.exe"^|find /c /v ""`) do set INSTANCES=%%t goto :eof ）并创建一个帮助程序批处理，用于处理（或生成处理程序）但可以将其窗口标题设置为参数：

%RANDOM%

这将是一个简单的批处理，它将其标题设置为第一个参数，然后使用第三个参数运行第二个参数。然后，您可以通过仅选择具有指定窗口标题（@echo off title %1 "%2" "%3" ）的进程来过滤任务列表。这应该相当可靠，并防止过多的误报。如果您仍然运行某些实例，那么搜索tasklist /fi "windowtitle eq ..."将是一个坏主意，因为这会限制您的工作进程池。

您可以使用cmd.exe创建一个合理的默认值来生成多少个实例。

您也可以轻松地对此进行调整以使用%NUMBER_OF_PROCESSORS%远程生成进程（但由于您必须在其他计算机上具有管理员权限以及在批处理中提供密码，因此不太可行）。但是，您必须使用进程名称进行过滤。

6
投票

有一个基本的Windows xargs-like-clone，它支持psexec上的-P并行处理选项

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.