减少bcp导出的输出

问题描述 投票:2回答:4

在我们的项目中,我们使用bcp命令导出大约数百万行并将输出记录到输出文件中。对于bcp导入,我可以使用指定no的bcp开关来控制-b命令的输出。批量导入的行数。输出是这样的:

Starting copy...
1000 rows sent to SQL Server. Total sent: 1000
1000 rows sent to SQL Server. Total sent: 2000
1000 rows sent to SQL Server. Total sent: 3000
1000 rows sent to SQL Server. Total sent: 4000
1000 rows sent to SQL Server. Total sent: 5000
1000 rows sent to SQL Server. Total sent: 6000
1000 rows sent to SQL Server. Total sent: 7000
1000 rows sent to SQL Server. Total sent: 8000
1000 rows sent to SQL Server. Total sent: 9000
1000 rows sent to SQL Server. Total sent: 10000
1000 rows sent to SQL Server. Total sent: 11000
1000 rows sent to SQL Server. Total sent: 12000
SQLState = 22001, NativeError = 0

可以通过增加-b开关发送的数量轻松减少:

Starting copy...
10000 rows sent to SQL Server. Total sent: 10000
SQLState = 22001, NativeError = 0

12406 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 75     Average : (165413.3 rows per sec.)

但是对于bcp导出,我无法控制输出,对于一百万行,日志变得太大,例如。以下命令

bcp  Temp.dbo.TestTable out outdata.txt -t , -f file.fmt -S Server -U user-P password -m 10

输出:

Starting copy...
1000 rows successfully bulk-copied to host-file. Total received: 1000
SQLState = S1000, NativeError = 0
Error = [Microsoft][ODBC Driver 11 for SQL Server]Warning: BCP import with a format file will convert empty strings in delimited columns to NULL.
1000 rows successfully bulk-copied to host-file. Total received: 2000
1000 rows successfully bulk-copied to host-file. Total received: 3000
1000 rows successfully bulk-copied to host-file. Total received: 4000
1000 rows successfully bulk-copied to host-file. Total received: 5000
1000 rows successfully bulk-copied to host-file. Total received: 6000
1000 rows successfully bulk-copied to host-file. Total received: 7000
1000 rows successfully bulk-copied to host-file. Total received: 8000
1000 rows successfully bulk-copied to host-file. Total received: 9000
1000 rows successfully bulk-copied to host-file. Total received: 10000
1000 rows successfully bulk-copied to host-file. Total received: 11000
1000 rows successfully bulk-copied to host-file. Total received: 12000
1000 rows successfully bulk-copied to host-file. Total received: 13000
1000 rows successfully bulk-copied to host-file. Total received: 14000
1000 rows successfully bulk-copied to host-file. Total received: 15000
1000 rows successfully bulk-copied to host-file. Total received: 16000
1000 rows successfully bulk-copied to host-file. Total received: 17000
1000 rows successfully bulk-copied to host-file. Total received: 18000
1000 rows successfully bulk-copied to host-file. Total received: 19000
1000 rows successfully bulk-copied to host-file. Total received: 20000
1000 rows successfully bulk-copied to host-file. Total received: 21000
1000 rows successfully bulk-copied to host-file. Total received: 22000

我试过用-b传递bcp out开关,但它总是以1000的批量导出它们并通过grepingseding过滤它们将花费太多时间。谢谢你的帮助。

sql-server-2008 export-to-csv bcp
4个回答
2
投票

在bcp中似乎没有解决方案。但是,有一种解决方法;将bcp命令行打包到xp_cmdshell语句中并指定no_output选项:

EXEC xp_cmdshell "bcp  Temp.dbo.TestTable out outdata.txt -t , -f file.fmt -S Server -U user-P password -m 10", no_output

资料来源:click here


1
投票
  1. 您可以使用>将输出重定向到文件 bcp sometable out outfile -S Server -U user -P password > export.log 注意最后的> export.log位。这将使用log填充export.log。因此,如果您的命令失败,您可以检查。有关此方法的更多详细信息,请参阅here
  2. bcp还提供输出参数-o bcp sometable out outfile -S Server -U user -P password -o export.log 这次注意到-o export.log到底了。

0
投票

阻止命令行输出:

bcp Temp.dbo.TestTable out outdata.txt -t , -f file.fmt -S Server -U user-P password -m 10>nul


0
投票

我知道这是旧的,但我偶然发现它寻找相同的答案,并看到这里没有好的答案。我想出了以下内容,它让我得到了我想要的东西,并且认为我会在这里发布,以防它帮助其他人:

set @BCP_CMD = 'bcp ...etc...'; -- declare and set this appropriately
DROP TABLE IF EXISTS #bcpOutput;
CREATE TABLE #bcpOutput ([BCP_Output] varchar(2048)); 
Insert into #bcpOutput ([BCP_Output]) EXECUTE master..xp_cmdshell @BCP_CMD;
SELECT BCP_Output FROM #bcpOutput where 
-- Here I filter out Blank lines, all those "rows successfully..." lines, etc.
BCP_Output not like '' 
    AND BCP_Output not like '%rows successfully%' 
    AND BCP_Output not like '%S1000%' 
    AND BCP_Output not like '%empty strings in delimited%' 
    AND BCP_Output not like '%Starting copy%' 
    AND BCP_Output not like '%Network packet size%' ;

这会产生:

BCP_Output
69673 rows copied.
Clock Time (ms.) Total     : 406    Average : (171608.38 rows per sec.)

另一种方法在技术上“工作”就好了:追加

 | grep -v -e "rows successfully\|Starting copy\|...and so on..."

到你的“bcp ...”命令。不过,对我来说,它将<5秒的bcp操作变成了近30秒,这是不可接受的。也许在您的环境中,grep的管道效果更好。也许值得尝试一下。

© www.soinside.com 2019 - 2024. All rights reserved.