更快的方式导入Excel电子表格,以阵列ADO

问题描述 投票:2回答:2

我试图导入和排序从大的Excel报表数据为使用Excel 2007 VBA的新文件。我想出了两种方法,到目前为止这样做的:

  1. 让Excel实际打开文件(下面的代码),收集所有数据存入数组和输出数组到新表在同一个文件并保存/关闭它。 Public Sub GetData() Dim FilePath As String FilePath = "D:\File_Test.xlsx" Workbooks.OpenText Filename:=FilePath, FieldInfo:=Array(Array(2, 2)) ActiveWorkbook.Sheets(1).Select End Sub
  2. 使用ADO获得所有数据走出封闭的工作簿,整个数据表导入到一个数组(下面的代码)和排序从那里数据,然后输出数据到一个新的工作簿,并保存/关闭。 Private Sub PopArray() 'Uses ADO to populate an array that will be used to sort data Dim dbConnection As ADODB.Connection, rs As ADODB.Recordset Dim Getvalue, SourceRange, SourceFile, dbConnectionString As String SourceFile = "D:\File_Test.xlsx" SourceRange = "B1:Z180000" dbConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;" & _ "Data Source=" & SourceFile & ";" & _ "Extended Properties=""Excel 12.0 Xml;HDR=No"";" Set dbConnection = New ADODB.Connection dbConnection.Open dbConnectionString 'open the database connection Set rs = dbConnection.Execute("SELECT * FROM [" & SourceRange & "]") Arr = rs.GetRows UpBound = UBound(Arr, 2) rs.Close End Sub

所使用的测试文件大约有65000条记录进行排序通过(约我最终会使用它的三分之一)。我是那种失望的是,ADO版本只比开放工作进行稍微好一些(〜44秒VS〜40秒跑完时间)。我想知道是否有某种方式来提高ADO导入方法(或一个完全不同的方法 - ExecuteExcel4Macro可能 - 如果有的话),这将提高我的速度。我能想到的唯一的事情是,我使用"B1:Z180000"作为我SourceRange的,然后由设置Arr = rs.GetRows准确反映记录总数截断的最大射程。如果这是什么原因造成的慢了下来,我不知道我怎么会去寻找有多少行的表。

编辑 - 我使用的范围( “A1:A” &I)=(阵列)将数据插入到新的工作表。

excel vba excel-vba excel-2007 ado
2个回答
0
投票

这个答案可能不是你所期待的,但根据您的边注[...]或完全不同的方法]我还是觉得有必要将它张贴...]。

在这里,我有200MB(及以上)的文件时,每个仅仅是文本文件,包括分隔符。我不把它们加载到Excel了。我也有这样的Excel太慢,需要加载整个文件的问题。然而,Excel是在使用Open方法打开这些文件的速度非常快:

Open strFileNameAndPath For Input Access Read Lock Read As #intPointer

在这种情况下,Excel将不加载整个文件,但只是逐行读取它行。因此,Excel可以已处理数据(转发),然后获取数据的下一行。像这样的Excel不啃老族的内存加载200MB。

有了这个方法,我再装填在本地安装SQL将数据直接传送到我们的数据仓库(也SQL)的数据。为了加快使用上述mething和快速获取数据到我在每1000行数据块传输数据的SQL服务器转移。在Excel中的字符串变量最多可容纳2十亿个字符。所以,没有问题存在。

为什么如果我已经使用SQL的本地安装,我不能简单地使用SSIS人们可能不知道。然而,问题是,我不是一个加载所有这些文件了。使用Excel生成允许我这些工具转发给其他人,谁现在上传所有这些文件对于我这个“导入工具”。所有的人都给予访问SSIS不是一种选择,也没有使用注定的网络驱动器,其中一个可以放置这些文件和SSIS会自动加载它们(曾经10+分钟左右)的可能性。

在结束我的代码看起来是这样的。

Set conRCServer = New ADODB.Connection
conRCServer.ConnectionString = "PROVIDER=SQLOLEDB; " _
    & "DATA SOURCE=" & Ref.Range("C2").Value2 & ";" _
    & "INITIAL CATALOG=" & Ref.Range("C4").Value & ";" _
    & "Integrated Security=SSPI "
On Error GoTo SQL_ConnectionError
conRCServer.Open
On Error GoTo 0

'Save the name of the current file
strCurrentFile = ActiveWorkbook.Name

'Prepare a dialog box for the user to pick a file and show it
'   ...if no file has been selected then exit
'   ...otherwise parse the selection into it's path and the name of the file
Call Application.FileDialog(msoFileDialogOpen).Filters.Clear
Call Application.FileDialog(msoFileDialogOpen).Filters.Add("Extracts", "*.csv")
Application.FileDialog(msoFileDialogOpen).Title = "Select ONE Extract to import..."
Application.FileDialog(msoFileDialogOpen).AllowMultiSelect = False
intChoice = Application.FileDialog(msoFileDialogOpen).Show
If intChoice <> 0 Then
    strFileToPatch = Application.FileDialog(msoFileDialogOpen).SelectedItems(1)
Else
    Exit Sub
End If

'Open the Extract for import and close it afterwards
intPointer = FreeFile()
Open strFileNameAndPath For Input Access Read Lock Read As #intPointer

intCounter = 0
strSQL = vbNullString
Do Until EOF(intPointer)
    Line Input #intPointer, strLine
    If Left(strLine, 4) = """@@@" Then Exit Sub
    '*********************************************************************
    '** Starting a new SQL command
    '*********************************************************************
    If intCounter = 0 Then
        Set rstResult = New ADODB.Recordset
        strSQL = "set nocount on; "
        strSQL = strSQL & "insert into dbo.tblTMP "
        strSQL = strSQL & "values "
    End If
    '*********************************************************************
    '** Transcribe the current line into SQL
    '*********************************************************************
    varArray = Split(strLine, ",")
    strSQL = strSQL & " (" & varArray(0) & ", " & varArray(1) & ", N'" & varArray(2) & "', "
    strSQL = strSQL & " N'" & varArray(3) & "', N'" & varArray(4) & "', N'" & varArray(5) & "', "
    strSQL = strSQL & " N'" & varArray(6) & "', " & varArray(8) & ", N'" & varArray(9) & "', "
    strSQL = strSQL & " N'" & varArray(10) & "', N'" & varArray(11) & "', N'" & varArray(12) & "', "
    strSQL = strSQL & " N'" & varArray(13) & "', N'" & varArray(14) & "', N'" & varArray(15) & "' ), "
    '*********************************************************************
    '** Execute the SQL command in bulks of 1.000
    '*********************************************************************
    If intCounter >= 1000 Then
        strSQL = Mid(strSQL, 1, Len(strSQL) - 2)
        rstResult.ActiveConnection = conRCServer
        On Error GoTo SQL_StatementError
        rstResult.Open strSQL
        On Error GoTo 0
        If Not rstResult.EOF And Not rstResult.BOF Then
            strErrorMessage = "The server returned the following error message(s):" & Chr(10)
            While Not rstResult.EOF And Not rstResult.BOF
                strErrorMessage = Chr(10) & strErrorMessage & rstResult.Fields(0).Value
                rstResult.MoveNext
            Wend
            MsgBox strErrorMessage & Chr(10) & Chr(10) & "Aborting..."
            Exit Sub
        End If
    End If
    intCounter = intCounter + 1
Loop

Close intPointer

Set rstResult = Nothing

Exit Sub

SQL_ConnectionError:
Y = MsgBox("Couldn't connect to the server. Please make sure that you have a working internet connection. " & _
            "Do you want me to prepare an error-email?", 52, "Problems connecting to Server...")
If Y = 6 Then
    Set OutApp = CreateObject("Outlook.Application")
    Set OutMail = OutApp.CreateItem(0)
    With OutMail
        .To = Ref.Range("C7").Value2
        .CC = Ref.Range("C8").Value2
        .Subject = "Problems connecting to database '" & Ref.Range("C4").Value & "' on server '" & Ref.Range("C2").Value & "'"
        .HTMLBody = "<span style=""font-size:10px"">---Automatically generated Error-Email---" & _
                "</span><br><br>Error report from the file '" & _
                "<span style=""color:blue"">" & ActiveWorkbook.Name & _
                "</span>' located and saved on '<span style=""color:blue"">" & _
                ActiveWorkbook.Path & "</span>'.<br>" & _
                "Excel is not able to establish a connection to the server. Technical data to follow." & "<br><br>" & _
                "Computer Name:    <span style=""color:green;"">" & Environ("COMPUTERNAME") & "</span><br>" & _
                "Logged in as:     <span style=""color:green;"">" & Environ("USERDOMAIN") & "/" & Environ("USERNAME") & "</span><br>" & _
                "Domain Server:    <span style=""color:green;"">" & Environ("LOGONSERVER") & "</span><br>" & _
                "User DNS Domain:  <span style=""color:green;"">" & Environ("USERDNSDOMAIN") & "</span><br>" & _
                "Operating System: <span style=""color:green;"">" & Environ("OS") & "</span><br>" & _
                "Excel Version:    <span style=""color:green;"">" & Application.Version & "</span><br>" & _
                "<br><span style=""font-size:10px""><br>" & _
                "<br><br>---Automatically generated Error-Email---"
        .Display
    End With
    Set OutMail = Nothing
    Set OutApp = Nothing
End If
Exit Sub

SQL_StatementError:
Y = MsgBox("There seems to be a problem with the SQL Syntax in the programming. " & _
            "May I send an error-email to development team?", 52, "Problems with the coding...")
If Y = 6 Then
    Set OutApp = CreateObject("Outlook.Application")
    Set OutMail = OutApp.CreateItem(0)
    With OutMail
        .To = Ref.Range("C8").Value2
        '.CC = ""
        .Subject = "Problems with the SQL Syntax in file '" & ActiveWorkbook.Name & "'."
        .HTMLBody = "<span style=""font-size:10px"">" & _
                "---Automatically generated Error-Email---" & _
                "</span><br><br>" & _
                "Error report from the file '" & _
                "<span style=""color:blue"">" & _
                ActiveWorkbook.Name & _
                "</span>" & _
                "' located and saved on '" & _
                "<span style=""color:blue"">" & _
                ActiveWorkbook.Path & _
                "</span>" & _
                "'.<br>" & _
                "It seems that there is a problem with the SQL-Code within trying to upload an extract to the server." & _
                "SQL-Code causing the problems:" & _
                "<br><br><span style=""color:green;"">" & _
                strSQL & _
                "</span><br><br><span style=""font-size:10px"">" & _
                "---Automatically generated Error-Email---"
        .Display
    End With
    Set OutMail = Nothing
    Set OutApp = Nothing
End If
Exit Sub

End Sub

0
投票

我认为@Mr。卡罗是正确的,过去从Recordset您的数据到电子表格中是最简单的方法:

Private Sub PopArray()
    .....
    Set rs = dbConnection.Execute("SELECT * FROM [" & SourceRange & "]")  
    '' This is faster
    Range("A1").CopyFromRecordset rs
    ''Arr = rs.GetRows
End Sub

但如果你仍然想使用Arrays你可以试试这个:

Sub ArrayTest  

'' Array for Test
Dim aSingleArray As Variant  
Dim aMultiArray as Variant  

'' Set values 
aSingleArray = Array("A","B","C","D","E")  
aMultiArray = Array(aSingleArray, aSingleArray)

'' You can drop data from the Array using 'Resize'
'' Btw, your Array must be transpose to use this :P
Range("A1").Resize( _
            UBound(aMultiArray(0), 1) + 1, _  
            UBound(aMultiArray, 1) + 1) = Application.Transpose(aMultiArray)

End Sub
© www.soinside.com 2019 - 2024. All rights reserved.