System.Data.SQLite缓慢执行大脚本

Question

我正在尝试从我的.NET应用程序运行一个SQL文件，但运行时间非常慢。我不确定如何优化它以更快地运行。我知道该脚本有效，因为当我在SQLite shell应用程序中运行它时，它会在不到1秒的时间内执行。在我的.NET应用程序中，它只挂在sqlite_cmd.ExecuteNonQuery()上。

这是script.sql文件的摘录：

--
-- File generated with SQLiteStudio v3.1.1 on Thu Feb 15 11:33:12 2018
--
-- Text encoding used: System
--
PRAGMA foreign_keys = off;
BEGIN TRANSACTION;

-- Table: Document_Type
CREATE TABLE [Document_Type] (
    [Id]    integer PRIMARY KEY AUTOINCREMENT NOT NULL,
    [Type]  nvarchar(50) NOT NULL COLLATE NOCASE

);
INSERT INTO Document_Type (Id, Type) VALUES (1, '.docx');
INSERT INTO Document_Type (Id, Type) VALUES (2, '.xlsm');
INSERT INTO Document_Type (Id, Type) VALUES (3, '.jpeg');

-- Table: Learn_More
CREATE TABLE [Learn_More] (
    [item_ID]   integer PRIMARY KEY AUTOINCREMENT NOT NULL,
    [field_name]    nvarchar(100) COLLATE NOCASE,
    [description]   text(1073741823) COLLATE NOCASE,
    [location]  char(10) COLLATE NOCASE

);
INSERT INTO Learn_More (item_ID, field_name, description, location) VALUES (1, 'System Name', 'Full descriptive name of the system.*      Example: Agency Billing System ', 'SSP1      ');
.
.
.
.
.
COMMIT TRANSACTION;
PRAGMA foreign_keys = on;

script.sql文件是35,831行。我计划在使用.NET运行后清理列类型。

我的.NET代码：

Dim baseDir As String = Application.UserAppDataPath()
Dim db3Path As String = baseDir + "\Schema Scripts\ProgrammaticallyGenerateDatabase\Test.db3"
Dim sqlPath As String = baseDir + "\Schema Scripts\script.sql"

SQLiteConnection.CreateFile(db3Path)

Dim sqlite_conn = New SQLiteConnection("Data Source= " & db3Path)
Using sqlite_conn
    sqlite_conn.Open()
    Dim sqlite_cmd = sqlite_conn.CreateCommand()
    'Dim sqlite_tran = sqlite_conn.BeginTransaction()

    'Using sqlite_tran

        Using sqlite_cmd
            Try
                sqlite_cmd.CommandText = File.ReadAllText(sqlPath)
                sqlite_cmd.ExecuteNonQuery()

                sqlite_cmd.CommandText = "SELECT Type FROM Document_Type"

                Dim sqlite_datareader = sqlite_cmd.ExecuteReader()
                While (sqlite_datareader.Read())
                    Dim textReader As String = sqlite_datareader.GetString(0)

                    Console.WriteLine(textReader)
                End While
            Catch ex As Exception
                'End Using
                'sqlite_tran.Commit()
            End Try
        End Using
    sqlite_conn.Close()
End Using

我尝试过使用File.OpenText(sqlPath).ReadToEnd()，但性能问题相同。

关于如何提高速度的任何想法？重申一下，使用.read script.sql命令在SQLite shell程序中运行的同一文件不到1秒。

编辑：

这是一个胜利的应用程序。代码在后台线程上运行，需要483801毫秒才能完成。

我想出了一个适合我案例的解决方案。该查询现在运行在~700-900ms。我只是将sqlite3.exe与我的应用程序捆绑在一起并使用命令行调用它。

这是代码：

Private Sub SQLiteButtonPress()
        Dim baseDir As String = Application.UserAppDataPath() + "\Schema Scripts"
        Dim db3Path As String = baseDir + "\ProgrammaticallyGenerateDatabase\Test.db3"
        Dim sqlPath As String = baseDir + "\Baseline Schema.sql"

        Dim watch = System.Diagnostics.Stopwatch.StartNew()
        Console.WriteLine("Starting Stopwatch")

        Try
            Dim proc As New System.Diagnostics.ProcessStartInfo()
            proc.Arguments = "/c sqlite3.exe """ + db3Path + """ < """ + sqlPath + """"
            proc.FileName = "cmd.exe"
            proc.UseShellExecute = False
            proc.CreateNoWindow = True

            Dim p As New Process()
            p.StartInfo = proc
            p.Start()
            p.WaitForExit()

            Dim sqlite_conn = New SQLiteConnection("Data Source= " & db3Path)
            Using sqlite_conn
                sqlite_conn.Open()
                Dim sqlite_cmd = sqlite_conn.CreateCommand()

                Using sqlite_cmd
                        sqlite_cmd.CommandText = "SELECT Type FROM Document_Type"

                        Dim sqlite_datareader = sqlite_cmd.ExecuteReader()
                        While (sqlite_datareader.Read())
                            Dim textReader As String = sqlite_datareader.GetString(0)

                            Console.WriteLine(textReader)
                        End While
                End Using
                sqlite_conn.Close()
            End Using

        Catch ex As Exception
            Throw ex
        End Try

        watch.Stop()
        Dim elapsedMs = watch.ElapsedMilliseconds
        Console.WriteLine("Finished Stopwatch: " & elapsedMs & "ms")
    End Sub

Answer 1

我有同样的问题。来自@MarkBenningfield的建议解决了这个问题。正如他所提到的，在分号上拆分脚本并在循环中单独运行每个语句。在那之后，我的一切都很快。

Answer 2

出于某种原因，.NET SQLite库不会很快在单个字符串中运行大型查询，即使您将每个批量插入包装到事务中也是如此;或者在事务中包装整个“ExeecuteNonQuery”方法调用。

上述解决方案有效，将您的大型SQL文件拆分为单独的语句，在事务中执行所有这些，然后提交。我能够以0.22秒的方式将15,000条记录加载到基于内存的SQLite实例中。

干杯。