SQLiteBlobTooBigException在将请求分成1 MB的块（和cursor.close（））之后仍然发生，

问题描述 投票：0回答：1

可以将6MB的文本文件直接导入我的数据库。但是，由于CursorWindow的限制为2MB，因此无法提取文本。 （我应该使用过文件，但是一些用户已经遇到了这个问题，我需要阅读整个文本才能将其放入文件中）我使用substr（一种特殊的SQL函数）仅请求了1 MB的内存，并且可以正常工作。但是，以下while循环在第二次迭代后不起作用（这意味着即使我调用cursor.close（），也不会清空CursorWindow，因此对于第一次迭代，它只有1MB，但是在第二次迭代之后，有2MB，并抛出SQLiteBlobTooBigException异常）：

        //Load in chunks
        BookDbHelper bookDbHelper = new BookDbHelper(GlobalContext.get());
        SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
        //Query length
        int chunk_size = (int) Math.pow(2, 20);//mb
        String query_length = "SELECT _id, length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
        Cursor cursor_length = readableDatabase.rawQuery(query_length, null);
        cursor_length.moveToFirst();
        int length = cursor_length.getInt(1);
        cursor_length.close();
        bookDbHelper.close();
        readableDatabase.close();
        //Query text
        int numSteps = length / chunk_size + 1;
        int i = 0;
        while(i < numSteps) {
            BookDbHelper bookDbHelper2 = new BookDbHelper(GlobalContext.get());
            SQLiteDatabase readableDatabase2 = bookDbHelper2.getReadableDatabase();
            int from = i * chunk_size + 1;
            int to = (i + 1) * chunk_size + 1;
            //L.v(from + ", " + to);
            String query = "SELECT _id, substr(text," + from + "," + to + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
            Cursor cursor = readableDatabase2.rawQuery(query, null);
            //Read
            cursor.moveToFirst();
            String string = cursor.getString(1);
            cursor.close();
            bookDbHelper2.close();
            readableDatabase2.close();
            //stringBuilder.append(string);
            i++;
        }

相关的列是_id和text（包含一个非常大的字符串），而相关的sql函数是length（）（以了解必要的迭代次数）和substr（）（这样就不会发生SQLiteBlobTooBigException因为没有达到2MB的限制）。

我尝试关闭bookDbHelper和可读数据库，但没有帮助。

如何强制关闭CursorWindow，以便发出1MB的请求，清空CursorWinow，然后继续发出其他请求？

android sqlite exception android-cursor
1个回答
0
投票

如何强制关闭CursorWindow，以便发出1MB的请求，清空CursorWinow，然后继续发出其他请求？

我不认为关闭游标是您的问题，好像通过不关闭游标会追加到游标并展开。

您的问题在于您所构建的查询。

简而言之，substr函数不是from to，它是from for（for是返回的大小/长度字符串。您的计算基于第二个值（即字符的偏移量）。因此，提取的字符串的长度会减小，直到其超过字符串的末尾（在此之前先吹过CursorWindow），然后再增加块的大小。

因此，使用1MB的第二个块（如果将其视为使用偏移量）注定要在第二次运行中失败，因为它实际上是要提取的长度（2MB）。减小到小于1MB会留出一些余地，但可能会破坏CursorWindow（但会获取其他数据）。

但是，作为替代方案，它使用单个光标，每个块都作为伸出的行。姿态可以是：-

    //Load in chunks
    BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
    SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
    //Query length
    StringBuilder wholeBookText = new StringBuilder();
    int chunk_size = (int) Math.pow(2, 20);//mb
    String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
    Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
    int length = 0;
    if (cursor.moveToFirst()) {
        length = cursor.getInt(0);
    }
    int numSteps = length / chunk_size + 1;
    int i = 0;
    Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);

    StringBuilder sb = new StringBuilder();
    for (i=1; i < length + 1; i+= chunk_size) {
        if (sb.length() > 1) sb.append(" UNION ALL ");
        sb.append("SELECT substr(text,")
                .append(String.valueOf(i)).append(",").append(String.valueOf(chunk_size))
                .append(") FROM ").append(BookContract.TABLE_NAME)
                .append(" WHERE _id=").append(String.valueOf(id));

    }
    sb.append(";");
    Log.d("BOOKINFOV2","SQL generated :-\n\t" + sb.toString());
    cursor = readableDatabase.rawQuery(sb.toString(),null);
    wholeBookText = new StringBuilder();
    while (cursor.moveToNext()) {
        wholeBookText.append(cursor.getString(0));
        Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
    }

而不是单个查询循环运行。这将生成一个查询，该查询将每个块提取为一行。那就是在所有查询之间建立一个UNION。例如

SELECT substr(text,1,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
  • 摘自上述测试。
  • 可以看出，to（应该是for）是块的大小。最后一块将根据剩余数据被截断。

测试运行的完整输出：-

2019-12-16 14:21:35.546 D/BOOKINFOV2: SQL generated :-
        SELECT substr(text,1,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
2019-12-16 14:21:35.555 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.585 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 1048576
2019-12-16 14:21:35.599 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.616 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 2097152
2019-12-16 14:21:35.653 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 3145728
2019-12-16 14:21:35.654 D/BOOKINFO: Obtained String who's length is 51
        Total Extracted = 3145779
  • 如您所见，CursorWindow将溢出，但未添加该行，下次将添加并访问该行。

当然，您可以采用多种查询方法，在这种情况下，代码可以是：-

    //Load in chunks
    BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
    SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
    //Query length
    StringBuilder wholeBookText = new StringBuilder();
    int chunk_size = (int) Math.pow(2, 19);//mb
    chunk_size = (1024 * 1024);
    String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
    Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
    int length = 0;
    if (cursor.moveToFirst()) {
        length = cursor.getInt(0);
    }
    int numSteps = length / chunk_size + 1;
    int i = 0;
    Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);

    int from = 1, to = chunk_size;
    while (i < numSteps && length > 0) {
        if (to > length) to = length;
        String query = "SELECT substr(text," + from + "," + (chunk_size) + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
        Log.d("BOOKINFOSQL",query);
        cursor.close();
        cursor = readableDatabase.rawQuery(query, new String[]{String.valueOf(id)});
        //Read
        if (cursor.moveToFirst()) {
            wholeBookText.append(cursor.getString(0));
            Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
        }
        cursor.close();
        i++;
        from = (i * chunk_size) + 1;
        to = from + chunk_size;
    }
    if (!cursor.isClosed()) {
        cursor.close();
    }
    Log.d("BOOKINFO", "The length of the extracted data is " + wholeBookText.length());

以上结果为：-

2019-12-16 14:16:15.336 D/BOOKINFO: Length of Text is 3145779 Number of Chunks = 4 Chunk Size = 1048576
2019-12-16 14:16:15.336 D/BOOKINFOSQL: SELECT substr(text,1,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.358 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 1048576
2019-12-16 14:16:15.358 D/BOOKINFOSQL: SELECT substr(text,1048577,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.382 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 2097152
2019-12-16 14:16:15.383 D/BOOKINFOSQL: SELECT substr(text,2097153,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.409 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 3145728
2019-12-16 14:16:15.409 D/BOOKINFOSQL: SELECT substr(text,3145729,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.418 D/BOOKINFO: Obtained String who's length is 51
        Total Extracted = 3145779
2019-12-16 14:16:15.418 D/BOOKINFO: The length of the extracted data is 3145779
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.