SQLite slower than expected when querying

350 views Asked by At

I have a fairly large (3,000,000 rows) SQLite database. It consist of one table. The table has an integer id column, a text-based tag column, a timestamp column saved as an int, and 15 double number columns. I have a unique index on the tag and timestamp columns, since I always look entries up using both.

I need to run though the database and do quite a few calculations. Mainly calling a bunch of select statements. The complexity of the select statements is really simple. I am using the GRDB library.

Here is an example query.

do {
    try dbQueue.read { db in
        let request = try DataObject
            .filter(Columns.tag == tag)
            .filter(Columns.dateNoDash = date)
            .fetchOne(db)
    }
} catch { Log.msg("Was unable to query database. Error: \(error)") }

When I run the debugged trace on the queries my program generates (using explain query plan), I can see that the index is being used.

I have to loop over a lot of queries, so I benchmarked a segment of the queries. I am finding that 600 queries roughly take 28 seconds. I am running the program on a 10-core iMac Pro. This seems slow. I was always under the impression that SQLite was faster.

The other code in the loop basically adds certain numbers together and possible creates an average, so nothing complex and computationally expensive.

I tried to speed things up by adding the following configuration to the database connection.

var config = Configuration()
config.prepareDatabase { db in
    try db.execute(sql: "PRAGMA journal_mode = MEMORY")
    try db.execute(sql: "PRAGMA synchronous = OFF")
    try db.execute(sql: "PRAGMA locking_mode = EXCLUSIVE")
    try db.execute(sql: "PRAGMA temp_store = MEMORY")
    try db.execute(sql: "PRAGMA cache_size = 2048000")
}
let dbQueue = try DatabaseQueue(path: path, configuration: config)

Is there anything I can do to speed things up? Is GRDB slowing things down? Am I doing anything wrong? Should I be using a different database like mySQL or something?

Thanks for any tips/input

0

There are 0 answers