Issues with LevelDB Database Size Reduction in Go (levigo)

Question

Hello Stack Overflow community,

I'm currently working on a Go program that utilizes LevelDB for data storage using the levigo package. My goal is to manage the database size efficiently, particularly by deleting old records when the available storage is running low. However, I've observed an unexpected behavior: after deleting records, the LevelDB database folder size does not decrease proportionally.

Here's a simplified version of the code to reproduce the issue:

Saving Data Code:

package main

import (
    "crypto/rand"
    "fmt"
    "log"

    "github.com/jmhodges/levigo"
)

func main() {
    // Specify the LevelDB options
    options := levigo.NewOptions()
    cache := levigo.NewLRUCache(5 << 20)
    options.SetCache(cache)
    options.SetCreateIfMissing(true)
    options.SetMaxOpenFiles(100)

    // Open or create the LevelDB database
    db, _ := levigo.Open("/tmp/mydatabase", options)
    defer db.Close()

    dataSize := 1024 * 1024 * 5 // 5MB
    randomData := make([]byte, dataSize)
    rand.Read(randomData)

    // Enqueue 5 pieces of data
    for i := 1; i <= 5; i++ {
        key := []byte(fmt.Sprintf("key%d", i))

        // Write the batch to the database
        if err := db.Put(levigo.NewWriteOptions(), key, randomData); err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Enqueued: %s \n", key)
    }

    fmt.Println("Enqueue completed.")
}

Deleting Data Code:

package main

import (
    "fmt"
    "log"

    "github.com/jmhodges/levigo"
)

func main() {
    // Specify the LevelDB options
    options := levigo.NewOptions()
    cache := levigo.NewLRUCache(5 << 20)
    options.SetCache(cache)
    options.SetCreateIfMissing(true)
    options.SetMaxOpenFiles(100)

    // Open or create the LevelDB database
    db, _ := levigo.Open("/tmp/mydatabase", options)
    defer db.Close()

    // Dequeue (remove) the 3 pieces of data
    for i := 1; i <= 3; i++ {
        key := []byte(fmt.Sprintf("key%d", i))

        // Create a WriteOptions for deleting from the database
        wo := levigo.NewWriteOptions()
        defer wo.Close()

        // Delete the key from the database
        if err := db.Delete(wo, key); err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Dequeued: %s\n", key)
    }

    fmt.Println("Dequeue completed.")
}

After running the code to save 5 items, the database folder size is 30MB. Subsequently, when I run the code to delete 3 items, the folder size reduces to 26MB. I expected a more significant decrease in size, considering the amount of data deleted.

I've set up LevelDB options such as cache size and file limits, but it seems that I might be missing something in the configuration or the deletion process.

Questions:

1. What could be causing the LevelDB database folder size not to decrease proportionally after deleting records?
2. Are there additional configurations or optimizations I should consider to manage the database size more effectively?
3. Is there a specific method in levigo for compacting the database to release the unused space?

Any insights or guidance on resolving this issue would be greatly appreciated. Thank you!

Answer 1

By reading this issue on level DB repo I realized that I can add this line at the end of the deletion loop db.CompactRange(levigo.Range{})
so the DB will delete the unused data and the total DB folder size is decreasing according to it.