How to read a gob file line by line

Question

I'm storing Message (struct defined below) inside a file using Gob serialization.

type Message struct {
    Message string `json:"message"`
    From    string `json:"from"`
}

I managed to do this by putting my Message inside a slice that I serialized using gob, then I store this serialized slice inside a file.
But, by doing this way I need to load my entire serialized slice from the file, decode it, append the new Message, Encode the slice and save it once again inside the file. This seems complexe and not well optimized to me..

Function I use to encode / decode and write / read

func (m Message) Encode() ([]byte, error) {
    var res bytes.Buffer
    encoder := gob.NewEncoder(&res)
    err := encoder.Encode(m)

    if err != nil {
        return []byte{}, err
    }
    return res.Bytes(), nil
}

func (m Message) Write(path string) error {
    messages, err := Read(path)
    if err != nil {
        return err
    }

    messages = append(messages, m)

    f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0644)
    if err != nil {
        return err
    }
    defer f.Close()

    encoder := gob.NewEncoder(f)
    encoder.Encode(messages)

    return nil
}

func Read(path string) ([]Message, error) {
    f, err := os.OpenFile(path, os.O_RDWR, 0644)
    if err != nil {
        return []Message{}, err
    }
    defer f.Close()

    m := []Message{}
    decoder := gob.NewDecoder(f)

    if err = decoder.Decode(&m); err != nil {
        if err == io.EOF {
            return []Message{}, nil
        }
        return []Message{}, err
    }

    return m, nil
}

A solution would be to store serialized Message directly inside the file and simply append new Message at the end.

I achived by using os.O_APPEND to append instead of overwrite the entiere file :

    f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)

I also made others basics changes like replace []Message with Message and so on..

Now I'm able to store Message inside my file and simply append new message at the end of the file without rewritting the entiere file each time.

But I have to idea how to read Message stored inside the file.
Previous code just read the first message and ignore the rest of the file

I found many solutions to read a file line by line but none seems to work with gob serialized object

Is it possible to read a file storing gob serialized object line by line ? Or do I have to stay with my current solution i.e storing a serialized slice ?

Note : I found this topic (Retrieving gobs written to file by appending several times) which look to describe same type of issue but it's almost from 7 years ago + describe a little bit more complexe issue

Answer 1

I post this as an "answer" but my issue isn't solved yet, I can move this inside my initial post if you think this is more appropriated.

I made some test starting from code I found here (Retrieving gobs written to file by appending several times) and tried to take into consideration the answer.

I wrote 2 new function to write gob inside a file and to read a given number of gob from a file :

func write(enc *gob.Encoder, m Message) {
    err := enc.Encode(m)
    if err != nil {
        panic(err)
    }
}

func read(filename string, to_load int) {
    f, err := os.OpenFile(filename, os.O_RDWR, 0644)
    defer f.Close()
    if err != nil {
        panic(err)
    }

    dec := gob.NewDecoder(f)

    for i := 0; i < to_load; i++ {
        var m Message
        err = dec.Decode(&m)
        if err != nil {
            panic(err)
        }
        fmt.Println("loaded struct:", m)
    }
}

Something I "understood" from the the answer (https://stackoverflow.com/a/36386843/17070383) is that it can be complicated to read many gob from the same file if all these gob were written with differente instance of gob.Encoder.

So I wrote a function that generate and return a gob.Encoder

func getEncoder(fileName string) (*gob.Encoder, *os.File) {
    file, err := os.OpenFile(fileName, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0666)
    if err != nil {
        panic(err)
    }

    return gob.NewEncoder(file), file
}

Then :

func main() {
    enc, f := getEncoder("test.bin")
    defer f.Close()

    m1 := Message{"Bob", "Hello"}
    m2 := Message{"Bob2", "Hello2"}
    m3 := Message{"Bob3", "Hello3"}

    write(enc, m1)
    write(enc, m2)
    write(enc, m3)

    fmt.Println("Read 2 Message from file : ")
    read("test.bin", 2)

    m4 := Message{"Bob4", "Hello4"}

    write(enc, m4)

    fmt.Println()
    fmt.Println("Read 4 Message from file : ")
    read("test.bin", 4)
}

Output :

Read 2 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}

Read 4 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}
loaded struct: {Bob3 Hello3}
loaded struct: {Bob4 Hello4}

Well, it looks to work perfectly.
Now if I re launch the code, I try to read 8 gob instead of 4 (4 wrote by first code execution and 4 wrote by the seconde execution)

read("test.bin", 8)

I have this output :

Read 2 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}

Read 4 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}
loaded struct: {Bob3 Hello3}
loaded struct: {Bob4 Hello4}
panic: gob: duplicate type received

goroutine 1 [running]:
main.read({0x10df092?, 0xc000012018?}, 0x8)
    /{path}/main.go:34 +0x1c8
main.main()
    /{path}/main.go:72 +0x2c5
exit status 2

It read the first 4 gob written from the first execution then panic when reading the 5th element, written by the second execution with a new instancde of gob.Encoder

To verify this I changed my code and write 3 gob with a first gob.Encoder and a last one with another encoder, then try to to read the 4 gob :

func main() {
    enc, f := getEncoder("test.bin")
    defer f.Close()

    m1 := Message{"Bob", "Hello"}
    m2 := Message{"Bob2", "Hello2"}
    m3 := Message{"Bob3", "Hello3"}

    write(enc, m1)
    write(enc, m2)
    write(enc, m3)

    fmt.Println("Read 2 Message from file : ")
    read("test.bin", 2)

    new_enc := gob.NewEncoder(f) // New encoder

    m4 := Message{"Bob4", "Hello4"} 

    write(new_enc, m4)// Write m4 with new encoder

    fmt.Println()
    fmt.Println("Read 4 Message from file : ")
    read("test.bin", 4)
}

NOTE : I reseted the file "test.bin"

Output :

Read 2 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}

Read 4 Message from file : 
loaded struct: {Bob Hello}
loaded struct: {Bob2 Hello2}
loaded struct: {Bob3 Hello3}
panic: gob: duplicate type received
...

As we can see the 4th (wrote with seconde encoder) can't be read

Conclusion :

I can't really explain why it works like that, but it seems not to be possible too store (and read) gob serialized struct directly inside a file and append new gob at the end everytime there is a new record to save.

I'm really new to Go and I would be pleased to have more explaination about that

I found 2 piece of solution :

1. Each time the app start you create a new file and instantiate a new gob.Encoder linked to this file to write your gob inside

2. Same as first solution, but you don't create a new file each time, you just load the content of the file and rewrite the content insidethe same file using the new gob.Encoder. Then each time there is a new record you can use this gob.Encoder again to append your gob to the file.

Both solution look "bad" as it means you have to keep a stream open (with the file) during the entiere execution of the app. I'm not familiar with that but it looks to be something you should avoid...

Feel free to complete my post with all your knowlage !

nem0z