Flatbuffers VerifySizePrefixedBuffer from stream

Question

This question is about the correct pattern for reading size prefixed Flatbuffers from a stream in C++.

Consider a data stream - ie. TCP where messages are are delimited by their size prefix. I was initially under the impression that these could be extracted with the following code:

while (true) {
  flatbuffers::Verifier v(buf.data(), buf.size());
  if (v.VerifySizePrefixedBuffer<Message>(nullptr)) {
    const flatbuffers::uoffset_t off = flatbuffers::GetPrefixedSize(buf.data());
    const auto msg = flatbuffers::GetSizePrefixedRoot<Message>(buf.data());

    // Use message

    buf.erase(buf.begin(), buf.begin() + 4 + off);
  } else {
    break;
  }
}

However, this proves to be incorrect - VerifySizePrefixedBuffer checks that the size of the buffer provided to the verifier is equal to the size prefix, rather than "at least".

From verifier.h:

Check(ReadScalar<uoffset_t>(buf_) == size_ - sizeof(uoffset_t))

This means that if there are multiple messages in the buffer, we cannot use the verifier to check that just the first one is valid. Instead, the message size must be read first and provided to the verifier:

while (buf.size() >= 4) {
  const flatbuffers::uoffset_t off = flatbuffers::GetPrefixedSize(buf.data());
  flatbuffers::Verifier v(buf.data(), 4 + off);
  if (v.VerifySizePrefixedBuffer<Message>(nullptr)) {
    const auto msg = flatbuffers::GetSizePrefixedRoot<Message>(buf.data());

    // Use message

    buf.erase(buf.begin(), buf.begin() + 4 + off);
  } else {
    break;
  }
}

This seems to be duplicating the work of the verifier to check the validity of the prefix. Is there some other pattern I should be using or is this the expected usage?

Answer 1

The verifier only works on a "completed" buffer, it wouldn't be able to verify a partial buffer. It verifies that all the offsets point to a valid region in the buffer, i.e., you won't attempt to read data not in the buffer.

Say if your first chunk comes in and an Offset in that chuck points to some region in the second chunk. The verifier would fail saying that the Offset is out of bounds of the buffer it has so far.

The Size Prefix is to help your chunking code to properly receive the length of the buffer and stitch the chunks together. You should use GetPrefixedSize() to get the expected size, and then use that to stitch the chunks together until you read that much data. After that, you can use the sized prefix verifier.

You want something like:

const auto expected_size = flatbuffers::GetPrefixedSize(buf.data());
while(buf.size() < expected_size) {
  append_to_buf(buf); // however you get your next chunk
}

flatbuffers::Verifier v(buf.data(), expected_size + sizeof(expected_size));
if (!v.VerifySizePrefixedBuffer<Message>()) {
  // ERROR in verification
}
const auto msg = flatbuffers::GetSizePrefixedRoot<Message>(buf.data());

// Use the msg.

// Optionally, consume the buffer to start reading the next message if you have more than one.
buf.erase(buf.begin(), buf.begin() + sizeof(expected_size) + expected_size);