Using protocol buffers for binary logging

We're thinking of using Protocol Buffers for binary logging because:

  • It's how we're encoding our objects anyway
  • It is relatively compact, fast to read / write etc.

That said, it isn't obvious how we should go about it because the APIs tend to focus on creating whole objects, so wrapping a list of DataLogEntry as a repeated field in a DataLogFile would be what you'd do in messaging terms but what we really want is just to be able to write and then read a whole DataLogEntry out, appending it to the end of a file.

The first issue we've hit by doing that is that doing this (in a test:

        FileInputStream fileIn = new FileInputStream(logFile);
        CodedInputStream in = CodedInputStream.newInstance(fileIn);
        while(!in.isAtEnd()) {
            DataLogEntry entry = DataLogEntry.parseFrom(in);
            // ... do stuff
        }

Only results in 1 DataLogEntry being read from the stream. Without the isAtEnd, it never stops.

Thoughts?

Edit: I've switched to using entry.writeDelimitedTo and BidLogEntry.parseDelimitedFrom and that seems to work...

Answer:1

From my understanding of protocol buffers it does not support multiple messages in a single stream. So you will probably need to track the boundaries of the messages yourself. you can do this by storing the size of the message before each message in the log.

public class DataLog {

    public void write(final DataOutputStream out, final DataLogEntry entry) throws IOException {
        out.writeInt(entry.getSerializedSize());
        CodedOutputStream codedOut = CodedOutputStream.newInstance(out);
        entry.writeTo(codedOut);
        codedOut.flush();
    }

    public void read(final DataInputStream in) throws IOException {
        byte[] buffer = new byte[4096];
        while (true) {
            try {
                int size = in.readInt();
                CodedInputStream codedIn;
                if (size <= buffer.length) {
                    in.read(buffer, 0, size);
                    codedIn = CodedInputStream.newInstance(buffer, 0, size);
                } else {
                    byte[] tmp = new byte[size];
                    in.read(tmp);
                    codedIn = CodedInputStream.newInstance(tmp);
                }
                DataLogEntry.parseFrom(codedIn);
                // ... do stuff
            }
            catch (final EOFException e) {
                break;
            }
        }
    }
}

NB: I've used an EOFException to find the end of file, you may wish to use a delimiter or track the number of byte read manually.

Answer:2

I am running Gnome on OpenSuse. As a result, my system look and feel is GTK+, which has numerous ugly problems (see some of them here). During development of my swing app, I can run the client from ...

Pesquisei muitos sites sobre como fazer isso usando apenas a chave da planilha, mas sem sucesso. Posso recuperá-lo, obter todos os dados que desejar, mas não consigo anexar uma nova linha a ele. Eu já ...

Eu implementei a gaveta de navegação que corresponde à altura dos pais. Agora acho que o botão de seta para trás que aparece depois de clicar no botão da gaveta de hambúrguer é realmente necessário aqui, pois não posso ...

Então, eu vi o Unicode sendo explorado para produzir uma grande bagunça assustadora de caracteres a partir de um texto de entrada normal, mais conhecido como texto do Zalgo. HTML (edit: javascript) parece fazer um trabalho maravilhoso em ...