In this post, I outline a simple, easy to understand implementation for two components of a Redis client in Go as a way of understanding how the Redis protocol works and what makes it great.
If you’re looking for a full-featured, production-ready Redis client in Go, we recommend taking a look at Gary Burd’s redigo library.
Before we get started, be sure you read our gentle introduction to the Redis protocol - it covers the basics of the protocol that you’ll need to understand for this guide.
For our hypothetical Redis client, there’s only one kind of object that we’ll need to write: an array of bulk strings for sending commands to Redis. Here is a simple implementation of a command-to-RESP writer:
package redis
import (
"bufio"
"io"
"strconv" // for converting integers to strings
)
var (
arrayPrefixSlice = []byte{'*'}
bulkStringPrefixSlice = []byte{'$'}
lineEndingSlice = []byte{'\r', '\n'}
)
type RESPWriter struct {
*bufio.Writer
}
func NewRESPWriter(writer io.Writer) *RESPWriter {
return &RESPWriter{
Writer: bufio.NewWriter(writer),
}
}
func (w *RESPWriter) WriteCommand(args ...string) (err error) {
// Write the array prefix and the number of arguments in the array.
w.Write(arrayPrefixSlice)
w.WriteString(strconv.Itoa(len(args)))
w.Write(lineEndingSlice)
// Write a bulk string for each argument.
for _, arg := range args {
w.Write(bulkStringPrefixSlice)
w.WriteString(strconv.Itoa(len(arg)))
w.Write(lineEndingSlice)
w.WriteString(arg)
w.Write(lineEndingSlice)
}
return w.Flush()
}Rather than writing to a net.Conn object, RESPWriter writes to a io.Writer
object. This allows us to test our parser without tightly coupling to the net
stack. We simply test the network protocol the way we would any other io.
For example, we can pass it a bytes.Buffer to inspect the final RESP:
var buf bytes.Buffer
writer := NewRESPWriter(&buf)
writer.WriteCommand("GET", "foo")
buf.Bytes() // *2\r\n$3\r\nGET\r\n$3\r\nfoo\r\nAfter sending a command to Redis with RESPWriter, our client would use
RESPReader to read from the TCP connection until it has received a full RESP
reply. To start with, we’ll need a few packages to handle buffering and parsing
the incoming data:
package redis
import (
"bufio"
"bytes"
"errors"
"io"
"strconv"
)And we’ll use a few variables and constants to make our code a little easier to read:
const (
SIMPLE_STRING = '+'
BULK_STRING = '$'
INTEGER = ':'
ARRAY = '*'
ERROR = '-'
)
var (
ErrInvalidSyntax = errors.New("resp: invalid syntax")
)Like RESPWriter, RESPReader doesn’t care about the implementation details of
the object that it’s reading RESP from. All it needs the ability to read bytes
until it has read a full RESP object. In this case, it needs an io.Reader,
which it wraps with a bufio.Reader to handle the buffering of the incoming
data.
Our object and initializer are simple:
type RESPReader struct {
*bufio.Reader
}
func NewReader(reader io.Reader) *RESPReader {
return &RESPReader{
Reader: bufio.NewReaderSize(reader, 32*1024),
}
}The buffer size for bufio.Reader is just a guess during development. In an
actual client, you’d want to make its size configurable and perhaps test to find
the optimal size. 32KB will work fine for development.
RESPReader has only one method: ReadObject(), which returns a byte slice
containing a full RESP object on each call. It will pass back any errors
encountered from io.Reader, and will also return errors when it encounters any
invalid RESP syntax.
The prefix nature of RESP means we only need to read the first byte to decide
how to handle the following bytes. However, because we’ll always need to read at
least the first full line (i.e. up until the first \r\n), we can start by
reading the whole first line:
func (r *RESPReader) ReadObject() ([]byte, error) {
line, err := r.readLine()
if err != nil {
return nil, err
}
switch line[0] {
case SIMPLE_STRING, INTEGER, ERROR:
return line, nil
case BULK_STRING:
return r.readBulkString(line)
case ARRAY:
return r.readArray(line) default:
return nil, ErrInvalidSyntax
}
}When the line that we read has a simple string, integer, or error prefix, we return the full line as the received RESP object because those object types are contained entirely within one line.
In readLine(), we read up until the first occurrence of \n and then check to
make sure that it was preceded by a \r before returning the line as a byte
slice:
func (r *RESPReader) readLine() (line []byte, err error) {
line, err = r.ReadBytes('\n')
if err != nil {
return nil, err
}
if len(line) > 1 && line[len(line)-2] == '\r' {
return line, nil
} else {
// Line was too short or \n wasn't preceded by \r.
return nil, ErrInvalidSyntax
}
}In readBulkString() we parse the length specification for the bulk string to
know how many bytes we need to read. Once we do, we read that count of bytes and
the \r\n line terminator:
func (r *RESPReader) readBulkString(line []byte) ([]byte, error) {
count, err := r.getCount(line)
if err != nil {
return nil, err
}
if count == -1 {
return line, nil
}
buf := make([]byte, len(line)+count+2)
copy(buf, line)
_, err = io.ReadFull(r, buf[len(line):])
if err != nil {
return nil, err
}
return buf, nil
}I’ve pulled getCount() out to a separate method because the length
specification is also used for arrays:
func (r *RESPReader) getCount(line []byte) (int, error) {
end := bytes.IndexByte(line, '\r')
return strconv.Atoi(string(line[1:end]))
}To handle arrays, we get the number of array elements, and then call
ReadObject() recursively, adding the resulting objects to our current RESP
buffer:
func (r *RESPReader) readArray(line []byte) ([]byte, error) {
// Get number of array elements.
count, err := r.getCount(line)
if err != nil {
return nil, err
}
// Read `count` number of RESP objects in the array.
for i := 0; i < count; i++ {
buf, err := r.ReadObject()
if err != nil {
return nil, err
}
line = append(line, buf...)
}
return line, nil
}The above hundred lines are all that’s needed to read any RESP object from Redis. However, there are a number of missing pieces we’d need to implement before using this library in a production environment:
RESPReader currently
only returns the full RESP response, it does not, for example, return a string
from a bulk string response. However, implementing this would be easy.RESPReader needs better syntax error handling.This code is also entirely unoptimized and does more allocations and copies than
it needs to. For example, the readArray() method: for each object in the
array, we read in the object and then copies it to our local buffer.
If you’re interested in learning how to implement these pieces, I recommend looking at how popular libraries like hiredis or redigo implement them.
Special thanks to Niel Smith for helping us catch some bugs in the code contained in this post.