Write buffer protocol

[vallsv]

Hi,

This pull request features the write binary support for the Python buffer protocol.

As result, on top of the already supported str and bytes, the API now supports bytearray and memoryview. Everything use the same new Cython write_buffer function. Pure python writer was also implemented.

This can be used to reduce a memory copy while using numpy arrays.

I don't have checked the result with a complex benchmark, but in my use case (arrays of about 16MB), i can notice a time reduction of about 10% in the average (client and server on the same machine). For small messages, this does not change anything. I don't really know what could be a good benchmark.

Actually i use it such way:

struct NDArray {
    1: string dtype;
    2: list<i32> shape;
    3: binary buffer;
}

def create_ndarray_record(array: numpy.ndarray):
    msg = NDArray()
    msg.shape = array.shape
    msg.dtype = array.dtype.str

    # msg.buffer = array.tobytes()  # <-- previous code, this creates a memory copy

    array = numpy.ascontiguousarray(array)
    msg.buffer = array.data

    return msg

The pull request does not have any dependency to numpy.

Would you be interested by such contribution? Tell me the way i could improve it.

[aisk]

The idea sounds reasonable to me. However, we should just support buffer protocol instead of just support memoryview. As a result, memoryview / numpy.array / torch.tensor both implemented buffer protocol, they can be used in thriftpy2 directly.

[vallsv]

@aisk thanks for your feedback. Here is a rework with the buffer protocol.

I have also added few tests to check if both binary/cybinary can serialize few objects exposing the buffer protocol.

[vallsv]

struct.pack is not supporting memorymap.

It is possible to pack it like that.

def pack_memoryview(mem):
    return struct.pack("!i", len(mem)) + mem

But I have choose to write it in 2 steps instead. I guess it's better.

[vallsv]

@aisk here the problem is that the pure python binary module is using TCyMemoryBuffer.

That's in between 2 worlds.

I am thinking if it makes sens to propagate that at such place? Ill take a look anyway.