Store raw binary value in Apache Ignite through thin python client

Question

I'm trying to save some raw binary data into Apache Ignite using thin Python client and the process is very slow (actually it looks like it depends on the size of the data).

For my simple test case I've started locally Ignite with single node (version 2.7.0 without any specific configuration):

$ bin/ignite.sh

Also, I have a binary file with the content I want to store in Ignite. The size if around 6MB. Here is my source code:

with open("/var/bin_text.bin", mode='rb') as file:
    file_content = file.read()

client = Client()
client.connect('127.0.0.1', 10800)
my_cache = client.get_or_create_cache('my cache')
my_cache.put("key_bin", file_content)

After my_cache.put the process just freezes. I haven't even managed to wait till the end (I've terminated it after 2 minutes).

However, passing the file content transformed into a string works very fast .

my_cache.put("key_str", str(file_content))

What I'm missing?

Update

I've played around with the size of the data I want to put to Ignite using the approach proposed by @jock-tanner. Here is what I've got:

size: 1024, elapsed time: 0.0045130252838134766 secs
size: 2048, elapsed time: 0.007149457931518555 secs
size: 4096, elapsed time: 0.004557132720947266 secs
size: 8192, elapsed time: 0.010631561279296875 secs
size: 16384, elapsed time: 0.025577783584594727 secs
size: 32768, elapsed time: 0.07686495780944824 secs
size: 65536, elapsed time: 0.2685544490814209 secs
size: 131072, elapsed time: 0.8761806488037109 secs
size: 262144, elapsed time: 3.121284246444702 secs
size: 524288, elapsed time: 12.343520879745483 secs
size: 1048576, elapsed time: 53.10914897918701 secs
size: 2097152, elapsed time: 205.68292760849 secs

It takes more than 3 minutes to put 2MB of data to locally installed Ignite. Could it be my Ignite configuration issue?

Jock Tanner Jock Tanner · Accepted Answer · 2019-05-14T12:25:15

Sorry, but you just can not store binary data in Ignite the way you trying to do in your first example. You should convert it either to a Unicode string if possible (as you did in your second example), or a sequence of bytes:

cache1.put('key', [ord(x) for x in file_content], value_hint=ByteArrayObject)

Unlike Redis keys and values, that are just binary strings, Ignite keys and values are actually typed. In pyignite I tried hard to hide that fact behind the duck typing-friendly get/put semantics, but it's still showing.

You can learn about Ignite data types and how to use them in the following documents:

Python thin client: sections “Data Types” and “Key-Value Operations”,
“Binary client protocol”, section “Data Format”, it is written from the perspective of data serializing/deserializing,
if you want to use SQL queries, you will need also “Ignite SQL Data Types”.

Note that Ignite typing system is based on Java one, that is why there's no document on Ignite data types in general.

Store raw binary value in Apache Ignite through thin python client

2 Answers