How to configure cache for ignite-spark-dataframe?

Question

I managed to save and load spark dataframe from ignite by the example here: https://apacheignite-fs.readme.io/docs/ignite-data-frame

By following the code example when the cache is created in ignite it automatically has a name like "SQL_PUBLIC_name_of_table_in_spark".

One the other hand if I want to change some cache configuration I need to specify the same cache name in xml or code before creating ignite cache. Because cache configuration can not be changed after cache is created. See following code.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <!-- Set a cache name. -->
            <property name="name" value="SQL_PUBLIC_name_of_table_in_spark"/>
            <!-- Set cache mode. -->
            <property name="cacheMode" value="PARTITIONED"/>
        </bean>
    </property>
</bean>

Then one of them will be reject by "cache already exists" The result is I could't change any cache configuration by xml/code.

Is this expected? And how can I change the cache configuration in this case?

What do you need to change that can't be set using the OPTION_CREATE_TABLE_PARAMETERS option when creating the DataFrame? — Stephen Darlington

Stanislav Lukyanov Stanislav Lukyanov · Accepted Answer · 2018-10-11T10:39:37

The doc page you've link contains a code piece that creates an SQL table:

CREATE TABLE person ( 
    id LONG,  
    name VARCHAR,  
    city_id LONG,  
    PRIMARY KEY (id, city_id) 
) WITH "backups=1, affinityKey=city_id”;

This SQL command is what actually creates the cache. You can change this command to change the parameters of the cache that will be created. Refer to the CREATE TABLE doc.

In particular, the parameter that gives the most flexibility is WITH template=mytemplate. It lets you create a cache from a pre-existing template configuration. To register a template you can specify it in your cacheConfiguration with a name ending with asterisk, like

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <property name="name" value="mytemplate*"/>
            <!-- your parameters. -->
        </bean>
    </property>
</bean>

You can also specify the WITH parameters for CREATE TABLE in the OPTION_CREATE_TABLE_PARAMETERS setting if the table is being created automatically by Spark.

How to configure cache for ignite-spark-dataframe?

1 Answers