One common method of packing variable-length data sets to a single continuous array is using one element to describe the length of the next data sequence, followed by that many data items, with a zero length terminating the array.
In other words, if you have data "strings" 1, 2 3, 4 5 6, and 7 8 9 10, you can pack them into an array of 1+1+1+2+1+3+1+4+1 = 15 bytes as 1 1 2 2 3 3 4 5 6 4 7 8 9 10 0.
The functions to access said sequences are quite simple, too. In OP's case, each data item is an uint8:
uint8 dataset[] = { ..., 0 };
To loop over each set, you use two variables: one for the offset of current set, and another for the length:
uint16 offset = 0;
while (1) {
const uint8 length = dataset[offset];
if (!length) {
offset = 0;
break;
} else
++offset;
/* You have 'length' uint8's at dataset+offset. */
/* Skip to next set. */
offset += length;
}
To find a specific dataset, you do need to find it using a loop. For example:
uint8 *find_dataset(const uint16 index)
{
uint16 offset = 0;
uint16 count = 0;
while (1) {
const uint8 length = dataset[offset];
if (length == 0)
return NULL;
else
if (count == index)
return dataset + offset;
offset += 1 + length;
count++;
}
}
The above function will return a pointer to the length item of the index'th set (0 referring to the first set, 1 to the second set, and so on), or NULL if there is no such set.
It is not difficult to write functions to remove, append, prepend, and insert new sets. (When prepending and inserting, you do need to copy the rest of the elements in the dataset array forward (to higher indexes), by 1+length elements, first; this means that you cannot access the array in an interrupt context or from a second core, while the array is being modified.)
If the data is immutable (for example, generated whenever a new firmware is uploaded to the microcontroller), and you have sufficient flash/rom available, you can use a separate array for each set, an array of pointers to each set, and an array of sizes of each set:
static const uint8 dataset_0[] PROGMEM = { 1 };
static const uint8 dataset_1[] PROGMEM = { 2, 3 };
static const uint8 dataset_2[] PROGMEM = { 4, 5, 6 };
static const uint8 dataset_3[] PROGMEM = { 7, 8, 9, 10 };
#define DATASETS 4
static const uint8 *dataset_ptr[DATASETS] PROGMEM = {
dataset_0,
dataset_1,
dataset_2,
dataset_3,
};
static const uint8 dataset_len[DATASETS] PROGMEM = {
sizeof dataset_0,
sizeof dataset_1,
sizeof dataset_2,
sizeof dataset_3,
};
When this data is generated at firmware compile time, it is common to put this into a separate header file, and simply include it from the main firmware .c source file (or, if the firmware is very complicated, from the specific .c source file that accesses the data sets). If the above is dataset.h, then the source file typically contains say
#include "dataset.h"
const uint8 dataset_length(const uint16 index)
{
return (index < DATASETS) ? dataset_len[index] : 0;
}
const uint8 *dataset_pointer_P(const uint16 index)
{
return (index < DATASETS) ? dataset_ptr[index] : NULL;
}
i.e., it includes the dataset, and then defines the functions that access the data. (Note that I deliberately made the data itself static, so they are only visible in the current compilation unit; but the dataset_length() and dataset_pointer(), the safe accessor functions, are accessible from other compilation units (C source files), too.)
When the build is controlled via a Makefile, this is trivial. Let's say the generated header file is dataset.h, and you have a shell script, say generate-dataset.sh, that generates the contents for that header. Then, the Makefile recipe is simply
dataset.h: generate-dataset.sh
@$(RM) $@
$(SHELL) -c "$^ > $@"
with the recipes for the compilation of the C source files that need it, containing it as a prerequisite:
main.o: main.c dataset.h
$(CC) $(CFLAGS) -c main.c
Do note that the indentation in Makefiles always uses Tabs, but this forum does not reproduce them in code snippets. (You can always run sed -e 's|^ *|\t|g' -i Makefile to fix copy-pasted Makefiles, though.)
OP mentioned that they are using Codevision, that does not use Makefiles (but a menu-driven configuration system). If Codevision does not provide a pre-build hook (to run an executable or script before compiling the source files), then OP can write a script or program run on the host machine, perhaps named pre-build, that regenerates all generated header files, and run it by hand before every build.
In the hybrid case, where you know the length of each data set at compile time, and it is immutable (constant), but the sets themselves vary at run time, you need to use a helper script to generate a rather large C header (or source) file. (It will have 1500 lines or more, and nobody should have to maintain that by hand.)
The idea is that you first declare each data set, but do not initialize them. This makes the C compiler reserve RAM for each:
static uint8 dataset_0_0[3];
static uint8 dataset_0_1[2];
static uint8 dataset_0_2[9];
static uint8 dataset_0_3[4];
/* : : */
static uint8 dataset_0_97[1];
static uint8 dataset_0_98[5];
static uint8 dataset_0_99[7];
static uint8 dataset_1_0[6];
static uint8 dataset_1_1[8];
/* : : */
static uint8 dataset_1_98[2];
static uint8 dataset_1_99[3];
static uint8 dataset_2_0[5];
/* : : : */
static uint8 dataset_4_99[9];
Next, declare an array that specifies the length of each set. Make this constant and PROGMEM, since it is immutable and goes into flash/rom:
static const uint8 dataset_len[5][100] PROGMEM = {
sizeof dataset_0_0, sizeof dataset_0_1, sizeof dataset_0_2,
/* ... */
sizeof dataset_4_97, sizeof dataset_4_98, sizeof dataset_4_99
};
Instead of the sizeof statements, you can also have your script output the lengths of each set as a decimal value.
Finally, create an array of pointers to the datasets. This array itself will be immutable (const and PROGMEM), but the targets, the datasets defined first above, are mutable:
static uint8 *const dataset_ptr[5][100] PROGMEM = {
dataset_0_0, dataset_0_1, dataset_0_2, dataset_0_3,
/* ... */
dataset_4_96, dataset_4_97, dataset_4_98, dataset_4_99
};
On AT90CAN128, the flash memory is at addresses 0x0 .. 0x1FFFF (131072 bytes total). Internal SRAM is at addresses 0x0100 .. 0x10FF (4096 bytes total). Like other AVRs, it uses Harvard architecture, where code resides in a separate address space -- in Flash. It has separate instructions for reading bytes from flash (LPM, ELPM).
Because a 16-bit pointer can only reach half the flash, it is rather important that the dataset_len and dataset_ptr arrays are "near", in the lower 64k. Your compiler should take care of this, though.
To generate correct code for accessing the arrays from flash (progmem), at least AVR-GCC needs some helper code:
#include <avr/pgmspace.h>
uint8 subset_len(const uint8 group, const uint8 set)
{
return pgm_read_byte_near(&(dataset_len[group][set]));
}
uint8 *subset_ptr(const uint8 group, const uint8 set)
{
return (uint8 *)pgm_read_word_near(&(dataset_ptr[group][set]));
}
The assembly code, annotated with the cycle counts, avr-gcc-4.9.2 generates for at90can128 from above, is
subset_len:
ldi r25, 0 ; 1 cycle
movw r30, r24 ; 1 cycle
lsl r30 ; 1 cycle
rol r31 ; 1 cycle
add r30, r24 ; 1 cycle
adc r31, r25 ; 1 cycle
add r30, r22 ; 1 cycle
adc r31, __zero_reg__ ; 1 cycle
subi r30, lo8(-(dataset_len)) ; 1 cycle
sbci r31, hi8(-(dataset_len)) ; 1 cycle
lpm r24, Z ; 3 cycles
ret
subset_ptr:
ldi r25, 0 ; 1 cycle
movw r30, r24 ; 1 cycle
lsl r30 ; 1 cycle
rol r31 ; 1 cycle
add r30, r24 ; 1 cycle
adc r31, r25 ; 1 cycle
add r30, r22 ; 1 cycle
adc r31, __zero_reg__ ; 1 cycle
lsl r30 ; 1 cycle
rol r31 ; 1 cycle
subi r30, lo8(-(dataset_ptr)) ; 1 cycle
sbci r31, hi8(-(dataset_ptr)) ; 1 cycle
lpm r24, Z+ ; 3 cycles
lpm r25, Z ; 3 cycles
ret
Of course, declaring subset_len and subset_ptr as static inline would indicate to the compiler you want them inlined, which increases the code size a bit, but might shave off a couple of cycles per invocation.
Note that I have verified the above (except using unsigned char instead of uint8) for at90can128 using avr-gcc 4.9.2.
