Adding more data in PROGMEM breaks SPI transfer on Arduino Mega 2560

Question

I'm working on a small project (or so I thought) involving an Arduino Mega (2560) and a Waveshare ePaper display.

I've got it working alright with the library (epd7in5) and I've added two images into PROGMEM. But as soon as I add a third image (and thus a third entry into the PROGMEM), somehow the ePaper screen doesn't initiate anymore. Adding some debugging in the library shows that the code gets stuck on a specific SPI.transfer().

EDIT: Theory Is it possible that SPI is not compatible when there's too much data in flash? I've read up on it that 64kb is the max. I'm slightly above that with two images, but significantly so with three. Could be that this breaks SPI? And if so: can I fix it?

I've added the code below and the specific part of the library where the SPI.transfer() fails.

Main.cpp

Removing the code related to dummy3 ensures that the dummy3 array doesn't compile. Only using dummy1 and dummy2 everything works fine. Adding dummy3 and the program gets stuck on epd.Init().

#include <SPI.h>
#include <epd7in5.h>
#include "imagedata.h"

Epd epd;

void debug(String);

void setup() {
  Serial.begin(9600);
  debug("Serial begin");

  if (epd.Init() != 0) {
    debug("INIT FAILED!");
    return;
  }

  debug("Changing image");
  epd.DisplayFrame(dummy1);  //DisplayFrame by default includes WaitUntilIdle.
  debug("dummy1 on ePaper");
  delay(1000);
  debug("Changing image");
  epd.DisplayFrame(dummy2);
  debug("dummy2 on ePaper");
  delay(1000);
  debug("Changing image");
  epd.DisplayFrame(dummy3);
  debug("dummy2 on ePaper");
  epd.SendCommand(POWER_OFF); 
  debug("POWER_OFF");
}

void loop() {
}

void debug(String message) {
  Serial.print(millis());
  Serial.print("\t");
  Serial.println(message);
}

imagedata.cpp

I've removed the actual image data as it's A LOT. Two images result in a total flash of 67326 bytes (about 26% of total flash mem of 2560). Three images result in a total flash of 98052 bytes (about 38% of total flash mem of 2560). Headerfile contains simply the declarations.

#include "imagedata.h"
#include <avr/pgmspace.h>

const unsigned char dummy1[30726] PROGMEM = {...data...};
const unsigned char dummy2[30726] PROGMEM = {...data...};
const unsigned char dummy3[30726] PROGMEM = {...data...};

epd7in5.cpp

I've added the debug function. SendData is also included and uses the debug as well.

void Epd::debug(String message) {
  Serial.print(millis());
  Serial.print("\t");
  Serial.print("EPD");
  Serial.print("\t");
  Serial.println(message);
}

int Epd::Init(void) {
  if (IfInit() != 0) {
    return -1;
  }

  debug("Resetting");
  Reset();

  debug("SendCommand(POWER_SETTING);");
  SendCommand(POWER_SETTING);
  debug("SendData(0x37);");
  SendData(0x37);
  debug("SendData(0x00);");
  SendData(0x00);

  debug("SendCommand(PANEL_SETTING);");
  SendCommand(PANEL_SETTING);
  SendData(0xCF);
  SendData(0x08);

  SendCommand(BOOSTER_SOFT_START);
  SendData(0xc7);
  SendData(0xcc);
  SendData(0x28);

  SendCommand(POWER_ON);
  WaitUntilIdle();

  SendCommand(PLL_CONTROL);
  SendData(0x3c);

  SendCommand(TEMPERATURE_CALIBRATION);
  SendData(0x00);

  SendCommand(VCOM_AND_DATA_INTERVAL_SETTING);
  SendData(0x77);

  SendCommand(TCON_SETTING);
  SendData(0x22);

  SendCommand(TCON_RESOLUTION);
  SendData(0x02);  //source 640
  SendData(0x80);
  SendData(0x01);  //gate 384
  SendData(0x80);

  SendCommand(VCM_DC_SETTING);
  SendData(0x1E);  //decide by LUT file

  SendCommand(0xe5);  //FLASH MODE
  SendData(0x03);

  return 0;
}

void Epd::SendData(unsigned char data) {
  debug("DigitalWrite(dc_pin, HIGH);");
  DigitalWrite(dc_pin, HIGH);
  debug("SpiTransfer(data);");
  SpiTransfer(data);
}

epdif.cpp

This is the SPI transfer part that doesn't continue.

void EpdIf::SpiTransfer(unsigned char data) {
    digitalWrite(CS_PIN, LOW);
    SPI.transfer(data);
    digitalWrite(CS_PIN, HIGH);
}

The serial print of the project is as follows...

0   Serial begin
0   EPD Resetting
400 EPD SendCommand(POWER_SETTING);
400 EPD SendData(0x37);
400 EPD DigitalWrite(dc_pin, HIGH);
435 EPD SpiTransfer(data);

So, as soon as it runs SpiTransfer the code simply stops working. It seems it is in an infinite loop within SPI.transfer(); but I don't know exactly how that would happen. I don't see how PROGMEM could interfere with the transfer and I have enough flash memory left...

What can be solution for this? Is it a problem in SPI that I need to change? Or do I need to store my data differently in PROGMEM? I am a bit at a loss.

Thanks in advance for your help, greatly appreciated.

Answer 1

Your problem is not with SPI (itself).

It is related to the amount of data you have in progmem and reading it. If you are using more than 64k of Progmem you run into 2 different sets of problems:

1. Reading the data in a reliable way

Reading the progmem should use the pgm_read_xxx_far macros for any address after 65536. Epd::DisplayFrame uses pgm_read_byte(&frame_buffer[i]); so you have a problem on the library level. I'm not familiar with this library, so I'm not sure if there's an alternative function that you can call, and provide the buffer yourself, after reading it from PROGMEM with pgm_read_byte_far. This is the easy part of the problem

2. Getting said data into PROGMEM in the first place

The Arduino core, and the avr compiler themselves assume that all pointers are only 16 bits. Putting data into PROGMEM makes it appear before the arduino core and executable code in the final executable that runs your program.

The reliable way to get this going correctly is with a custom linker script, that will get your data out of the way and place it after all executable code. This would be hard, and I'm afraid I can't give any info on how to accomplish that.

Alternatively, you can try to use _attribute__((section(".fini2"))) to use the .fini2 section. Some people have done it before, and it has worked for them.

You would use it like that:

const unsigned char __attribute__((section(".fini2"))) dummy1[30726] = {...data...};

The last alternative would be to not use PROGMEM at all, and have some kind of external storage that you use for your data (e.g. some SDCard). This would most probably be the easiest way to tackle that issue.

Source: This excellent thread and the wonderfull insight by westfw on the arduino forums: https://forum.arduino.cc/index.php?topic=485507.0