Your error is telling you all you need to know for this problem. You are attempting to reference two report items, and this is not possible in the header or footer.
Have you considered putting your calculated value in the report body, and then referencing this from the header?
Considering the below example, I have set the value of RCalc to be your expression
= IIF(ISNothing(ReportItems!R.value),ReportItems!R1.value,ReportItems!R.value)
I have also set the visibility of RCalc to be hidden
Then the text box in the header instead only references RCalc
=ReportItems!RCalc.value
The design therefore looks like this
And when run gives this output
Is this the behaviour you were after, and is this applicable in your situation. Pelease let me know if I can assit further.
UPDATE
The theory is the same when the values are coming from different Tablix.
Assuming the textboxes R and R1 are in totals Rows they can be referenced by RCalc in the same way. Note they cannot be in the main row set of data
The layout like this has the textboxes (in Red Text) set at the bottom of the report outside of the row grouping/filtering. These are R and R1. Note in the output I have set their visibility to hidden.
This results in the output
RCalc can be a textbox anywhere on the page – it doesn’t have to be in a dataset.