A simple approach would be to use a div element and put the content of the editor inside that div. The div should have set a width (using css).
<div class="calculation_preview"></div>
And the css should be something like:
div.calculation_preview {
width: 200px;
visibility: hidden;
position: absolute;
top: 0px;
left:0px;
}
After each key-stroke, you can measure the height of the div (using the function offsetHeight on the div).
If it is too height, remove the last character the user entered.
To restore the previous content, you can declare a javascript variable as for example:
var savedContent = "";
If you have some initial content in your editor, then you should initialize the variable with that content.
For each key-stroke, you do the following:
// Get current editor content:
var content = tinyMCE.activeEditor.getContent({format : 'raw'});
// Measure the new height of the content:
var measurer = document.getElementById("calculation_preview");
measurer.innerHTML = content;
if(measurer.offsetHeight > 100) {
// Content is too big.. restore previous:
tinyMCE.activeEditor.setContent(savedContent, {format : 'raw'});
} else {
// Content height is ok .. save to variable:
savedContent = content;
}