This is the best one I've seen: http://sharemath.com/
It covers a wide range of mathematical inputs. Most of it adjustable from XML I think as well. It's worth looking into even if you just want to get a base of what it might look to make it from scratch.
There is also something known as LaTeX that we use often: http://validi.fi/latex2flash/
Either way speaking from experience, it is a very tricky process doing this from scratch so having something to at least base it off of is a extremely useful.
LaTeX integration example taken from the link provided above:
package
{
import flash.display.*;
import flash.text.*;
import flash.events.KeyboardEvent;
public class Test extends MovieClip
{
private var latexField : LatexField;
private var inputField : TextField;
public function Test()
{
inputField = new TextField();
inputField.type = TextFieldType.INPUT;
inputField.border = true;
latexField = new LatexField();
latexField.eqString = "\\sum_{i=0}^ni=1+2+3+4+\\ldots +n";
latexField.x = 200;
addChild(inputField);
addChild(latexField);
inputField.addEventListener(KeyboardEvent.KEY_UP, keyUpListener);
}
private function keyUpListener(e:KeyboardEvent):void
{
//ENTER
if (e.keyCode == 13) {
latexField.eqString = escape(TextField(e.target).text);
}
}
}
}
Good luck!