You can't define a table calculation that will produce two different output types: this isn't Excel, this is database technology where each field has to contain a single type.
But there are ways around the problem that still give the result you want. Instead of trying to flag the dodgy value in the single result field, define a different field for the rows containing the dodgy data. So, instead of a single output called something like DisplayPercentage use a calculation to create a field called something like %OK defined like this:
if [percentage] > 1000
then 'NA'
else 'percentage OK'
end
and use the resulting text field to highlight or filter the dodgy values.
This is both better and more versatile than trying to combine flags for bad data with the data itself in a single field, variabel or calculation. You can make it even more advanced by defining the cutoff into a parameter and using that rather than a hardcoded value in the calculation so the cutoff can be altered dynamically without having to rewrite any code.