Strategy suggestions for removing special characters in java

Question

I've created a Java application that parses a text file to extract fields that are being loaded to a data table. We're discovering some exception processing where the table can't accept special characters, specifically Â and the like.

These characters appear in the input file as spaces when I look at it, but Java interprets them differently. I suspect it's a character code interpreted differently.

My question is this: in order to filter out these characters, is there any way I can generate a list of what Java is seeing? I'm thinking of printing the CHAR and the character code, and if possible, the character ~set~ (ASCII, ANSI, UTF-8, etc). From that, I could substitue a space for the character in my ending file and solve my problem.

Is there a simpler solution I'm not seeing?

Correct solution will be to enable your database to handle such characters. — Jayan
Don't filter the 'special' characters. Keep them and learn to handle character encodings properly, end-to-end. — artbristol
Are you certain you read in UTF-8 data correctly in the first place? — Thorbjørn Ravn Andersen
Unfortunately, @Jayan, that's not an option. The text files are what I was given to work with and I have no idea in which character set(s) they were created. That's the rock and the hard place between which I'm caught, and severly limits my ability to do this the RIGHT way. :) — dwwilson66
@ dwwilson66 : how will you convert it back to real data? If you have the input as file you could guess the encoding- jchardet.sourceforge.net. Well you can only guess. — Jayan

hallizh hallizh · Accepted Answer · 2012-05-09T13:05:27

Try decoding to say, UTF8?

public static byte[] stringToByteArray(String s)
    throws UnsupportedEncodingException {
    return s.getBytes("UTF-8");
}

Or some other like "iso-8859-1" and convert that bytearray to string and try printing it?

Strategy suggestions for removing special characters in java

2 Answers