Is it possible to know what text is displaying nearby the mouse?

23
votes

I am just planning to implement some features like "Look up in Dictionary" when the mouse hovers over some text/words on the screen on the Mac OS X platform.

How can I get the displaying text/words nearby the mouse on the screen, even if they're not in my own application.

What I can do is:

  1. Using WorkSpace to know what applications are running.
  2. Use accessibility API to know the top UIElement on the screen.
  3. Use accessibility API to know the selected string of a UIElement if the application use something like NSTextView (I get get it by check the "selected text" attribute of the UIElement).

What I can't do is:

  1. Some applications like Safari, using WebKit framework, the only things I can get from Accessibility API are "value attribute", which is the current content of the HTML, and some attributes named like "AXTextMarker". These attributes can't not be found in Google or any document.
  2. Some applications even do not support Accessibility API, what I can get is just a screen shot of it.

Is there any way to call system API to recognize the text in the image? In Snow Leopard, there is a handwriting recognizer feature for input in Chinese or Japanese, so should it be some OCR feature? I can't find any open API for it.

1
objective-ccocoamacos
Handwriting recognition typically goes by gestures (endpoints and pauses/corners), not OCR. Doing OCR on the typical person's handwriting would be no better than catting from /dev/random. ☺ – Peter Hosey
OK, forgive my stupid, since I really don't know where to go. – icespace
Then another question, can I send some event or even call Quartz to re-render the text in some area on the screen? Maybe I can get sth from it. – icespace
You'd need to have the text to tell Quartz to render it. The issue is not drawing, since that would necessitate OCR; what you need is some sort of accessible control to extract the text from. Unfortunately, some apps just aren't written with support for accessibility. It comes free with Cocoa, but some apps don't use Cocoa. There is nothing you can do about this from your application. – Peter Hosey
Thank you Peter. It seems no cocoa OCR API opened by Apple, it is really a pitty. – icespace

1 Answers

1
votes

What you can probably do is this:
Use a meta-key + a key chord to activate the dictionary once text is selected. I'd actually find it incredibly obnoxious if whenever I was near text for a bit, a dictionary popped up. Particularly because I read a tremendous number of white papers, where almost any dictionary imaginable would spend most of its time choking and dying.