NSMenuItem's setEnabled doesn't work even with autoenablesItems set to NO for NSMenu

Question

I'm maintaining some legacy code and when it was brought forward to compile with the 10.8 SDK rather than the 10.7 SDK, some key equivalent shortcuts for certain menu items stopped working. My hypothesis is that this is because those menu items are in a disabled state. The NSMenuItems that are no longer working are ones that were part of a submenu where autoenablesItems had been set to NO (I confirmed this in the xib via the attributes inspector for that submenu as well as programmatically by querying [NSMenu autoenablesItems]. It basically appears that the call to [NSMenuItem setEnabled:] is having no effect because if I query [NSMenuItem isEnabled] immediately after calling setEnabled:YES, the state hasn't changed and it still returns NO for isEnabled. Here's a snippet of code with the output it generates:

printf("DEBUG: Current state of menu item is ");
[nsMenuItem isEnabled] ? printf("enabled\n") : printf("DISABLED!\n");
printf("DEBUG: Current state of menu autoenablesItems is ");
[nsMenu autoenablesItems] ? printf("YES\n") : printf("NO\n");

[nsMenuItem setEnabled:YES];

printf("DEBUG: Current state of menu after setting it is ");
[nsMenuItem isEnabled] ? printf("enabled\n") : printf("DISABLED!\n");

Output:

DEBUG: Current state of menu item is DISABLED!
DEBUG: Current state of menu autoenablesItems is NO
DEBUG: Current state of menu after setting it is DISABLED!

I also tried subclassing NSMenuItem and overriding setEnabled to see if there was another call being made to setEnabled that was overwriting my call, but there are no other calls that go through setEnabled.

If I click on the submenu's parent menu then it seems to correct the state and these NSMenuItems get enabled, but it isn't going through my setEnabled code to change this state. I tried adding an observer to the subclass on enabled to try and catch where the state was getting set to enabled when clicking on the parent menu, but that didn't provide any insight either as the observer was only triggered for my calls to setEnabled which aren't actually changing the state.

From what I can tell from reading the Apple documentation, NSMenuItem's setEnabled should work as long as autoenablesItems is set to NO on the parent menu, but that doesn't seem to be working in this case, and I can't figure out why.

In the same code if I use the 10.7 SDK to compile (caveat: this is actually an older version of the code base, so there are other differences as well, but this particular code is the same), I do see that calling setEnabled changes the state of the NSMenuItem. When it enters this code in the 10.7 build the NSMenuItem is already in an enabled state, but I did try changing the call to setEnabled:NO to confirm whether this did in fact toggle the isEnabled state, and it did, unlike in the 10.8 build.

Any ideas why this isn't working in 10.8? I also tried with 10.9 and it didn't work there either. I've been unable to try 10.10 or 10.11 yet since there is some other code that needs updating in order to get it to compile with the newer SDKs (again, this is rather old legacy code).

When those logs are hit, are you sure that nsMenuItem is not in fact nil? That would result in NO being printed for both logs. Another avenue of attack would be to define the menu validation method on whatever object is relevant (the target, the first responder) and see if (1) it gets called, and (2) its return value is respected, even when autoenabling is turned off. Something fishy is going on, because this stuff is old code and has worked fine in Cocoa for many years. Detective work is what is needed. — bhaller
I have confirmed that nsMenu and nsMenuItem are not nil. I've also confirmed that after clicking on the top level menu item (that nsMenu is a submenu of), then the next time after that that the code is called, nsMenu and nsMenuItem are still the same objects (same address in the debugger), and that isEnabled is then returning true. The target on the nsMenuItem is nil, but there is an action defined. It has the same state when it works though too. — user1176103
When the key equivalent works, we end up in the code of the defined action (via NSApplication's sendEvent followed by NSObject's performSelector:withObject, followed by the action). When the key equivalent doesn't work, then from NSApplication's sendEvent we go to NSWindow's sendEvent (via the subclass's sendEvent first) followed by NSWindow's _reallySendEvent:isDelayedEvent: followed by the view's keyDown:. So the action is never called, and I presume it is because the menu item is disabled. — user1176103
Is the menu item actually disabled, in the UI? Displayed as greyed-out text, etc? I can't help but think that either the nsMenu or nsMenuItem variables aren't pointing at the objects you think they are, so you're actually manipulating the wrong objects. — Mark Bessey
Opening the menu in the UI results in the state changing - after clicking on the menu the short cuts work and the menu item shows as enabled. It is like trying to tell if the light is on in a fridge - by opening the door you're altering the state that you're trying to observe. Is there another way other than visually to see if the menu items are disabled? My guess was that they're disabled since the key equivalent doesn't work, and it is my understanding that key equivalents are disabled if the menu item is disabled. I'll try programmatically changing the title to confirm it is the same item. — user1176103

Mark Bessey Mark Bessey · Accepted Answer · 2015-12-01T01:18:53

Not sure why the SDK version would matter, but have you checked the menu in InterfaceBuilder to make sure it doesn't have bindings set up for the "enable" state? That would be another way for the items to get mysteriously-disabled.

Otherwise, a menu delegate could be replacing the menu items out from under your other code, or explicitly disabling items. There's also an NSMenuDelegate method

- menuHasKeyEquivalent:forEvent:target:action:

Which gets called specifically in cases where an accelerator key has been pressed, and which can totally bypass the enabled/disabled state of the menu items.

NSMenuItem's setEnabled doesn't work even with autoenablesItems set to NO for NSMenu

2 Answers