Toggle NSStatusItem's Menu Open/Closed Using Hot Key - Code Execution Queued/Blocked

Question

I'm editing this question because I think I may have oversimplified the way in which my status item's menu opens. It's ridiculously complicated for such a simple function!

My status item supports both left and right click actions. The user can change what happens which each click type. Also, due to a macOS bug, I have to do some extra-special work when there are 2 or more screens/displays connected and they are arranged vertically.

I’m using MASShortcut to open an NSStatusItem's menu via a system-wide hot key ("⌘ ⌥ M", let's say), and I'm finding that once the menu has been opened, it's not possible to close it with a hot key. I'm trying to toggle the menu from closed to open and vice versa. When the menu is open, code execution is blocked, however. Are there any ways around this? I found this question which seems like a similar issue, but sadly no answer was ever found.

Thanks in advance for any assistance!

UPDATE: Sample Project Demonstrating Issue

When the user executes the designated hot key to show the status item menu, the following runs:

[[MASShortcutBinder sharedBinder] bindShortcutWithDefaultsKey: kShowMenuHotkey toAction: ^
     {
         if (!self.statusMenuOpen)
         {
             [self performSelector:@selector(showStatusMenu:) withObject:self afterDelay:0.01];
         }
         else
         {
             [self.statusMenu cancelTracking];
         }
     }];

And here's the other relevant code:

- (void) applicationDidFinishLaunching: (NSNotification *) aNotification
{     
     // CREATE AND CONFIGURE THE STATUS ITEM
     self.statusItem = [[NSStatusBar systemStatusBar] statusItemWithLength: NSVariableStatusItemLength];
     [self.statusItem.button sendActionOn:(NSLeftMouseUpMask|NSRightMouseUpMask)];
     [self.statusItem.button setAction: @selector(statusItemClicked:)];
     self.statusMenu.delegate = self;
}

- (IBAction) statusItemClicked: (id) sender
{
     // Logic exists here to determine if the status item click was a left or right click 
     // and whether the menu should show based on user prefs and click type

     if (menuShouldShow)
     {
          [self performSelector:@selector(showStatusMenu:) withObject:self afterDelay:0.01];
     }
}

- (IBAction) showStatusMenu: (id) sender
{
     // macOS 10.15 introduced an issue with some status item menus not appearing 
     // properly when two or more screens/displays are arranged vertically
     // Logic exists here to determine if this issue is present on the current system

     if (@available(*, macOS 10.15))
     {
          if (verticalScreensIssuePresent)
          {
               [self performSelector:@selector(popUpStatusItemMenu) withObject:nil afterDelay:0.05];
          }
          else // vertical screens issues not present
          {
               // DISPLAY THE MENU NORMALLY
               self.statusItem.menu = self.statusMenu;
               [self.statusItem.button performClick:nil];
          }                    
     }
     else // not macOS 10.15+
     {
        // DISPLAY THE MENU NORMALLY
        self.statusItem.menu = self.statusMenu;
        [self.statusItem.button performClick:nil];
     }
}

- (void) popUpStatusItemMenu
{
      // Logic exists here to determine how wide the menu is
      // If the menu is too wide to fit on the right, display
      // it on the left side of the status item

     // menu is too wide for screen, need to open left side
     if (pt.x + menuWidth >= NSMaxX(currentScreen.frame))
     {
          [self.statusMenu popUpMenuPositioningItem:[self.statusMenu itemAtIndex:0]
                                         atLocation:CGPointMake((-menuWidth + self.statusItem.button.superview.frame.size.width), -5)
                                             inView:[self.statusItem.button superview]];

    }
    else // not too wide
    {
        
          [self.statusMenu popUpMenuPositioningItem:[self.statusMenu itemAtIndex:0]
                                         atLocation:CGPointMake(0, -5)
                                             inView:[self.statusItem.button superview]];

    }
}

Answer 1

I ended up solving this issue by programmatically assigning an NSMenuItem's keyEquivalent to be the same hot key as the MASShortcut hot key value. This allows the user to use the same hot key to perform a different function (close the NSMenu.)

When setting up the hot key:

-(void) setupOpenCloseMenuHotKey
{
    [[MASShortcutBinder sharedBinder] bindShortcutWithDefaultsKey: kShowMenuHotkey toAction: ^
    {
        // UNHIDES THE NEW "CLOSE MENU" MENU ITEM
        self.closeMenuItem.hidden = NO; 
                
        // SET THE NEW "CLOSE MENU" MENU ITEM'S KEY EQUIVALENT TO BE THE SAME
        // AS THE MASSHORTCUT VALUE
        [self.closeMenuItem setKeyEquivalentModifierMask: self.showMenu.shortcutValue.modifierFlags];
        [self.closeMenuItem setKeyEquivalent:self.showMenu.shortcutValue.keyCodeString];
            
        self.showMenuTemp = [self.showMenu.shortcutValue copy];
        self.showMenu.shortcutValue = nil;
    
        dispatch_async(dispatch_get_main_queue(), ^{
            [self performSelector:@selector(showStatusMenu:) withObject:self afterDelay:0.01];
        });
    }];
}

Then, when the menu closes:

- (void) menuDidClose : (NSMenu *) aMenu
{
    // HIDE THE MENU ITEM FOR HOTKEY CLOSE MENU 
    self.closeMenuItem.hidden = YES;
        
    self.showMenu.shortcutValue = [self.showMenuTemp copy];
    self.showMenuTemp = nil;
        
    [self setupOpenCloseMenuHotKey];
}

Answer 2

I can confirm your observation

I'm trying to toggle the menu from closed to open and vice versa. When the menu is open, code execution is blocked

The reason is NSMenu when opened takes over the app's NSEvents handling (it's internal __NSHLTBMenuEventProc handles that) over the standard [NSApplication run] queue.

The events that will actually trigger the shortcut handling eventually are NSEventTypeSystemDefined with subtype 6 (9 being the following keyUp which aren't really relevant here).

Those NSEventTypeSystemDefined aren't fired at all when the menu is opened. Some mechanism is postponing their firing until the menu is dismissed and the app gets back to [NSApplication run] queue. A tried a lot of tricks and hacks to circumvent that, but to no avail.

MASShortcut uses legacy Carbon API to install this custom event handler. I was able to plug it in instead to the NSMenu internal Event Dispatcher (it works when the menu is not opened) but it doesn't solve the problem because the aforementioned NSEvents weren't fired in the first place (until the menu dismisses).

My educated guess is it's the MacOS WindowServer that's governing this (since it's aware of things like control keys pressed among others).

Anyway I'm glad you've found your workaround.

If anyone would like debug those events (I guess this is the best starting point I can offer) here's the code I used:

    int main(int argc, const char * argv[]) {
    @autoreleasepool {
        // Setup code that might create autoreleased objects goes here.
    }
    
    Class clazz = NSApplication.class;

    SEL selectorNextEventMatchingEventMask = NSSelectorFromString(@"_nextEventMatchingEventMask:untilDate:inMode:dequeue:");
    Method method = class_getInstanceMethod(clazz, selectorNextEventMatchingEventMask);
    const char *typesSelectorNextEventMatchingMask  = method_getTypeEncoding(method);
    IMP genuineSelectorNextEventMatchingMask = method_getImplementation(method);
    
    IMP test = class_replaceMethod(clazz, selectorNextEventMatchingEventMask, imp_implementationWithBlock(^(__unsafe_unretained NSApplication* self, NSEventMask mask, NSDate* expiration, NSRunLoopMode mode, BOOL deqFlag) {

        NSEvent* (*genuineSelectorNextEventMatchingMaskTyped)(id, SEL, NSEventMask, NSDate*, NSRunLoopMode, BOOL) = (void *)genuineSelectorNextEventMatchingMask;
        NSEvent* event = genuineSelectorNextEventMatchingMaskTyped(self, selectorNextEventMatchingEventMask, mask, expiration, mode, deqFlag);
        
        if (event.type == NSEventTypeSystemDefined) {
            if (event.subtype == 6l) {
                NSLog(@"⚪️ %@ %i %@", event, mask, mode);
            }
            else if (event.subtype == 9l) {
                NSLog(@"⚪️⚪️ %@ %i %@", event, mask, mode);
            }
            else if (event.subtype == 7l) {

                NSLog(@" UNKNOWN %@ %i %@", event, mask, mode);
            }
            else {
                NSLog(@" %@ %i %@", event, mask, mode);
            }
            
        } else if (event == NULL && [mode isEqualToString:NSEventTrackingRunLoopMode]) {
            //NSMenu "null" events happening here
            NSLog(@"⚪️⚪️⚪️ %@ %i %@", event, mask, mode);
        } else if (event == NULL) {
            NSLog(@"⭐️ %@ %i %@", event, mask, mode);
        } else {
            NSLog(@" %@ %i %@", event, mask, mode);
        }
        
        return event;
        
    }), typesSelectorNextEventMatchingMask);
    
    return NSApplicationMain(argc, argv);
}

One can notice the NSMenu triggered events will operate in NSEventTrackingRunLoopMode but that's not particularly useful to solve anything.