c# WPF: System.ExecutionEngineException while using Windows keyboard hooks

Question

Im trying to make hot keys over windows hooks. It's works, but sometimes wpf window, where i m using this solution, throws System.ExecutionEngineException (it's happens often when I press a lot of buttons quikly. I founded that this is happens because message loop become broken by my hooks. Is there are some ways to avoid it?

p.s. I know about RegisterHotKey function, but i need capabilities to abort key pressure if my hot key activated(to prevent executing similar hotkeys in other programs).

There is my hook setup:

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr SetWindowsHookEx(int idHook,
            LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool UnhookWindowsHookEx(IntPtr hhk);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode,
            IntPtr wParam, IntPtr lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr GetModuleHandle(string lpModuleName);

 private static IntPtr SetHook(LowLevelKeyboardProc proc)
        {
            using (Process curProcess = Process.GetCurrentProcess())
            using (ProcessModule curModule = curProcess.MainModule)
            {
                return SetWindowsHookEx(WH_KEYBOARD_LL, proc,
                    GetModuleHandle(curModule.ModuleName), 0);
            }
        }

There is my hook:

private static IntPtr HookCallback(
            int nCode, IntPtr wParam, IntPtr lParam)
        {
            bool isKeyProcessed = false;
            try
            {
                if (nCode >= 0 && (wParam == (IntPtr)WM_KEYDOWN || wParam == (IntPtr)WM_SYSKEYDOWN))
                {
                    int vkCode = Marshal.ReadInt32(lParam);
                    isKeyProcessed = KeyDown?.Invoke((Keys)vkCode) ?? false;
                }
                else if (nCode >= 0 && (wParam == (IntPtr)WM_KEYUP || wParam == (IntPtr)WM_SYSKEYUP))
                {
                    int vkCode = Marshal.ReadInt32(lParam);
                
                    isKeyProcessed = KeyUp?.Invoke((Keys)vkCode) ?? false;
                }
            }
            catch (Exception ex)
            {
                isKeyProcessed = false;
                try
                {
                    ExceptionHappend?.Invoke(ex);
                }
                catch { }
            }

            if (!isKeyProcessed)
                return CallNextHookEx(_hookID, nCode, wParam, lParam);
            else
                return (IntPtr)1;
        }

There is my hook events handler (for key down, in key up i just delete upped button from current key combo)

public bool HandleKeyDown(Keys key)
        {
            lock (_lock)
            {
                currentCombo.Keys.Add(key);
                if (hotkeys.ContainsKey(currentCombo))
                {
                    RunHandler(hotkeys[currentCombo]);
                    currentCombo.Keys.Clear();
                    return true;
                }
                return false;
            }
        }


private void RunHandler(Action handler)
        {
            _runningActions = _runningActions.Where(x=>x.Status == TaskStatus.Running).ToList();
            var task = new Task(handler);
            task.Start();
            _runningActions.Add(task);
        }

Finally, my hotkey, example:

 public void ShowWorkedQ()
        {
            Dispatcher.Invoke(new Action(() =>
            {
                OutLabel.Text = "Pressed Alt+Q";
            }));
        }

Thank you for long reading!

Answer 1

It looks like you are not keeping the callback delegate alive. You need to store it in a field in order to prevent it from being destroyed.

When you pass a delegate to a PInvoke function, PInvoke will create a little native function that enables the callback to jump back to managed code. But GC cannot track this object correctly unless you still hold a reference to the original delegate.

static LowLevelKeyboardProc _proc;

private static IntPtr SetHook(LowLevelKeyboardProc proc)
{
    using (Process curProcess = Process.GetCurrentProcess())
    using (ProcessModule curModule = curProcess.MainModule)
    {
        if (_proc != null) throw new Exception("Cannot set hook more than once");
        _proc = proc;  // NEW!
        return SetWindowsHookEx(WH_KEYBOARD_LL, proc,
            GetModuleHandle(curModule.ModuleName), 0);
    }
}

Note also that lParam in a Low Level KB hook actually represents a pointer to a KBDLLHOOKSTRUCT. While your current code techincally works to just read the first value, you should really marshal the whole struct.

[StructLayout(LayoutKind.Sequential)]
struct KBDLLHOOKSTRUCT
{
  int vkCode;
  int scanCode;
  int flags;
  int time;
  IntPtr dwExtraInfo;
}


delegate IntPtr LowLevelKeyboardProc (
  int    nCode,
  IntPtr wParam,
  [In]
  in KBDLLHOOKSTRUCT lParam
);