Delphi - How do you generate an event when a user clicks outside modal dialog?

Question

Is it possible to fire an event when the user clicks outside a modal dialog?

OK, Windows provides it\'s own clues when you do this by making the \"bonk\" sound, or

Answer 1

First, to answer the question:

You could capture the mouse when it moves outside the dialog, or when is already outside the dialog at showing. Then you can catch WM_CAPTURECHANGED to fire an OnMouseClickOutside event:

type
  TDialog = class(TForm)
  private
    FMouseInDialog: Boolean;
    FOnMouseClickOutside: TNotifyEvent;
    procedure WMCaptureChanged(var Message: TMessage);
      message WM_CAPTURECHANGED;
    procedure CMMouseLeave(var Message: TMessage); message CM_MOUSELEAVE;
    procedure CMMouseEnter(var Message: TMessage); message CM_MOUSEENTER;
  protected
    procedure DoShow; override;
  public
    property OnMouseClickOutside: TNotifyEvent read FOnMouseClickOutside
      write FOnMouseClickOutside;
  end;

...

procedure TDialog.CMMouseLeave(var Message: TMessage);
begin
  // CM_MOUSELEAVE is also send to the dialog when the mouse enters a control that
  // is within the dialog:
  if not PtInRect(BoundsRect, Mouse.CursorPos) then
  begin
    // Now the mouse is really outside the dialog. Start capturing it:
    MouseCapture := True;
    FMouseInDialog := False;
  end;
  inherited;
end;

procedure TDialog.CMMouseEnter(var Message: TMessage);
begin
  FMouseInDialog := True;
  // Only release capture when it had, otherwise it might affect another control:
  if MouseCapture then
    MouseCapture := False;
  inherited;
end;

procedure TDialog.DoShow;
begin
  inherited DoShow;
  // When mouse is outside the dialog when it should become visible, CM_MOUSELEAVE
  // isn't send because the mouse hasn't been inside yet. So also capture mouse
  // when the dialog is shown:
  MouseCapture := True;
end;

procedure TDialog.WMCaptureChanged(var Message: TMessage);
begin
 // When the dialog loses mouse capture and the mouse is outside the dialog, fire:
 if (not FMouseInDialog) and Assigned(FOnMouseClickOutside) then
    FOnMouseClickOutside(Self);
  inherited;
end;

This works. For both visible and obfuscated dialogs. But as David gratefully commented, this has consequences for controls which depend on mouse capture. There are not many that I know of and most controls like a memo or a menu bar will function normally. But take a combo box: when a combo box is dropped down, the list box captures the mouse. When it loses the mouse, the list is wrapped up. So when your users move the mouse outside the dialog (note that the dropped down list may bé outside the dialog), the combo box will exhibit non-default behaviour.

Secondly, to address the real problem a little more:

Furthermore, the question states specifically the need for this event in case of an hidden dialog. Well, the above mouse leaving and entering code depends on the dialog being visible, so let's forget about all of that, get rid of the drawbacks and reduce the code to:

type
  TDialog = class(TForm)
  private
    FOnMouseClickOutside: TNotifyEvent;
    procedure WMCaptureChanged(var Message: TMessage);
      message WM_CAPTURECHANGED;
  protected
    procedure DoShow; override;
  public
    property OnMouseClickOutside: TNotifyEvent read FOnMouseClickOutside
      write FOnMouseClickOutside;
  end;

...

procedure TDialog.DoShow;
begin
  inherited DoShow;
  MouseCapture := True;
end;

procedure TDialog.WMCaptureChanged(var Message: TMessage);
begin
  if Assigned(FOnMouseClickOutside) then
    FOnMouseClickOutside(Self);
  inherited;
end;

Now, what to do if the event fires? The dialog is still hidden, and a call to BringToFront does not work. (Trust me, I have tested it, although is was pretty nasty to reproduce a hidden dialog). What you should do is bring the dialog above all other windows with SetWindowPos:

procedure TAnyForm.MouseClickOutsideDialog(Sender: TObject);
begin
  if Sender is TDialog then
    SetWindowPos(TWinControl(Sender).Handle, HWND_TOPMOST, 0, 0, 0, 0,
      SWP_NOMOVE or SWP_NOSIZE or SWP_NOACTIVATE or SWP_NOOWNERZORDER);
end;

But since a dialog should always be shown on top of all others, you could rather eliminate the event completely and modify the code to:

type
  TDialog = class(TForm)
  private
    procedure CMShowingChanged(var Message: TMessage);
      message CM_SHOWINGCHANGED;
  end;

...

procedure TDialog.CMShowingChanged(var Message: TMessage);
begin
  if Showing then
    SetWindowPos(Handle, HWND_TOPMOST, 0, 0, 0, 0, SWP_NOMOVE or SWP_NOSIZE
      or SWP_NOACTIVATE or SWP_NOOWNERZORDER);
  inherited;
end;

In conclusion:

Now, this still does not work for message or system dialogs (although you could use these nice dialogs which do), and I have to agree with David to find out why the modal dialog becomes obfuscated. If you have forms with FormStyle = fsStayOnTop (or any window with HWND_TOPMOST as Z-order), then you shóuld use the following appropriate application methods to temporary compensate for these windows:

procedure TAnyForm.Button1Click(Sender: TObject);
var
  Dialog: TDialog;
begin
  Application.NormalizeAllTopMosts;
  Dialog := TDialog.Create(Application);
  try
    Dialog.ShowModal;
  finally
    Dialog.Free;
    Application.RestoreTopMosts;
  end;
end;

In all other cases, the disappearance of a modal dialog indicates that you are doing something out of the ordinary that probably cannot be handled by the VCL.