My computer is connected to a domain, but when I go to create a public queue:
MessageQueue.Create(@\".\\testqueue\");
I get this error:
On the server I was having trouble running MSMQ and getting different kinds of errors, including the error asked in the question.
A workgroup installation computer does not support the operation
What worked for me was not fiddling with Server Manager, but reinstalling MSMQ using Powershell.
Remove-WindowsFeature Msmq; Add-WindowsFeature MsMq
These two cmdlets can be run in a Powershell console running as Administrator. At least it fixed the error for me, but this will install the entire Msmq feature, including subfeatures.
Adding for documentation purpose... I was getting error "A workgroup installation computer does not support the operation" while trying to access transactional dead letter queue and it was due to not specifying the machine name. I was using period to denote computer name. e.g. "FORMATNAME:DIRECT=OS:.\SYSTEM$;DEADXACT". It does not work even with using complete format name. Problem solved after replacing the period with computer name. Below is the working code.
using (var queue = new MessageQueue($@"FORMATNAME:DIRECT=OS:{Environment.MachineName}\SYSTEM$;DEADXACT"))
{
queue.Purge();
}