I see this parameter in all kinds of places (forums, etc.) and the common answer it help highly concurrent servers. Still, I cannot find an official documentation from sun expla
butterchicken only explains half of the story, I would like to augment more detail to kmatveev's answer. Yes, the option is for thread state changes, and (pseudo) memory barriers are used to ensure that the change is visible from other threads, especially VM thread. Thread states used in OpenJDK6 are as follows:
// _thread_new : Just started, but not executed init. code yet (most likely still in OS init code)
// _thread_in_native : In native code. This is a safepoint region, since all oops will be in jobject handles
// _thread_in_vm : Executing in the vm
// _thread_in_Java : Executing either interpreted or compiled Java code (or could be in a stub)
...
_thread_blocked = 10, // blocked in vm
Without UseMembar option, in Linux, Hotspot uses memory serialize page instead of memory barrier instruction. Whenever a thread state transition happens, the thread writes to a memory address in memory serialize page with volatile pointer. When the VM thread needs to look at up-to-date state of all the threads, VM changes the protection bits for the memory serialize page to read only and then recovers it to read/write to serialize state changes. More detailed mechanism is introduced in the following page:
http://home.comcast.net/~pjbishop/Dave/Asymmetric-Dekker-Synchronization.txt