I have a multi-threaded process where a file is shared (read and written) by multiple threads. Is there any way a thread can lock one file segment so that other threads cannot a
Yes - but not with the same mechanism. You'll have to use something like pthread mutexes, and keep track of the bookkeeping yourself.
Possible outline for how to make this work
Release the process-level mutex
Grab fnctl lock for process (if necessary)
Release fnctl lock to allow other processes to use the segment (if necessary)
Wait again on process-levelbookkeeping structure mutex (may not be necessary if you can mark it unused atomically)