I wish do several little projects experimenting with video and audio streaming from client to server and from client-server-multiples points, I have several questions:
WebRTC is made for browsers. You said it right:
or the benefit in webrtc is avoid middle communication
It is especially made for browsers. You can make a connection to your server, but then it must have a UI (ubuntu server with GUI), and install a browser (but that is not what you want I guess). Next to that there is no other way to stream to your server than websockets.
According to the other answer, it is possible to stream to and from RTC enabled servers
About multiple streams, that is a but hard. I answered a question about using webrtc as a multiple peer to peer. Maybe that is what you can do, but will require a good code managing who shall connect to who, and just chain your users. Server is in that case the best (and if you want to go with browser compatibility, websockets are a bit more supported than webRTC, even IE supports them now (?!?!).
You might want a conclusion:
Please note that webRTC requires some data exchange before it can start. You can do this with a websocket server in node.js faily easy.
Hope this will help you further in your development process, and I hope to hear from you what kind of solution you are going to use!