( edit: I added the best working approach in my augmented reality framework and now also take the gyroscope into account which makes it much more stable again: Droid
It would be easier to test and debug Method 5 using GLU's lookAt function: http://www.opengl.org/sdk/docs/man2/xhtml/gluLookAt.xml
Also, as villoren suggested it's good to filter your sensor data, but it wouldn't really cause bugs if you move de device slowly. If you want to try, a simple one would be as follows:
newValue = oldValue * 0.9 + sensorValue * 0.1;
oldValue = newValue;
Note that if you are getting consistently wrong readings, you may have to calibrate your compass, by moving it with your wrists in a figure 8.
Hard to explain this in words; watch this video: http://www.youtube.com/watch?v=sP3d00Hr14o
Check out the Sensor fusion demo app which uses different sensors (gyroscope, rotation-vector, accelerometer + compass, etc.) and renders the outputs from the onSensorChanged-events as a coloured cube that rotates accordingly to your phone.
The results from those events are stored as quaternions and rotation matrices and used in this class which OpenGL.
You can use and-engine for Using sensors with OpenGL Just check the example https://github.com/nicolasgramlich/AndEngineExamples/tree/GLES2/src/org/andengine/examples/app/cityradar
I couldn't test the code yet (but I will, looks really interesting). One thing that caught my attention is that you don't seem to filter the sensor data in any way.
Sensor readings are very noisy by nature, specially the magnetic sensor. I'd suggest you implement some low pass filtering.
See my previous answer for further reading.
After analyze your code above, in method 5 you are assigning the orientation data as follows,
resultingAngles[1] = orientationData[0]; // orientation z axis to y axis
resultingAngles[2] = orientationData[1]; // orientation x axis to z axis
resultingAngles[0] = orientationData[2]; // orientation y axis to x axis
You have done rotation in y z x manner. Try to change the orientation..
I think it might be the problem there.. Please check and let me know..
Please refer the documentation for the event values, http://developer.android.com/guide/topics/sensors/sensors_position.html
Thanks for your tough work..