Audio performance with Javafx for Android (MediaPlayer and NativeAudioService)

Question

I have created, with JavaFX, a game on desktop that works fine (20000 Java lines. As it is a game, the Real Time constraint is important (response time of player's actions).

The final aim is to run this application with Android. I have almost finished to "transfer the Java code" from PC to Android, even if I have encountered some real time trouble. I think almost all of them are solved now.

For instance, I have minimized the CPU time (consumption) of Shape or Rectangle.intersect(node1, node2) calls that are used for detecting impacts between two mobiles. Thus, the real time has been divided by 3. Great!

For testing this Android version, I use Eclipse + Neon2, JavaFX, JavaFXports + gluon and my phone (Archos Diamond S).

---

But, for Android phones, I had a real time problem related to the sounds that are generated with MediaPlayer and NativeAudioSrvice.

Yet, I have followed this advice that suggests the synchronous mode: javafxports how to call android native Media Player

1st question:

Does it exist an asynchronous mode with this Mediaplayer class?I think that would solve this latency problem? In practice, I have tried the asynchronous solution ... without success: the real time problem due to the audio generation with MediaPlayer stays: an audio generation costs from 50 ms to 80 ms whereas the main cyclic processing runs each 110 ms. Each audio generation can interfer with the main processing execution.

And, in each periodic task (rate: 110 ms), I can play several sounds like that. And, in a trace, there was up to six sound activations that take (together) about 300 ms (against the 110 ms of the main cyclic task )

QUESTION:

How to improve the performance of NativeAudio class (especially, the method play() with its calls that create the real time problem: setDataSource(...), prepare() and start() )?

---

THE SOLUTION

The main processing must be a "synchronized" method to be sure that this complete processing will be run, without any audio interruption.

More, each complete processing for generating a sound is under a dedicated thread, defined with a Thread.MIN_PRIORITY priority.

Now, the main processing is run each 110 ms and, when it begins, it cannot be disturbed by any audio generation. The display is very "soft" (no more jerky moving).

There is just a minor problem: when an audio seDataSource(), a start() or a prepare() method has begun, it seems to be that the next main processing shall wait the end of the method before beginning (TBC)

I hope this solution could help another people. It is applicable in any case of audio generations with MediaPlayer.

---

JAVA code of the solution

The main processing is defined like that:

public static ***synchronized*** void mainProcessing() {
// the method handles the impacts, explosions, sounds, movings, ... , in other words almost the entiere game  .. in a CRITICAL SECTION
}

/****************************************************/

In the NativeAudio class that implements "NativeAudioService":

    @Override
        public void play() {
            if (bSon) {
                Task<Void> taskSound = new Task<Void>() {

                @Override
                protected Void call() throws Exception {
                generateSound(); 
                return null;
                }}; 

              Thread threadSound = new Thread(taskSound);
              threadSound.setPriority(Thread.MIN_PRIORITY);
              threadSound.start();
            } 
        }


  /****************************************************/    
  private void generateSound() {
        currentPosition = 0;
        nbTask++;
        noTask = nbTask;
        try {
            if (mediaPlayer != null) {
                stop();
            }
            mediaPlayer = new MediaPlayer();

            AssetFileDescriptor afd = FXActivity.getInstance().getAssets().openFd(audioFileName);

            mediaPlayer.setDataSource(afd.getFileDescriptor(), afd.getStartOffset(), afd.getLength());      

            mediaPlayer.setAudioStreamType(AudioManager.STREAM_RING);

            float floatLevel = (float) audioLevel;
            mediaPlayer.setVolume(floatLevel, floatLevel);

            mediaPlayer.setOnCompletionListener(new OnCompletionListener() {
                @Override
                public void onCompletion(MediaPlayer mediaPlayer) {
                    if (nbCyclesAudio >= 1) {
                              mediaPlayer.start();
                          nbCyclesAudio--;
                        } else {
                        mediaPlayer.stop(); 
                        mediaPlayer.release(); // for freeing the resource - useful for the phone codec 
                        mediaPlayer = null;
                        }
                }
            });

            mediaPlayer.prepare();

            mediaPlayer.start();
            nbCyclesAudio--;

        } catch (IOException e) {
            }
      }

Answer 1

I've changed a little bit the implementation you mentioned, given that you have a bunch of short audio files to play, and that you want a very short time to play them on demand. Basically I'll create the AssetFileDescriptor for all the files once, and also I'll use the same single MediaPlayer instance all the time.

The design follows the pattern of the Charm Down library, so you need to keep the package names below.

EDIT

After the OP's feedback, I've changed the implementation to have one MediaPlayer for each audio file, so you can play any of them at any time.

1. Source Packages/Java:

package: com.gluonhq.charm.down.plugins

AudioService interface

public interface AudioService {
    void addAudioName(String audioName);
    void play(String audioName, double volume);
    void stop(String audioName);
    void pause(String audioName);
    void resume(String audioName);
    void release();
}

AudioServiceFactory class

public class AudioServiceFactory extends DefaultServiceFactory<AudioService> {

    public AudioServiceFactory() {
        super(AudioService.class);
    }

}

2. Android/Java Packages

package: com.gluonhq.charm.down.plugins.android

AndroidAudioService class

public class AndroidAudioService implements AudioService {

    private final Map<String, MediaPlayer> playList;
    private final Map<String, Integer> positionList;

    public AndroidAudioService() {
        playList = new HashMap<>();
        positionList = new HashMap<>();
    }

    @Override
    public void addAudioName(String audioName) {
        MediaPlayer mediaPlayer = new MediaPlayer();
        mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
        mediaPlayer.setOnCompletionListener(m -> pause(audioName)); // don't call stop, allows reuse
        try {
            mediaPlayer.setDataSource(FXActivity.getInstance().getAssets().openFd(audioName));
            mediaPlayer.setOnPreparedListener(mp -> {
                System.out.println("Adding  audio resource " + audioName);
                playList.put(audioName, mp);
                positionList.put(audioName, 0);
            });
            mediaPlayer.prepareAsync();
        } catch (IOException ex) {
            System.out.println("Error retrieving audio resource " + audioName + " " + ex);
        }

    }

    @Override
    public void play(String audioName, double volume) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            if (positionList.get(audioName) > 0) {
                positionList.put(audioName, 0);
                mp.pause();
                mp.seekTo(0);
            }
            mp.start();
        }

    }

    @Override
    public void stop(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.stop();
        }
    }

    @Override
    public void pause(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.pause();
            positionList.put(audioName, mp.getCurrentPosition());
        }
    }

    @Override
    public void resume(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.start();
            mp.seekTo(positionList.get(audioName));
        }
    }

    @Override
    public void release() {
        for (MediaPlayer mp : playList.values()) {
            if (mp != null) {
                mp.stop();
                mp.release();
            }
        }

    }

}

3. Sample

I've added five short audio files (from here), and added five buttons to my main view:

@Override
public void start(Stage primaryStage) throws Exception {

    Button play1 = new Button("p1");
    Button play2 = new Button("p2");
    Button play3 = new Button("p3");
    Button play4 = new Button("p4");
    Button play5 = new Button("p5");
    HBox hBox = new HBox(10, play1, play2, play3, play4, play5);
    hBox.setAlignment(Pos.CENTER);

    Services.get(AudioService.class).ifPresent(audio -> {

        audio.addAudioName("beep28.mp3");
        audio.addAudioName("beep36.mp3");
        audio.addAudioName("beep37.mp3");
        audio.addAudioName("beep39.mp3");
        audio.addAudioName("beep50.mp3");

        play1.setOnAction(e -> audio.play("beep28.mp3", 5));
        play2.setOnAction(e -> audio.play("beep36.mp3", 5));
        play3.setOnAction(e -> audio.play("beep37.mp3", 5));
        play4.setOnAction(e -> audio.play("beep39.mp3", 5));
        play5.setOnAction(e -> audio.play("beep50.mp3", 5));
    });

    Scene scene = new Scene(new StackPane(hBox), Screen.getPrimary().getVisualBounds().getWidth(), 
                        Screen.getPrimary().getVisualBounds().getHeight());
    primaryStage.setScene(scene);
    primaryStage.show();
}

@Override
public void stop() throws Exception {
    Services.get(AudioService.class).ifPresent(AudioService::release);
}

The prepare step takes place when the app is launched and the service is instanced, so when playing later on any of the audio files, there won't be any delay.

I haven't checked if there could be any memory issues when adding several media players with big audio files, as that wasn't the initial scenario. Maybe a cache strategy will help in this case (see CacheService in Gluon Charm Down).