Batches Inference for neural network

Question

I have an asynchronous querying system with 100 images arrived in sequential (say 30;40;20;10). So the first arrival is 30 calling the API and feeds them to the first batch of 3