Using multiple exo-players simultaneously

[dnutcracker]

Question:
I've been experimenting with playing multiple videos simultaneously but quickly got to the maximum of 4.
Using more than 4 resulted in failure to create a new MediaCodec (OMX.qcom.video.decoder.avc) - "Failed to allocate component instance".

Is there any way to bypass this limitation?

Thanks.

[martinbonnin]

Seems complicated to me.... Maybe you can try to fallback to software decoding. See https://github.com/google/ExoPlayer/blob/master/library/src/main/java/com/google/android/exoplayer/MediaCodecTrackRenderer.java#L289 for the place where the codec are created. You can try to add a fallback there that uses a software codec. As far as I can remember, they should be OMX.google.XXX.

But you wont have the same level of performance and battery life of course...

[ojw28]

There will be a limit on the maximum number of hardware decoders that you can instantiate, which you're hitting. Note that 4 is pretty generous. I believe some devices only allow 2. Your best bet is doing as Martin suggests, and modifying the code to use software decoders. You could implement this as a fallback (e.g. try and obtain the qcom one first, and use the google one if instantiation fails).

Whether this works well enough really depends on your use case. If you're trying to decode 5+ HD streams, that's likely just beyond the performance limitations of existing devices. If they're low resolution thumbnail-style videos, you might have more luck.

[dnutcracker]

Thank you for your help - this is also what I had in mind.

The strange thing is that on some devices (LG G2, HTC, Nexus 5) using a simple MediaPlayer + TextureView I have no problem rendering even 10+ videos simultaneously.
The videos playback is very smooth so I believe that hardware decoding is used - I can also assume that since there couldn't be 10+ dedicated hardware decoders there is some 'decoder sharing' going on by using the same decoder to decode multiple videos simultaneously.(On simpler devices like Samsung S2,S3 or even Nexus 7 - there is no 'decoder sharing' going on so you're limited to the number of hardware decoders that is available - according to my experience any attempt to play more than 3-4 video simultaneously will result in every thing getting stuck or even hard boot!)

I guess that to emulate this kind of behaviour in ExoPlayer would require a significant amount of non-trivial coding that would need to decode byte buffers from multiple videos using an available decoder and timing everything very tightly - I guess that this is beyond the scope of ExoPlayer but would make an interesting direction to explore :)

Please let me know what you think.

Thanks.

[martinbonnin]

What's your point ? MediaPlayer can decode 10+ videos and Exoplayer cannot ? If this is really the case, I don't believe too much in the 'sharing' thing. I would rather think that Exoplayer allocates bigger instances by default (since resolution could potentially change). That is just a blind guess though...

[ojw28]

Yeah, it's highly unlikely that there's any kind of clever sharing of decoders underneath MediaPlayer. Either the devices you mention also grant 10+ hardware decoders to ExoPlayer, or fewer are granted because ExoPlayer is allocating bigger instances as Martin says (if this is demonstrably the case, it's worth us investigating to see if we can allow more instances), or MediaPlayer is using software decoders and it just happens that they perform well.

Note that the term "hardware decoder" is used very loosely, and what it means varies from device to device. In general it's not true that a "hardware decoder instance" directly maps onto an actual dedicated physical piece of hardware. I guess "hardware accelerated decoder" would be a better term, where the actual hardware used, and the way it's used, can vary from chipset to chipset. For example a hardware accelerated decoder may require an allocation of GPU memory, and it's things like this that explain why allocating smaller instances may allow more instances to be granted.

[dnutcracker]

Martin, my point is very clear - I'm trying to understand how to utilize ExoPlayer for displaying 10+ videos while using "hardware accelerated decoding" (better term indeed!) since ExoPlayer is much more flexible to work than the (very) bad MediaPlayer.

ojw28, on the same device (LG G2) using MediaPlayer + TextureView allows rendering 10+ videos while ExoPlayer fails after 4 - so I guess it should be related to the size of instance allocation that you both mentioned - but where exactly do you control that? I see that the codec is instantiated only by its name.

[ojw28]

• It's probably better to validate that MediaPlayer isn't using software decoders first. Although I'm not sure I know of a great way to do that short of scouring through the logs looking for decoder instantiations (if the device logs them). I think it's the more likely of the possibilities. Your assumption that the decoders must be hardware accelerated because video is smooth seems tenuous. Another thing to check is what performance is like if you use 10 ExoPlayers where 6 of them are configured with the software decoder.
• There isn't a single size parameter as such. We provide information to the underlying decoder when we configure it, in the form of a MediaFormat. This happens in MediaCodecVideoTrackRenderer.configureCodec. The underlying decoder may or may not use this information to determine what resources it acquires (if it does, it would typically look at some subset of the values set in KEY_MAX_INPUT_SIZE, KEY_MAX_WIDTH, KEY_MAX_HEIGHT, KEY_WIDTH and KEY_HEIGHT).
• What's the actual use case for playing 10+ videos at once, out of interest? I can't think of a sane use case for wanting to do that, and I struggle to believe you'll find a good solution that works well across a wide range of devices, particularly at the low end. Wouldn't bandwidth also be a big concern?

[dnutcracker]

• My guess is that the MediaPlayer isn't really aware if the decoders it is using are software or hardware and it's probably up to the vendor's HAL implementation to decide what's happening after a certain number of instances - It could fail, it could provide software decoding, or it might be even sophisticated enough to reuse its hardware decoders in an intelligent way.
  Anyway, i'll try to figure out whether MediaPlayer is using software or hardware decoders in the 10+ videos scenario and update with the results.
• From a brief scan of the code I see that mediaExtractor.getTrackFormat(track) is used to determine the MediaFormat that is later given the configureCodec() method.
  How do you recommend me to test the assumption that smaller size may allow more decoders instances? You can assume that the video that I'm displaying are around thumbnail size.
• The use case in general is similar to a group chat - seeing each of the group member face as a separate video.(The more the merrier :))

[dnutcracker]

Btw - the solution of falling back to software decoding if the decoder allocation fails is not really feasible since the failure is not always reported.
For example - on Samsung S3, when you try to instantiate the 5th hardware decoder you get log cat error message from MediaCodec "Codec reported an error. (omx error 0x80001000, internalError -2147483648)" without getting any exception and the result is a disaster - the 5the video is not rendering and all the other 4 videos start to stutter until they come to a complete stop while the device is getting stuck.... :(

So I guess I'll need to force software rendering on all H264 video decoding in advance for ExoPlayer. (I already did it and works nicely - if you think it could help someone I can add it in a pull-request).
From what I tried - it seems that if the device is not very low-end - it handles rendering up to 8 small videos using software decoding quite nicely.

[ojw28]

• You may find that MediaCodec.createDecoderByType is clever enough to allocate hardware decoders up to the number allowed and then software ones after that. Worth a try using that rather than MediaCodec.createByCodecName.
• If ExoPlayer with only software renderers works, then I would say that makes it more likely that the MediaPlayer is using software renderers too (since the performance->hardware argument is no longer valid). The idea that the underlying platform might share decoders between multiple videos seems highly unlikely. I don't think it's theoretically possible to do this in the general case. There are special cases where it's possible to re-transmux multiple H264 streams into a single encoded stream, decode it with a single decoder and splice out the decoded frames, but my understanding is that this only works in very specific circumstances that aren't generally true (e.g. you'd definitely need every stream to have the exact same frame-rate for that to be possible, quite likely the same resolution too, and you'd have to synchronize frame releases across multiple player instances). It's crazily complicated to do this, and it's highly unlikely any OEM would have optimized such specific circumstances in this way.

[dnutcracker]

• You are correct -
  On devices that I was able to play many videos simultaneously MediaCodec.createDecoderByType was smart enough to provide a software renderer when decoder instance number was greater than the number of available hardware accelerated decoders... 👍

The problems starts on devices where MediaCodec.createDecoderByType attempts to create more hardware accelerated decoders than what's available on the device. Looks like it depends on how the ACodec.cpp was implemented or it might also be related to the Android version (I saw this behaviour only on API < 19) but I can't really say until I investigate this issue further.

Maybe you can consider allowing ExoPlayer to use MediaCodec.createDecoderByType instead of MediaCodec.createDecoderByName?

[HugoGresse]

@dnutcracker reading the issue, did you write your own MediaCodecTrackRenderer using a software decoder? If possible, can you describe the main parts, components, and maybe post the source ?

[dnutcracker]

No - I simply forced MediaCodecUtils.getDecoderInfo() to return a software decoder back to the ExoPlayer.

[HugoGresse]

@dnutcracker how did you find the software decoder name ?

[dnutcracker]

Just print all the available decoders and look for all that start with OMX.google

[marianpavel]

I am facing the same problem with stream of .mp3 files on a Samsung Galaxy S2, what I have read from all the comments here the problem seems to be the hardware decoding that s2 doesn't have and maybe other devices too. I already tried to use the mp3Extractor from dev-branch but nothing changed. One solution would be to change the hardware decoding into software but I need to do this change only on devices that don't support hardware. Is there a good approach to achieve this ?

I will come back with logcat from S2 and with some parts of my code.

[bongole]

I have the same problem on Nexus7 with multiple ExoPlayers.
I found cause of this problem in Android kernel header.

https://android.googlesource.com/kernel/msm.git/+/android-5.1.1_r0.10/include/media/msm/vidc_init.h#20

The VIDC_MAX_NUM_CLIENTS constant is 4.

[tavrinn]

How well does software decoding work? I have a similar requirement where I have to display 5 videos simultaneously. My videos are fairly small / low quality so it should theoretically play back ok. I tried adding OMX.google. instead of OMX and it returns the OMX.google.h264.decoder

Only problem is it won't play any videos...