iPhone HTTP video streaming: How to show still image when playing audio-only stream

Question

I have an iPad app playing http video streaming using variant playlist, which have multi streams with different bit rate for different bandwidths. One of then is audio only stream for lowest bandwidth. The app use AVPlayer to play the video. For some reason, I can't use MPMoviePlayerViewController.

The problem I have is that when the video player play audio-only stream the video display black screen, the audio still playing. Which I think is awful to users, they don't know what going on. I want to display a still image in place of the video player. like this

Is there any way I can detect when the player switch to different stream? Or detect if the stream is audio-only?

Answer 1

Since I encountered this issue and the previous answer is not complete (doesn't cover live stream case) - here is my improvement on top of it:

if ([keyPath isEqual:@"tracks"])
    {
        BOOL hasVideoTrack = NO;
        for (AVPlayerItemTrack* track in [[yourPlayer.currentItem] tracks]) 
        {                
            if ([track.assetTrack.mediaType isEqual:AVMediaTypeVideo]) 
            {
                 hasVideoTrack = YES;
                 break;
            }
        }
        if (hasVideoTrack)
        {
           // Remove audio only view
        } else {
           // Show audio only view
        }
    }

Do note - this however will only let you show a local Audio only screen. When playing a live stream - the Artwork for the audio only is supposed to come from the stream, so my code is more towards this:

if ([keyPath isEqualToString:@"timedMetadata"] == YES){
for (AVMetadataItem *metadata in self.player.currentItem.timedMetadata) {
    if ([[metadata commonKey] isEqualToString:@"artwork"]) {
        UIImage *overlayImage = [UIImage imageWithData:metadata.dataValue];
        UIImageView *overlayImageView = [[UIImageView alloc] initWithImage:overlayImage];
        overlayImageView.contentMode = UIViewContentModeScaleAspectFit;

        // If an audio only slide is already there, make it disappear.
        [self hideAudioOnlySlide];

        self.audioOnlyView = overlayImageView;
        [self showAudioOnlySlide];
        self.audioOnlyView.size = _playerView.size;
        break;
    }
}


}else if ([keyPath isEqualToString:@"tracks"] == YES){
    NSArray *tracks = self.player.currentItem.tracks;            
    if ([self.player.currentItem hasVideoTracks] == NO) {
        // Check if there is timed metadata with artwork that indicates audio only is handled at the stream level.
        BOOL hasAudioOnlyFromStream = NO;
        for (AVMetadataItem *metadata in self.player.currentItem.timedMetadata) {
            if ([[metadata commonKey] isEqualToString:@"artwork"]) {
                hasAudioOnlyFromStream = YES;
                break;
            }
        }
        // If we don't have audio only slide from the stream - carry on to show audio only slide.
        //Otherwise - this is handled by the timed metadata check for artwork.
        if (hasAudioOnlyFromStream == NO) {
            [self showAudioOnlySlide];
        }
    } else {
        [self hideAudioOnlySlide];
    }
}

Add observation code:

[item addObserver:self forKeyPath:@"timedMetadata" options:0 context:NULL];
[item addObserver:self forKeyPath:@"tracks" options:0 context:NULL];

Remove observation code:

@try {
    [item removeObserver:self forKeyPath:@"timedMetadata"];
    [item removeObserver:self forKeyPath:@"tracks"];
}

hasVideoTracks code (Inside a category over AVPlayerItem):

- (BOOL)hasVideoTracks{
    BOOL hasVideoTracks = NO;
    for (AVPlayerItemTrack* track in [self tracks]){
        if ([track.assetTrack.mediaType isEqual:AVMediaTypeVideo]){
            hasVideoTracks = YES;
            break;
        }
    }

    return hasVideoTracks;
}

Note:

1. DON'T use presentationSize to detect audio only - in iOS 7 it no longer becomes CGSizeZero when moving to audio only - but gives some funky sizes instead. It's not a reliable method.
2. Show/Hide audioOnlySlide is pretty straight forward and should contain your implementation.

Answer 2

You can set up observer for the tracks property in AVPlayerItem.

[yourPlayer.currentItem addObserver:self forKeyPath:@"tracks" options:0 context:nil];

Than you need to implement method that will be called every time when tracks will be changed

- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
    if ([keyPath isEqual:@"tracks"])
    {
        for (AVPlayerItemTrack* track in [[yourPlayer.currentItem] tracks]) 
        {
            if ([track.assetTrack.mediaType isEqual:AVMediaTypeAudio]) 
            {
                 // Audio track available
            }
            if ([track.assetTrack.mediaType isEqual:AVMediaTypeVideo]) 
            {
                 // Video track available
            }
        }
    }
}

There some space for tuning up, take a look at NSKeyValueObserving Protocol and AVPlayerItem

Answer 3

Mastering Modern Media Playback (WWDC 2014):

AVPlayer and AVPlayerItem
Deciding when to show audio-only UI
// Inside -observeValueForKeyPath:ofObject:change:context: implementation...
if (presentationSizeObservationContext == context) {
    // Check if new presentation size is CGSizeZero.
    CGSize size = change[NSKeyValueChangeNewKey].sizeValue;
    if (CGSizeEqualToSize(size, CGSizeZero)) {
       for (AVPlayerItemTrack *playerItemTrack in playerItem.tracks) {
          AVAssetTrack *track = playerItemTrack.assetTrack;
          if ([track hasMediaCharacteristic:AVMediaCharacteristicAudible]) {
              // Show audio-only UI.
          } 
       }
    }
}

Link: https://developer.apple.com/videos/wwdc/2014/#503