How to make VideoJS retry after errors

[aries-sd]

Description

We're seeing some issues where videojs with contrib-hls may lose connectivity to a live video source due to circumstances beyond its control such as network outage and similar. In this case videoJS stops playing and throws an error.

I need to sort out a way to capture this error and handle in my code. I would like to display a message that there is an issue and "reconnection is in progress" and then implement an exponential back off type re-try to reconnect the video source.

Basically I want to allow it to try as hard as it can to keep a stream up or re-establish a stream after interruption.

I've tried capturing errors thrown by the video tag by attaching to 'error' and capturing in my function. This works but I cannot determine how to re-start video JS on such a circumstance. I have tried using reset(), pause()/play(), load() etc in various patterns to attempt to re-initialize the player.

Is there a way to handle this currently? Or perhaps some inbuilt functionality that says "don't throw an error, just keep trying...."

Steps to reproduce

Connect to any HLS live stream and then interrupt your network.

Results

Expected

Would like to see a method or procedure to re-initialize the player to ask it to re-attempt playing the video.
Alternatively a functionality in the player that makes it aggressive about reconnections / automatically re-trying after an network (or media) error.

Actual

Player stops with network (or media) error indication.

Error output

If there are any errors at all, please include them here.

Additional Information

Please include any additional information necessary here. Including the following:

versions

videojs

Version 5.11.6
videojs-contrib-hls 3.5.3

browsers

All, eg firefox

OSes

what platforms (operating systems and devices) are affected?

plugins

videojs-contrib-hls 3.5.3

[brodjustice]

Would just like to +1 this. The only work around I have found is

1 Check for the player.readyState() for value 0 (HAVE NOTHING) as well as adding an error handler player.on('error', function(e) { ...

2 If either of the above occur then cause a complete page reload. All other attempts fail to reload the video including what I thought would definitely work:
player.dispose(); videojs(id);

More than happy to provide more info/debug if required.

[boushley]

Do you have any idea which library is failing?

The video element will send stalled events if it doesn't get data, but it shouldn't crash.

I'm guessing hls.js is the root that is getting into a nasty state. I am surprised that a dispose and re-initializaton doesn't solve the problem. Have you tried destroying the video element and creating a new one?

[brodjustice]

Actually, I can now re-initialize the video by doing an AJAX load of the video element and then calling player.dispose(); videojs(id);

The tricky part is figuring out when the the video is no longer going to restart/resync. By trial and error we have settled on running a JS function every 10 seconds which causes a complete reload via AJAX if any of the following are true:

1. player.error() is not null
2. player.ready() == 0
3. player.ready() == 1 for at least 30 seconds
4. player.ready() == 2 for at least 30 seconds

[Hailong]

Hi @brodjustice , I'm struggling with the same issue. Would you mind to provide a sample of your code?

[Hailong]

Put my code here following @brodjustice 's recommendation, while I'm not sure how to customize the error message, or simply hide it when retrying, can anybody help?

<script>
var myVideoPlayer = {
  checkInterval: 5, // seconds
  readyStateOneDuration: 0,
  readyStateTwoDuration: 0,

  healthCheck: function() {
    var error = this.player.error();
    console.log(error);
    if (error) {
      this.play();
      return;
    }

    var readyState = this.player.readyState();
    console.log(readyState);
    switch(readyState) {
      case 0:
        this.play();
        return;
      case 1:
        this.readyStateOneDuration += this.checkInterval;
        break;
      case 2:
        this.readyStateTwoDuration += this.checkInterval;
        break;
      default:
        return;
    }
    console.log(this.readyStateOneDuration);
    console.log(this.readyStateTwoDuration);
    if (this.readyStateOneDuration >= 30
      || this.readyStateTwoDuration >= 30) {
      this.play();
      return;
    }
  },

  play: function() {
    this.readyStateOneDuration = 0;
    this.readyStateTwoDuration = 0;
    try {
      // console.log('destroying old player');
      // this.player.dispose();
      this.player = null;
    } catch (e) {}
    this.player = videojs('ces-video');
    this.player.src({
      src: "http://yourdomain.com/video.m3u8",
      type: 'application/x-mpegURL',
      withCredentials: false
    });
    this.player.play();
  }
};
myVideoPlayer.play();
setInterval(function() {
  myVideoPlayer.healthCheck();
}, myVideoPlayer.checkInterval * 1000);
</script>

[brodjustice]

@Hailong Sorry about my late response, the holiday period and everything...

I cannot post my exact code here as it is all mixed up with other functions (yes, I know...). But next week I will try and re-factor it and post some proven working code.

[dstensnes]

@brodjustice How is the code sample going? 😄
I have run into the same problem, so I was curious to see how you solved it.

Thanks!

[kavin-90]

what i know videojs try to connect few times if video has network error after few try it show that network error

[brodjustice]

Sorry, I realize that I'm probably never going to get back to that project to rewrite the javascript. However, I can paste the nasty looking code here that actually works - hope it's all there as I had to cut and paste sections from all over the js file. If I ever get back to clean this up into a closure I'll post again but in the meantime this might give some hints - it needs jquery:

  // Your AJAX function to reload the video
  function reload_player(player, id){
    reload++;
    if(player != null){ player.dispose(); }
    $.ajax({
      url: "YOUR URL",
      dataType: 'script'
    }).done(function() {
      videojs(id);
    });
  }

  function player_error_functions(){
    var players = videojs.getPlayers()
    var id;
    var player;

    $('.rtmp-video-js').each(function(){
      id = $(this).attr('id');
      player = players[id]
      player.on('error', function(e) {
        e.stopImmediatePropagation();
        var error = this.player().error();
        post_status(this.player.id, error.message);
        post_status(this.player.id, 'Trying reload');
        reload_player(player, id);
      });
    });
  }

  var mon_loop = 0;
  var reload = 0;
  var readyState1 = 0;
  var readyState2 = 0;

  function monitor_players(){
    var players = videojs.getPlayers()
    var id;
    var player;

    $('.rtmp-video-js').each(function(){
      id = $(this).attr('id');
      player = players[id]

      if(player == null){
        reload_player(player, id);
      } else {
        var network_state = player.networkState();
        var player_error = player.error();
        var player_ready = player.readyState();
        var player_ready_str = '';

        player_ready_str = String(player_ready);
        if(player_ready == 1){ player_ready_str = '1/' + readyState1;  }
        if(player_ready == 2){ player_ready_str = '2/' + readyState2;  }
        post_status(player.id(), 'NS: ' + network_state + '<br/>E: ' + player_error + '<br/>RS: ' + player_ready_str + '</br>L: ' + mon_loop++ + '<br/>R: ' + reload);

        // Conditions under which to reload the player
        if(player_ready == 0){reload_player(player, id);}
        if(player_ready == 1){
          readyState1++;
          if(readyState1 > 2){ readyState1 = 0; reload_player(player, id); }
        }
        if(player_ready == 2){
          readyState2++;
          if(readyState2 > 2){ readyState2 = 0; reload_player(player, id); }
        }
        if(player_error != null){reload_player(player, id);}
      }
    });
  }

  function post_status(id, msg){
    if(id == null){
      $(".live:first").html(msg + '<br/>');
    } else {
      $('#rtmp_video_id_' + id.split('_').pop() + '_wrapper .live').html(msg + '<br/>');
    }
  }

  $(document).ready(function(){
    setTimeout(player_error_functions, 10000);
    setInterval(monitor_players, 10000);
  });

[mkhazov]

It was fixed in videojs-contrib-hls (reloadSourceOnError plugin):
videojs/videojs-contrib-hls#1030
videojs/videojs-contrib-hls#902

[gkatsev]

Yup, you basically want to listen to the error event and then set the source on the player again.

[sunnybear]

Actually it reloads source only once :( Need to get an endless loop.

[hoodsy]

@mkhazov @gkatsev I can't find any docs on how to import a plugin with videojs-contrib-hls and I get an error using reloadSourceOnError.

Do you know of any examples where this is used?

[rof20004]

Me too, I can't find any docs about reloadSourceOnError

[diegoje]

I've got the same issue here. Has anyone got a working example using reloadSourceOnError?

[rof20004]

@diegoje I am now using hls.js, this is the core lib for many players, including this, with hls.js I got it.

http://video-dev.github.io/hls.js/

[ghost]

@rof20004 I have the same problem right now. I can not get to the error messages ran. Can you explain your approach with hls.js?

[rof20004]

@badasscode
This is my entire code:

var isLoading = false;
var hideErrorMsg = true;
var video = document.getElementById('video-div');

func play() {
    if(Hls.isSupported()) {
        const config = {
            autoStartLoad: true,
            maxBufferSize: 1 * 1000 * 1000,
            manifestLoadingMaxRetry: 300000,
            manifestLoadingMaxRetryTimeout: 1000,
            levelLoadingMaxRetry: 300000,
            levelLoadingMaxRetryTimeout: 1000,
            fragLoadingMaxRetry: 300000,
            fragLoadingMaxRetryTimeout: 1000
        }
        
        var hls = new Hls(config);
        
        let retrying = false;
        const retry = setInterval(() => retryLiveStream(hls, url), 1000);
        
        video.onplaying = () => {
            isLoading = false;
            hideErrorMsg = false;
            clearInterval(retry);
            retrying = false;
            $scope.$apply(); // Update angular
        }
        
        hls.loadSource(url);
        hls.attachMedia(video);
        hls.on(Hls.Events.MANIFEST_PARSED,function() {
            video.play();
        });
    
        hls.on(Hls.Events.ERROR, function (event, data) {
            if (!hideErrorMsg) {
                isLoading = true;
                hideErrorMsg = true;
            }
            if (data.fatal) {
              switch(data.type) {
              case Hls.ErrorTypes.NETWORK_ERROR:
                console.log("fatal network error encountered, try to recover");
                if (!retrying) {
                    retry;
                }
                break;
              case Hls.ErrorTypes.MEDIA_ERROR:
                console.log("fatal media error encountered, try to recover");
                hls.recoverMediaError();
                break;
              }
            }
        });
    }
}

Retry method:

function retryLiveStream(hls, url) {
    retrying = true;
    hls.loadSource(url);
    hls.startLoad();
}

[rof20004]

hideErrorMsg is not important, do not use if you do not understand, but I used this variable only to show few messages.

[ghost]

Thx a lot 👍

[ssaguiar]

Hls.js has serious memory leak.
I was trying to use it in my project, wich is a mosaic with 20 live streams videos (for monitoring) and it increases the memory at the point of crashing the browser (my computer has 32G of ram memory).
After switching to videojs, I get, maximum, 1.2 to 1.4 G, but stays in 1.1 to 1.2 G most of the time.
The only problem is that when the source video stops and restart, the player stays waiting for it. it doesn't reconnect and I can't find some code example on how to use reloadSourceOnError.
I am using the lates version, as far I know.

[rof20004]

@ssaguiar

"wich is a mosaic with 20 live streams videos (for monitoring) and it increases the memory at the point of crashing the browser (my computer has 32G of ram memory)"

Maybe you need to change you goals with hls.js, see 20 live streams in same time in only one cpu is not usual.

[ssaguiar]

@rof20004
Nope, my friend.
Just to make an observation: I am encoding the 20 streams using ffmpeg and an nvidia 1050 video board, in h265, using an I7 with 32G of ram memory, The cpu used is about 25-30%, memory 6.9G and video csrd memory about 3.8G (this for 16 channels).
When I do so, I generate 2 streams for each input: one for production stream (wich is hd and has all audio channel, subtitles and so), and one small stream wich has no audios and no subtitles and one video with a size of 320x180 and 300Kbits max, wich is used for the monitoring. With this, to monitoring 20 channels (the 16 generated by the encoder and the last on repeated in mosaic 4 more times, to testing purposes), I need a max of 6 to 7 Mbits, wich is no problem because we use it in our internal network, wich has 100 Mbits.
Thus told, as I said before, with this setup using hls.js I had the problem I mention before, of memory leak, and now, with videojs this doesn't happens (memory consuption, with the 20 streams, in firefox, is about 1.2G).

[brainthinks]

@ssaguiar - here are the docs I found for reloadSourceOnError:

https://support.brightcove.com/hls-plugin

It won't solve your memory leak issue, though. I am working on something similar right now, which is a video monitoring site/tool, and memory management is a big issue when playing 12 - 36 streams simultaneously. From what I can tell, the virtual media source extension code in the newest hls plugin (https://github.com/videojs/http-streaming) doesn't "properly" handle all scenarios that can occur when using MSE, which contributes to the memory leak issues. I don't currently have a good workaround for this. We ended up not using HLS in the end because of the latency.