Re-try connecting to stream: Failure handling for invalid, temporary unavailable or broken Audio Streams

Currently connecting to some invalid stream URL is continuously retried, which breaks other schedules.

Steps to Reproduce

Test Case A:

  1. Create some timeslot with some invalid stream URL.
  2. Engine is scheduling (trying to pre-load) the timeslot with the stream source.
  3. Since the URL is not available (yet), log messages indicate that it is not possible and that it is going to retry.
  4. After the timeslot ends, the re-trying is still happening. The logs are polluted with re-try messages infinitely.

Test Case B:

  1. Create some timeslot with two media sources:
  2. Some invalid stream URL.
  3. Some valid file.
  4. Engine is scheduling (trying to pre-load) the timeslot with the stream source.
  5. Since the URL is not available (yet), log messages indicate that it is not possible and that it is going to retry.
  6. After the first media source ends, the re-trying is still happening. The logs are polluted with re-try messages infinitely.

Expected Result

Keep retrying, but only until the end of the playlist entry.

  • Test Case A: The next timeslot should start playing. No more re-try log message are happening.
  • Test Case B: The next media source (file) should start playing. No more re-try log message are happening.

Actual Result

The logs are polluted with re-try messages infinitely.

Other improvements to be done

  • Remove the configuration setting max_retries: 10 since it is automatically re-tried until the end of the playlist entry/media source.
  • Set a sensible default for retry_delay, like 30 seconds? As a consequence, there is a chance that scheduling may not happen correctly during pre-loading phase, because the stream might not be ready 15 seconds before planning broadcast time. But it might be ready 5 seconds before broadcast time. Having a longer retry delay, retries it only 30 seconds later. In a later improvement ticket, we could have a dual approach: when we are within proper scheduling timeframe, re-try more often. If it's "too late anyway", like we are already in the middle of some timeslot, retry less often.
Edited by David Trattnig