Skip to content

[Epic] Initial Requirements Testing of Engine Castor

Legend:

    • Test performed
  • ✅ Test success
  • ❌ Test failed

Sub Tasks

  • ✅ Install AURA Web using Docker (as per documentation)
  • ✅ Checkout this branch and install Engine Castor as described in the README.md on the mentioned branch.

1. Testing fallback

  • ✅ Put some music into the fallback folder of Engine Castor - Random Audio should be playing. Try to add more audio into the folder during play-out, the newly added music should also be played as part of the fallback (randomly).
  • ✅ Create some M3U playlist with fallback music for Engine Castor - Random Audio should be playing
  • ❌ Ensure the fallback is playing randomly in both variants. See #12

2. Test basic scheduling

  • ✅ Schedule some uploaded file to Dashboard > Engine Castor should play it at the exact time
  • ✅ Schedule some web radio stream in Dashboard > Engine Castor should play it at the exact time. Also try different types of Streams (format, bitrates, http/https etc.)
  • ✅ Connect some MP3 player or similar to the line-in of your audio interface. Configure this channel in Engine Castor and in the AURA Web Radio Station Settings. Schedule some analog line-in media source in Dashboard > Engine Castor should play it at the exact time; Observe the play-out for any latencies.
  • ❌ Schedule an episode with media sources exceeding the timeslot duration. Expected: The episodes fades-out at the end of the timeslot; any following episode starts playing, fades-in. Not working, because of #8 and #9

3. Test scheduling a sequence of episodes

  • ❌ Test sequence of different media sources: Schedule 3 episodes following each other. Each having a different type of media source (upload, livestream, line-in-1). Each episode should start without delay at the exact schedule time. At the end of these 3 schedule the fallback program should start again. Not working, see #9
  • ❌ Test sequence of same media sources: Schedule 6 episodes following each other. Each consecutive having a different type of media source (upload, upload, line-in-1, line-in-1, livestream-1, livestream-2). Each episode should start without delay at the exact schedule time. At the end of these 3 schedule the fallback program should start again. Not working, see #9
  • ❌ Test sequence of same media sources: Schedule 6 episodes following each other. Each consecutive having a different type of media source (upload, upload, line-in-1, line-in-2, livestream-1, livestream-2). Each episode should start without delay at the exact schedule time. At the end of these 3 schedule the fallback program should start again. Not working because of missing implementation of #8

4. Test scheduling of "default media sources"

  • ✅ Schedule 3 episodes following each other. The first and last do have a media source assigned. The middle one has no media sources assigned. Expected: The middle one plays fallback music, the other do playout what is assigned.
  • ✅ Schedule an episode where the show has a "default show media source assigned" (see the docs for info), but none on episode level. Expected: The playout airs the "default show media source".
  • ✅ Schedule an episode where the schedule has a "default schedule media source assigned" (see the docs for info), but none on episode level. Expected: The playout airs the "default schedule media source".
  • ✅ Schedule an episode having no episode media assigned - but both - a "default show media source assigned" and "default schedule media source assigned" (see the docs for info), Expected: The playout airs the "default schedule media source", not the one on show level.

5. Test scheduling multiple media sources per episode

  • ❌ Schedule 2 episodes, each having a mix of different media source types. Expected: Both episodes play on time, plus all media items within each episode play on time as well. Not working, see #9

6. Schedule a server-side M3U playlist

  • ✅ Manually create a M3U playlist in the playlist folder of Engine Castor. Schedule the playlist with the M3U playlist feature in dashboard by entering the path/name to the M3U playlist. The playout should correctly find the file and play its entries on time.
  • ❌ Schedule 3 episode one after each other. The middle one should hold a M3U playlist as described in the previous test. Everything should play on time. Test unsuccessfull, see #11

7. Testing live streams

  • ❌ Schedule a live stream but disable your internet connection before the scheduled playout time. Wait 5-10mins until to activate your internet connection. Playout should start picking up the stream as soon the internet connection is back. Test unsuccessfull, see #10
  • ❌ Try this even for longer periods, like if it's still working if there is no internet for 50mins of a 60mins timeslot. In that case the last 10mins should still be playing the planned stream. Test unsuccessfull, see #10

8. Testing silence detection

  • ✅ Schedule a live show with audio input coming from the line-in channel, but mute the audio at first. After 5mins un-mute the audio. Expected: The fallback music should play instantly at the beginning of the episode. After 5mins, when the line-in audio is back, the play-out should automatically start playing the line-in audio.
  • ✅ Perform the same as the previous test case, but using a file upload with 5mins of silence in the middle somewhere.
  • ✅ Schedule a live show with audio input coming from the line-in channel. In the middle of the show stop the line-in audio for e.g. 5mins. Expected: The fallback music should pick up after the configured time-out of the silence detector. After 5mins, when the line-in audio is back, the play-out should automatically return to that audio.
  • ✅ Schedule a live show with audio input coming from the line-in channel. Schedule a file upload episode immediately after that episode. In the last 5 mins of the live show, turn the line-in audio of. Expected: The fallback music should pick up after the configured time-out of the silence detector. Fallback should play until the end of the live show, fade-out and start playing and fade-in the next episode (file upload).
  • ✅ Schedule a live show with audio input coming from the line-in channel. In the middle of the show turn the volume of the line-in audio for e.g. 5mins way down. It should be almost not hearable, but some signal should come through. Expected: The fallback music should pick up after the configured time-out of the silence detector. After 5mins, when the line-in audio is back, the play-out should automatically return to that audio.
  • ✅ Prepare the line-in channel to play noise which is considered a microphone malfunction; no other audio source. Review the configuration for an option to set a "noise threshold". Schedule a live show with audio input coming from that line-in channel. Expected: The fallback music should pick up after the silence detector detects the given noise exceeds the configured noise treshhold.

9. Autonomous Playout

Since Engine caches calendar data locally, it is able to play autonomously without AURA Web connectivity up to 24 hours.

  • ✅ Schedule for example 3 episodes with different media sources: Schedule the first in 30mins, the second in 60mins and the third 2 hours in the future. After scheduling wait a little bit for Engine to read the scheduling information from AURA Web/Steering. After that shut the AURA Web Docker Compose down, so that it is not accessible to playout anymore. Expected: All the episodes should still play in time.

10. Heartbeat Monitoring

Engine is able to sent a heartbeat signal e.g. once per second to a monitoring server. A minimal sample server can be found in the engine repo under contrib/heartbeat-monitor.

  • Setup test heartbeat server.
  • ❌ Configure Engine Castor to send heartbeats to this server. At appears to be not implemented yet: #14
  • Monitor successful sending of heartbeats as long Engine Castor is up and running.

11. Playlogs

  • Start Engine API. Note: You may use the existing AURA Playout Docker Compose but comment out other services which are not needed for this test case and currently not compatible with AURA Web (engine, engine-core).
  • After starting Engine Castor any media scheduled should send playlogs to Engine API. Try this with different kinds of media sources and multiple episodes. Expected: The playlog table should be populated with all relevant information. Check if the logged start-time for each playlog is also matching the actual play-out time. You can query the playlogs with the %engine-api-host%/api/v1/playlog api endpoint (API Docs).
  • ❌ Similar to the previous test observe the correct logging of fallback music. Expected: The playlogs should contain relevant metadata from the audio files (e.g. MP3 tag information). Test unsuccessfull, see #13

12. Broadcast quality and loudness

  • ❌ Schedule various media sources and compare the perceived loudness with the one of fallback music. It should appear to be the same. Appears to be related to normalization, also compare engine-core#23 (closed)).
  • Schedule one line-in channel. There should be no crackling, delays or any other acoustic artifacts perceived as low-quality audio output.
  • ✅ Schedule one line-in channel. Turn the volume of the external device up & down. The delay of the played volume change should feel "almost" immediately. Some latencies are expected of course.
  • ❌ Schedule multiple, different line-in channels after each other. There should be no crackling, delays or any other acoustic artifacts perceived as low-quality audio output. Currently not testable because of #8

13. Test Icecast/Streaming Server connectivity

  • Either setup an Icecast Server locally or (which may be faster) use any free radio streaming server on the web (e.g. caster.fm may be working). Configure Engine Castor to connect to this streaming server.
  • Schedule different audio sources and listen to this streaming server. All audio should be correctly played on time. Observer if the meta-data provided by the scheduled media is also passed to the streaming server and displayed somewhere.
  • Schedule different live streams from public radio stations. Any meta-data provided by this stream should also be displayed in the end of chain streaming server.

14. Test recording capabilities

  • Configure the recording capabilities
  • ❌ Schedule multiple episodes with different variations of media sources. Some may have some fallback scenarios included. Every episodes should automatically be recorded to the configured recording folder (one file per episode). Test unsuccessfull, see #18
  • Review the recordings if they are exactly matching the timeslot duration. Ensure nothing is cut/or too long at the start and the end of the recording.
  • ❌ Try configuring different recording formats & qualities, like WAV, FLAC, MP3 and review the resulting files too. See #19

15. Test intervention during play-out

A pre-condition for these tests is dashboard#396 (closed).

  • During the broadcast of an episode, change the media source(s) in dashboard for this episode. Expected: After some expected timeout (playout is retrieving new API data like every 60 seconds), the playout should fade-out the currently playing item and fade-in the newly scheduled media source(s).
  • During the broadcast of an episode, remove the assignment of the episode to this broadcast date/timeslot and assign another episode with different media source(s). Expected: After some expected timeout (playout is retrieving new API data like every 60 seconds), the playout should fade-out the currently playing item and fade-in the newly scheduled media source(s).
  • During the broadcast of an episode, remove the assignment of the episode to this broadcast. Perform no new assignment Expected: After some expected timeout (playout is retrieving new API data like every 60 seconds), the playout should start playing fallback music.
  • During the broadcast of an episode, remove all media source(s) from that episode. Expected: After some expected timeout (playout is retrieving new API data like every 60 seconds), the playout should start playing fallback music.

16. Test graceful restarts

Pre-condition: Playout is not running for all of these tests. We want to test, what happens when play-out is started in the middle of an episode, like after a server crash.

  • Schedule an upload in dashboard. Start the playout after 10mins of the episode start time. Expected: After starting, the episode should not start from the beginning. Instead it pre-rolls to the 00:10:00 marker.
  • Schedule a stream media in dashboard. Start the playout after 10mins of the episode start time. Expected: After starting, the episode should start from the beginning, since there is no way to get the start of the stream.
  • Schedule a line-in media in dashboard. Start the playout after 10mins of the episode start time. Expected: After starting, the episode should play what is currently playing in the line-in channel.
Edited by David Trattnig