engine-features.md

# Aura Engine Features

This page gives a more detailed overview of the Aura Engine features and how to configure them.

<!-- TOC -->

- [Aura Engine Features](#aura-engine-features)
    - [Multi-channel Input (Filesystem, Stream, Analog)](#multi-channel-input-filesystem-stream-analog)
    - [Multi-channel output](#multi-channel-output)
        - [Analog line-out](#analog-line-out)
        - [Stream to Icecast](#stream-to-icecast)
    - [Scheduling](#scheduling)
    - [Default Playlists](#default-playlists)
    - [Auto DJ](#auto-dj)
        - [Silence Detector](#silence-detector)
    - [Monitoring](#monitoring)
        - [Send mails on errors and warnings](#send-mails-on-errors-and-warnings)
        - [Engine Health Information via Engine API](#engine-health-information-via-engine-api)
        - [Engine Heartbeat](#engine-heartbeat)
        - [Logging](#logging)
    - [Read more](#read-more)

<!-- /TOC -->

## Multi-channel Input (Filesystem, Stream, Analog)

It's possible to schedules playlists with music or pre-recorded show stored on the **file system**,
via external **streams** or live from an **analog input** in the studio. All types of sources can
be mixed in a single playlist.

> Note: Any live sources or streams not specifing a length property, are automatically expanded to
the left duration of the timeslot.

The switching between types of audio source is handled automatically. To learn more check out the
[Scheduling](### Secure Scheduling) section.

## Multi-channel output

### Analog line-out

In most scenarios it might be sufficient to broadcast via only one analog stereo output.
If needed you can configure up to five stereo pairs.

### Stream to Icecast

Engine allows to stream to multiple Icecast Servers simultaniousely. It is also sending meta information
to the streaming server, using the *Icy* protocol.

To configure your Icecast connectivity check-out the `[stream]` section in your configuration.

## Scheduling

Engine provide a scheduling functionality by polling external API endpoints frequently. Those API endpoints
are provided by [Steering](https://gitlab.servus.at/aura/steering) to retrieve schedule information and
[Tank](https://gitlab.servus.at/aura/tank) to retrieve playlist information.

Scheduling is split into multiple phases. Below you see a timeline with one timeslot planned at a certain
point in time and the involved phase before:

```ascii
================= [   Scheduling Window   ] ============ [        Timeslot Play-out                 ] ====

== (FILESYSTEM A) ========================== [ Preload ] [  Play 4  ] ====================================
== (STREAM A) ========================================== [ Preload ] [ Play 1 ] ==========================
== (LIVE 1) ====================================================== [ Preload ] [ Play 1 ] ================
== (FILESYSTEM B) ========================================================== [ Preload ] [  Play 4  ] ====
```

- **Scheduling Window**: Within the scheduling window any commands for controlling
    the mixer of the soundsystem are prepared and queued.

    Only until the start of the window, timeslots can be updated or removed via external API Endpoints
    (e.g. using Steering or Dashboard). Until here any changes on the timeslot itself will be reflected
    in the actual play-out. This only affects the start and end time of the "timeslot" itself.
    It does not involve related playlists and their entries. Those can still be modified after the
    scheduling window has started.

    The start and the end of the window is defined by the start of the timeslot minus
    a configured amount of seconds (see `scheduling_window_start` and `scheduling_window_end`
    in `engine.ini`). The actual start of the window is calcuated by (timeslot start - window start)
    and the end by (timeslot end - window end)

    During the scheduling window, the external API Endpoints are pulled continiously, to
    check for updated timeslots and related playlists. Also, any changes to playlists and
    its entries are respected within that window (see `fetching_frequency` in `engine.ini`).

    > Important: It's vital that the the scheduling window is wider than the fetching frequency.
    Otherwise one fetch might never hit a scheduling window, hence not being able to schedule stuff.

- **Queuing and Pre-Loading**: Before any playlist entries of the timeslot can be turned into
    sound, they need to be queued and pre-loaded. Ideally the pre-loading happens somewhat before
    the scheduled play-out time to avoid any delays in timing. Set the maximum time reserved for
    pre-loading in your configuration (compare `preload_offset`in `engine.ini`).

    If there is not enough time to reserve the given amount of time for preloading (i.e. some entry
    should have started in the past already) the offset is ignored and the entry is played as soon as possible.

    > Important: To ensure proper timings, the offset should not exceed the time between the start of
    the scheduling-window and the start of the actual timeslot playout. Practically, of course there
    are scenario where playout start later than planned e.g. during startup of the engine during a timeslot
    or due to some severe connectivity issues to some external stream.

- **Play-out**: Finally the actual play-out is happening. The faders of the virtual mixers are pushed
    all the way up, as soon it's "time to play" for one of the pre-loaded entries.
    Transitions between playlist entries with different types of sources (file, stream and analog
    inputs) are performed automatically. At the end of each timeslot the channel is faded-out,
    no matter if the total length of the playlist entries would require a longer timeslot.

    If for some reason the playout is corrupted, stopped or too silent to make any sense, then
    this <u>triggers a fallback using the silence detector</u> (see chapter below).

## Default Playlists

While a timeslot can have a specific playlist assigned, it's also possible to define default playlists
for schedules and shows:

- **Default Schedule Playlist**: This playlist is defined on the level of some recurrence rules (*Schedule*).
    In case the timeslot doesn't have any specific playlist assigned, this playlist is broadcasted.

- **Default Show Playlist**: This playlist can be assigned to some show. If neither the specific timeslot
    playlist nor the default schedule playlist is specificed the *default show playlist* is broadcasted.


![Setting a default show playlist in AURA Dashboard](images/dashboard-fallback-setting.png "Default Show Playlist in Dashboard")

If none of these playlists have been specified the *Fallback Handling* mechanism takes over (see next chapter).
## Auto DJ

There might be scenarios where some user has forgotting to assign a playlist to a timeslot or even
missed to define a timeslot. To avoid such broadcast ending up in [Dead Air](https://en.wikipedia.org/wiki/Dead_air), 
Engine provides a *Fallback Handling* or *Auto DJ* feature.

There a two ways to feed music into the Auto DJ:

- **Audio Folder on your harddrive**
- **[M3U Playlist](https://en.wikipedia.org/wiki/M3U)**

Both are watched and automatically updated upon content change. Media is played in an randomized order, meaning
they are shuffled and played until nothing is left for the given playlist/folder. Then it starts all over again.

To configure the behavior of fallbacks, check out the `[fallback]` section in your `engine.ini` configuration.

Please note that engine also extracts meta-information from music played by the Auto DJ and updates the *Studio
Clock* and *Track Service* feature accordingly.

It's also possible to report incidents where Auto DJ starts playing to e.g. programme coordinators. To do so
you simple have to define a mail server (see section [Monitoring](#Monitoring)) and a valid email address:

```ini
# Set to "true" if you want to notify programme-coordinators about about fallback situations, otherwise "false"
mail_coordinator_enabled="true"
# If you want to address multiple programme-coordinators separate their emails by space
coordinator_mail="programme-coordinator@your-radio.org"
```

### Silence Detector

In order to avoid [Dead Air](https://en.wikipedia.org/wiki/Dead_air), the aforementioned Auto DJ feature is
triggered using a Silence Detector reacting to situations where no or unwanted sound is on air. The Silence
Detector allows detection of absoulte silence, weak signals or even noise.

To configure the sensitivity of the Silence Detector adapt following properties in
`engine.ini`:

```ini
# max_blank => maximum time of blank from source (defaults to 20., seconds, float)
# min_noise => minimum duration of noise on source to switch back over (defaults to 0, seconds, float)
# threshold => power in dB under which the stream is considered silent (defaults to -40., float)
fallback_max_blank="10."
fallback_min_noise="0."
fallback_threshold="-50."
```

Let's assume you want to broadcast a live show. At the given example the Silence Detector will
react upon 10 seconds of silence, it will evaluate if there is a schedule or show fallback playlist
defined. If yes, it will immediately schedule and play such playlist. If there is no such playlist
the station fallback will kick in by playing any randomized music.

As soon some signal from the live source is sensed again, the audio routing switches back to the live
channel.

> Note, all these fallback source are fully integrated with the Engine's playlog and track-service
feature including indications from which fallback level some audio is broadcasted.

## Monitoring

You have following options to monitor the Engine:

- Send mails on errors and warnings
- Engine Status Information
- Engine Heartbeat
- Logging

### Send mails on errors and warnings

To activate you'll need to set some mail account within the `[monitoring]` section of your configuration.

```ini
[monitoring]
mail_server="mail.o94.at"
mail_server_port="587"
mail_user="aura@o94.at"
mail_pass="---SECRET--PASSWORD---"

# If you want to send multiple adminmails, make them space separated
admin_mail="admin-email@your.domain"

# Which from mailadress should be used
from_mail="monitoring@aura.engine"

# The beginning of the subject. With that you can easily apply filter rules using a mail client
mailsubject_prefix="[Aura Engine]"
```

### Engine Health Information via Engine API

Whenever the Engine's status turns into some unhealthy state this is logged to [Engine API](https://gitlab.servus.at/aura/engine-api).
Also, when it returns to some valid state this is logged to the Engine API.

### Engine Heartbeat

Instead of checking all status properties, the Heartbeat only validates the vital ones
required to run the engine. If all of those are valid, as network socket request is sent
to a defined server.

This heartbeat is sent continiously based on the configured `heartbeat_frequency`.

```ini
# Server where heartbeat info is sent to
heartbeat_server = "127.0.0.1"
# Some UDP port
heartbeat_port = 43334
# Seconds how often the vitality of the Engine should be checked (0 = disabled)
heartbeat_frequency = 1
```

The service receiving this heartbeat ticks can decide what to do with that information.
One scenario could be switching to another Engine instance or any other failover scenario.
Under `contrib/heartbeat-monitor` you'll find some sample application digesting these
heartbeat signals.

### Logging

In development all Engine logs can be found under `./log`, and for production they can
are located in `/var/log/aura`. Adapt the log-level within your configuration to get
more or less verbose log output:

```ini
logdir="/var/log/aura"

# Possible values: debug, info, warning, error, critical
loglevel="info"
```

The log directory holds individual logs from Engine Core, Liquidsoap and the API.
But also `stout` outputs from supervisor services are written there.

Additionally you'll finde Supervisor specific logs under`/var/log/supervisor`.



## Read more

- [Overview](/README.md)
- [Installation for Development](installation-development.md)
- [Installation for Production](installation-production.md)
- [Running with Docker](running-docker.md)
- [Setup the Audio Store](docs/setup-audio-store.md)
- [Developer Guide](developer-guide.md)
- [Engine Features](engine-features.md)
- [Frequently Asked Questions (FAQ)](docs/frequently-asked-questions.md)