engine-features.md 12.2 KB
Newer Older
David Trattnig's avatar
David Trattnig committed
1
2
# Aura Engine Features

David Trattnig's avatar
TOC.    
David Trattnig committed
3
4
5
6
7
This page gives a more detailed overview of the Aura Engine features and how to configure them.

<!-- TOC -->

- [Aura Engine Features](#aura-engine-features)
8
9
10
11
    - [Multi-channel Input (Filesystem, Stream, Analog)](#multi-channel-input-filesystem-stream-analog)
    - [Multi-channel output](#multi-channel-output)
        - [Analog line-out](#analog-line-out)
        - [Stream to Icecast](#stream-to-icecast)
12
    - [Scheduling](#scheduling)
13
14
    - [Default Playlists](#default-playlists)
    - [Auto DJ](#auto-dj)
David Trattnig's avatar
David Trattnig committed
15
        - [Silence Detector](#silence-detector)
David Trattnig's avatar
TOC.    
David Trattnig committed
16
17
    - [Monitoring](#monitoring)
        - [Send mails on errors and warnings](#send-mails-on-errors-and-warnings)
David Trattnig's avatar
David Trattnig committed
18
        - [Engine Health Information via Engine API](#engine-health-information-via-engine-api)
David Trattnig's avatar
TOC.    
David Trattnig committed
19
        - [Engine Heartbeat](#engine-heartbeat)
20
        - [Logging](#logging)
David Trattnig's avatar
TOC.    
David Trattnig committed
21
22
23
    - [Read more](#read-more)

<!-- /TOC -->
David Trattnig's avatar
David Trattnig committed
24

25
## Multi-channel Input (Filesystem, Stream, Analog)
David Trattnig's avatar
David Trattnig committed
26

27
It's possible to schedules playlists with music or pre-recorded show stored on the **file system**,
28
29
via external **streams** or live from an **analog input** in the studio. All types of sources can
be mixed in a single playlist.
David Trattnig's avatar
David Trattnig committed
30

David Trattnig's avatar
David Trattnig committed
31
32
> Note: Any live sources or streams not specifing a length property, are automatically expanded to
the left duration of the timeslot.
David Trattnig's avatar
David Trattnig committed
33

34
The switching between types of audio source is handled automatically. To learn more check out the
35
[Scheduling](### Secure Scheduling) section.
David Trattnig's avatar
David Trattnig committed
36

37
## Multi-channel output
David Trattnig's avatar
David Trattnig committed
38

39
40
### Analog line-out

David Trattnig's avatar
David Trattnig committed
41
42
In most scenarios it might be sufficient to broadcast via only one analog stereo output.
If needed you can configure up to five stereo pairs.
43
44
45
46
47
48
49

### Stream to Icecast

Engine allows to stream to multiple Icecast Servers simultaniousely. It is also sending meta information
to the streaming server, using the *Icy* protocol.

To configure your Icecast connectivity check-out the `[stream]` section in your configuration.
David Trattnig's avatar
David Trattnig committed
50

51
## Scheduling
52

53
54
55
Engine provide a scheduling functionality by polling external API endpoints frequently. Those API endpoints
are provided by [Steering](https://gitlab.servus.at/aura/steering) to retrieve schedule information and
[Tank](https://gitlab.servus.at/aura/tank) to retrieve playlist information.
56

57
Scheduling is split into multiple phases. Below you see a timeline with one timeslot planned at a certain
58
59
point in time and the involved phase before:

David Trattnig's avatar
David Trattnig committed
60
61
```ascii
================= [   Scheduling Window   ] ============ [        Timeslot Play-out                 ] ====
62

David Trattnig's avatar
David Trattnig committed
63
64
65
66
== (FILESYSTEM A) ========================== [ Preload ] [  Play 4  ] ====================================
== (STREAM A) ========================================== [ Preload ] [ Play 1 ] ==========================
== (LIVE 1) ====================================================== [ Preload ] [ Play 1 ] ================
== (FILESYSTEM B) ========================================================== [ Preload ] [  Play 4  ] ====
67
68
69
70
71
```

- **Scheduling Window**: Within the scheduling window any commands for controlling
    the mixer of the soundsystem are prepared and queued.

72
    Only until the start of the window, timeslots can be updated or removed via external API Endpoints
David Trattnig's avatar
David Trattnig committed
73
74
75
76
    (e.g. using Steering or Dashboard). Until here any changes on the timeslot itself will be reflected
    in the actual play-out. This only affects the start and end time of the "timeslot" itself.
    It does not involve related playlists and their entries. Those can still be modified after the
    scheduling window has started.
77

78
    The start and the end of the window is defined by the start of the timeslot minus
79
    a configured amount of seconds (see `scheduling_window_start` and `scheduling_window_end`
80
81
    in `engine.ini`). The actual start of the window is calcuated by (timeslot start - window start)
    and the end by (timeslot end - window end)
82
83

    During the scheduling window, the external API Endpoints are pulled continiously, to
84
    check for updated timeslots and related playlists. Also, any changes to playlists and
85
86
87
88
89
    its entries are respected within that window (see `fetching_frequency` in `engine.ini`).

    > Important: It's vital that the the scheduling window is wider than the fetching frequency.
    Otherwise one fetch might never hit a scheduling window, hence not being able to schedule stuff.

90
- **Queuing and Pre-Loading**: Before any playlist entries of the timeslot can be turned into
David Trattnig's avatar
David Trattnig committed
91
92
93
    sound, they need to be queued and pre-loaded. Ideally the pre-loading happens somewhat before
    the scheduled play-out time to avoid any delays in timing. Set the maximum time reserved for
    pre-loading in your configuration (compare `preload_offset`in `engine.ini`).
94

95
96
97
98
99
100
101
    If there is not enough time to reserve the given amount of time for preloading (i.e. some entry
    should have started in the past already) the offset is ignored and the entry is played as soon as possible.

    > Important: To ensure proper timings, the offset should not exceed the time between the start of
    the scheduling-window and the start of the actual timeslot playout. Practically, of course there
    are scenario where playout start later than planned e.g. during startup of the engine during a timeslot
    or due to some severe connectivity issues to some external stream.
David Trattnig's avatar
David Trattnig committed
102

103
- **Play-out**: Finally the actual play-out is happening. The faders of the virtual mixers are pushed
David Trattnig's avatar
David Trattnig committed
104
    all the way up, as soon it's "time to play" for one of the pre-loaded entries.
105
    Transitions between playlist entries with different types of sources (file, stream and analog
106
    inputs) are performed automatically. At the end of each timeslot the channel is faded-out,
107
108
    no matter if the total length of the playlist entries would require a longer timeslot.

109
    If for some reason the playout is corrupted, stopped or too silent to make any sense, then
David Trattnig's avatar
David Trattnig committed
110
    this <u>triggers a fallback using the silence detector</u> (see chapter below).
David Trattnig's avatar
David Trattnig committed
111

112
## Default Playlists
David Trattnig's avatar
David Trattnig committed
113

114
115
While a timeslot can have a specific playlist assigned, it's also possible to define default playlists
for schedules and shows:
David Trattnig's avatar
David Trattnig committed
116

117
118
- **Default Schedule Playlist**: This playlist is defined on the level of some recurrence rules (*Schedule*).
    In case the timeslot doesn't have any specific playlist assigned, this playlist is broadcasted.
David Trattnig's avatar
David Trattnig committed
119

120
121
- **Default Show Playlist**: This playlist can be assigned to some show. If neither the specific timeslot
    playlist nor the default schedule playlist is specificed the *default show playlist* is broadcasted.
David Trattnig's avatar
David Trattnig committed
122
123


124
![Setting a default show playlist in AURA Dashboard](images/dashboard-fallback-setting.png "Default Show Playlist in Dashboard")
David Trattnig's avatar
David Trattnig committed
125

126
127
If none of these playlists have been specified the *Fallback Handling* mechanism takes over (see next chapter).
## Auto DJ
David Trattnig's avatar
David Trattnig committed
128

129
130
131
132
133
134
135
136
137
138
139
There might be scenarios where some user has forgotting to assign a playlist to a timeslot or even
missed to define a timeslot. To avoid such broadcast ending up in [Dead Air](https://en.wikipedia.org/wiki/Dead_air), 
Engine provides a *Fallback Handling* or *Auto DJ* feature.

There a two ways to feed music into the Auto DJ:

- **Audio Folder on your harddrive**
- **[M3U Playlist](https://en.wikipedia.org/wiki/M3U)**

Both are watched and automatically updated upon content change. Media is played in an randomized order, meaning
they are shuffled and played until nothing is left for the given playlist/folder. Then it starts all over again.
David Trattnig's avatar
David Trattnig committed
140

David Trattnig's avatar
David Trattnig committed
141
To configure the behavior of fallbacks, check out the `[fallback]` section in your `engine.ini` configuration.
142

143
144
145
146
147
148
149
150
151
152
153
154
155
Please note that engine also extracts meta-information from music played by the Auto DJ and updates the *Studio
Clock* and *Track Service* feature accordingly.

It's also possible to report incidents where Auto DJ starts playing to e.g. programme coordinators. To do so
you simple have to define a mail server (see section [Monitoring](#Monitoring)) and a valid email address:

```ini
# Set to "true" if you want to notify programme-coordinators about about fallback situations, otherwise "false"
mail_coordinator_enabled="true"
# If you want to address multiple programme-coordinators separate their emails by space
coordinator_mail="programme-coordinator@your-radio.org"
```

David Trattnig's avatar
David Trattnig committed
156
### Silence Detector
157

158
159
160
In order to avoid [Dead Air](https://en.wikipedia.org/wiki/Dead_air), the aforementioned Auto DJ feature is
triggered using a Silence Detector reacting to situations where no or unwanted sound is on air. The Silence
Detector allows detection of absoulte silence, weak signals or even noise.
161

David Trattnig's avatar
David Trattnig committed
162
To configure the sensitivity of the Silence Detector adapt following properties in
163
`engine.ini`:
164

David Trattnig's avatar
David Trattnig committed
165
```ini
166
167
168
# max_blank => maximum time of blank from source (defaults to 20., seconds, float)
# min_noise => minimum duration of noise on source to switch back over (defaults to 0, seconds, float)
# threshold => power in dB under which the stream is considered silent (defaults to -40., float)
David Trattnig's avatar
David Trattnig committed
169
fallback_max_blank="10."
170
171
172
fallback_min_noise="0."
fallback_threshold="-50."
```
173

David Trattnig's avatar
David Trattnig committed
174
175
176
177
178
179
180
181
182
183
184
Let's assume you want to broadcast a live show. At the given example the Silence Detector will
react upon 10 seconds of silence, it will evaluate if there is a schedule or show fallback playlist
defined. If yes, it will immediately schedule and play such playlist. If there is no such playlist
the station fallback will kick in by playing any randomized music.

As soon some signal from the live source is sensed again, the audio routing switches back to the live
channel.

> Note, all these fallback source are fully integrated with the Engine's playlog and track-service
feature including indications from which fallback level some audio is broadcasted.

David Trattnig's avatar
David Trattnig committed
185
186
187
188
## Monitoring

You have following options to monitor the Engine:

David Trattnig's avatar
David Trattnig committed
189
190
191
192
- Send mails on errors and warnings
- Engine Status Information
- Engine Heartbeat
- Logging
David Trattnig's avatar
David Trattnig committed
193
194
195
196
197
198
199
200
201
202
203
204
205

### Send mails on errors and warnings

To activate you'll need to set some mail account within the `[monitoring]` section of your configuration.

```ini
[monitoring]
mail_server="mail.o94.at"
mail_server_port="587"
mail_user="aura@o94.at"
mail_pass="---SECRET--PASSWORD---"

# If you want to send multiple adminmails, make them space separated
David Trattnig's avatar
David Trattnig committed
206
admin_mail="admin-email@your.domain"
David Trattnig's avatar
David Trattnig committed
207
208
209
210
211
212
213
214

# Which from mailadress should be used
from_mail="monitoring@aura.engine"

# The beginning of the subject. With that you can easily apply filter rules using a mail client
mailsubject_prefix="[Aura Engine]"
```

David Trattnig's avatar
David Trattnig committed
215
### Engine Health Information via Engine API
David Trattnig's avatar
David Trattnig committed
216

David Trattnig's avatar
David Trattnig committed
217
218
Whenever the Engine's status turns into some unhealthy state this is logged to [Engine API](https://gitlab.servus.at/aura/engine-api).
Also, when it returns to some valid state this is logged to the Engine API.
David Trattnig's avatar
David Trattnig committed
219
220
221
222
223
224
225
226
227
228
229
230
231

### Engine Heartbeat

Instead of checking all status properties, the Heartbeat only validates the vital ones
required to run the engine. If all of those are valid, as network socket request is sent
to a defined server.

This heartbeat is sent continiously based on the configured `heartbeat_frequency`.

```ini
# Server where heartbeat info is sent to
heartbeat_server = "127.0.0.1"
# Some UDP port
David Trattnig's avatar
TOC.    
David Trattnig committed
232
heartbeat_port = 43334
David Trattnig's avatar
David Trattnig committed
233
234
235
236
237
238
239
240
241
# Seconds how often the vitality of the Engine should be checked (0 = disabled)
heartbeat_frequency = 1
```

The service receiving this heartbeat ticks can decide what to do with that information.
One scenario could be switching to another Engine instance or any other failover scenario.
Under `contrib/heartbeat-monitor` you'll find some sample application digesting these
heartbeat signals.

242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
### Logging

In development all Engine logs can be found under `./log`, and for production they can
are located in `/var/log/aura`. Adapt the log-level within your configuration to get
more or less verbose log output:

```ini
logdir="/var/log/aura"

# Possible values: debug, info, warning, error, critical
loglevel="info"
```

The log directory holds individual logs from Engine Core, Liquidsoap and the API.
But also `stout` outputs from supervisor services are written there.

Additionally you'll finde Supervisor specific logs under`/var/log/supervisor`.

David Trattnig's avatar
David Trattnig committed
260
261


262
263
264
265
266
267
## Read more

- [Overview](/README.md)
- [Installation for Development](installation-development.md)
- [Installation for Production](installation-production.md)
- [Running with Docker](running-docker.md)
268
- [Setup the Audio Store](docs/setup-audio-store.md)
269
270
- [Developer Guide](developer-guide.md)
- [Engine Features](engine-features.md)
David Trattnig's avatar
David Trattnig committed
271
- [Frequently Asked Questions (FAQ)](docs/frequently-asked-questions.md)