@@ -45,7 +45,7 @@ The Engine API stores and provides following information:
...
@@ -45,7 +45,7 @@ The Engine API stores and provides following information:
-**Track Service**: Same as playlogs, but stripped-down information. Used for implementing a track service feature or displaying the currently playing track information on the radio's website.
-**Track Service**: Same as playlogs, but stripped-down information. Used for implementing a track service feature or displaying the currently playing track information on the radio's website.
-**Active Source**: In redundant deployment scenarios the API stores and shares information on which engine instance is currently active. This could be extended to other audio sources.
-**Active Source**: In redundant deployment scenarios the API stores and shares information on which engine instance is currently active. This could be extended to other audio sources.
-**Health Information**: In case of some critical issue affecting the functionality of AURA Engine, the history of health status records of the respective engine is stored.
-**Health Information**: In case of some critical issue affecting the functionality of AURA Engine, the history of health status records of the respective engine is stored.
-**Studio Clock**: Information on the current and next show to be used in a *Studio Clock* application.
-**Studio Clock**: Information on the current and next show to be used in a _Studio Clock_ application.
You can find details on the available API endpoints here: https://app.swaggerhub.com/apis/AURA-Engine/engine-api/1.0.0
You can find details on the available API endpoints here: https://app.swaggerhub.com/apis/AURA-Engine/engine-api/1.0.0
...
@@ -60,53 +60,53 @@ Engine can be deployed and run manually, using [Systemd](#running-with-systemd),
...
@@ -60,53 +60,53 @@ Engine can be deployed and run manually, using [Systemd](#running-with-systemd),
This is the most simple case. In that scenario the Engine API is deployed on the same host as the [Engine](https://gitlab.servus.at/aura/engine) itself.
This is the most simple case. In that scenario the Engine API is deployed on the same host as the [Engine](https://gitlab.servus.at/aura/engine) itself.
> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX,
> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX,
acting as a reverse proxy.
> acting as a reverse proxy.
<imgsrc="docs/engine-api_single.png"width="550"/>
<imgsrc="docs/engine-api_single.png"width="550"/>
### Redundant Deployment
### Redundant Deployment
In this scenario there are two Engine instances involved. Here you will need to deploy one Engine API on the host of each Engine instance. Additionally you'll have to set up
In this scenario there are two Engine instances involved. Here you will need to deploy one Engine API on the host of each Engine instance. Additionally you'll have to set up
a third, so-called *Synchronization Node* of the Engine API. This sync instance of Engine API is in charge of synchronizing playlogs and managing the active engine state.
a third, so-called _Synchronization Node_ of the Engine API. This sync instance of Engine API is in charge of synchronizing playlogs and managing the active engine state.
<imgsrc="docs/engine-api_redundancy.png"/>
<imgsrc="docs/engine-api_redundancy.png"/>
#### Managing Active Engine State
#### Managing Active Engine State
In order to avoid duplicate playlog storage, the *Synchronization Node* requires to know what the currently active Engine is. This can be achieved by some external *Status Monitor*
In order to avoid duplicate playlog storage, the _Synchronization Node_ requires to know what the currently active Engine is. This can be achieved by some external _Status Monitor_
component which tracks the heartbeat of both engines. In case the Status Monitor identifies one Engine as dysfunctional, it sends a REST request to the *Sync Node*, informing it
component which tracks the heartbeat of both engines. In case the Status Monitor identifies one Engine as dysfunctional, it sends a REST request to the _Sync Node_, informing it
about the second, functional Engine instance being activated.
about the second, functional Engine instance being activated.
The history of active Engine instances is stored in the database of the *Sync Node*. It is not only used for playlog syncing, but is also handy as an audit log.
The history of active Engine instances is stored in the database of the _Sync Node_. It is not only used for playlog syncing, but is also handy as an audit log.
> At the moment AURA doesn't provide its own *Status Monitor* solution. You'll need to integrate your self knitted component which tracks the heartbeat of the engines and posts
> At the moment AURA doesn't provide its own _Status Monitor_ solution. You'll need to integrate your self knitted component which tracks the heartbeat of the engines and posts
the active engine state to the *Sync Node*.
> the active engine state to the _Sync Node_.
> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX,
> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX,
acting as a reverse proxy to shield your API.
> acting as a reverse proxy to shield your API.
#### Playlog Synchronization for High Availability deployment scenarios
#### Playlog Synchronization for High Availability deployment scenarios
Usually when some new audio source starts playing, AURA Engine logs it to its local Engine API instance via some REST call. Now, the *Local API server* stores this information in its
Usually when some new audio source starts playing, AURA Engine logs it to its local Engine API instance via some REST call. Now, the _Local API server_ stores this information in its
local database. Next, it also performs a POST request to the *Synchronization API Server*. This *Sync Node* checks if this request is coming from the currently active engine instance.
local database. Next, it also performs a POST request to the _Synchronization API Server_. This _Sync Node_ checks if this request is coming from the currently active engine instance.
If yes, it stores this information in its playlog database. This keeps the playlogs of individual (currently active) Engine instances in sync with the *Engine API synchronization node*.
If yes, it stores this information in its playlog database. This keeps the playlogs of individual (currently active) Engine instances in sync with the _Engine API synchronization node_.
The *Engine API synchronization node* always only stores the valid (i.e. actually played) playlog records.
The _Engine API synchronization node_ always only stores the valid (i.e. actually played) playlog records.
##### Active Sync
##### Active Sync
This top-down synchronization process of posting any incoming playlogs at the *Engine Node* also to the *Synchronization Node* can be called **Active Sync**. This **Active Sync**
This top-down synchronization process of posting any incoming playlogs at the _Engine Node_ also to the _Synchronization Node_ can be called **Active Sync**. This **Active Sync**
doesn't work in every scenario, as there might be the case, that the *Synchronization Node* is not available e.g. due to network outage, maintenance etc. In this situation the playlog
doesn't work in every scenario, as there might be the case, that the _Synchronization Node_ is not available e.g. due to network outage, maintenance etc. In this situation the playlog
obviously can not be synced. That means the local playlog at the *Engine Node* is marked as "not synced".
obviously can not be synced. That means the local playlog at the _Engine Node_ is marked as "not synced".
##### Passive Sync
##### Passive Sync
Such marked entries are focus of the secondary synchronization approach, the so called **Passive Sync**: Whenever the *Synchronization Node* is up- and running again, some automated job
Such marked entries are focus of the secondary synchronization approach, the so called **Passive Sync**: Whenever the _Synchronization Node_ is up- and running again, some automated job
on this node is continuously checking for records on remote nodes marked as "unsynced". If there are such records found, this indicates that there has been an outage of the *Sync Node*.
on this node is continuously checking for records on remote nodes marked as "unsynced". If there are such records found, this indicates that there has been an outage of the _Sync Node_.
Hence those "unsynced" records are pending to be synced. Now an automated job on the *Sync Node* reads those records as batches from that Engine Node and stores them in its local database.
Hence those "unsynced" records are pending to be synced. Now an automated job on the _Sync Node_ reads those records as batches from that Engine Node and stores them in its local database.
It also keeps track when the last sync has happened, avoiding to query unnecceary records on any remote nodes.
It also keeps track when the last sync has happened, avoiding to query unnecceary records on any remote nodes.
In order to avoid that this **Passive Sync** job might be causing high traffic on an engine instance, these batches are read with some configured delay time (see `sync_interval` and
In order to avoid that this **Passive Sync** job might be causing high traffic on an engine instance, these batches are read with some configured delay time (see `sync_interval` and
`sync_step_sleep` in the *Sync Node* configuration; all values are in seconds) and a configurable batch size (`sync_batch_size`; count of max unsynced playlogs which are read at once).
`sync_step_sleep` in the _Sync Node_ configuration; all values are in seconds) and a configurable batch size (`sync_batch_size`; count of max unsynced playlogs which are read at once).
## Getting started
## Getting started
...
@@ -268,9 +268,9 @@ For production Engine API defaults to using the WSGI HTTP Server [`Gunicorn`](ht
...
@@ -268,9 +268,9 @@ For production Engine API defaults to using the WSGI HTTP Server [`Gunicorn`](ht
You might also want to pair Gunicorn with some proxy server, such as Nginx.
You might also want to pair Gunicorn with some proxy server, such as Nginx.
> Although there are many HTTP proxies available, we strongly advise that you use Nginx. If you choose another proxy
> Although there are many HTTP proxies available, we strongly advise that you use Nginx. If you choose another proxy
server you need to make sure that it buffers slow clients when you use default Gunicorn workers. Without this buffering
> server you need to make sure that it buffers slow clients when you use default Gunicorn workers. Without this buffering
Gunicorn will be easily susceptible to denial-of-service attacks. You can use Hey to check if your proxy is behaving properly.
> Gunicorn will be easily susceptible to denial-of-service attacks. You can use Hey to check if your proxy is behaving properly.
To run Gunicorn, you first need to create the Gunicorn configuration
To run Gunicorn, you first need to create the Gunicorn configuration
by copying the sample `./config/sample/gunicorn/sample-production.gunicorn.conf.py`
by copying the sample `./config/sample/gunicorn/sample-production.gunicorn.conf.py`
...
@@ -354,7 +354,7 @@ The project also contains a convenience script to get started with a one-liner
...
@@ -354,7 +354,7 @@ The project also contains a convenience script to get started with a one-liner
## Development
## Development
This project is based on a [Flask](https://flask.palletsprojects.com/) server using an [*API First*](https://swagger.io/resources/articles/adopting-an-api-first-approach/) approach.
This project is based on a [Flask](https://flask.palletsprojects.com/) server using an [_API First_](https://swagger.io/resources/articles/adopting-an-api-first-approach/) approach.
The API is specified using [Open API 3](https://swagger.io/specification/). The resulting API implementation is utilizing the popular [Connexion](https://github.com/zalando/connexion) library on top of Flask.
The API is specified using [Open API 3](https://swagger.io/specification/). The resulting API implementation is utilizing the popular [Connexion](https://github.com/zalando/connexion) library on top of Flask.
### Using the API
### Using the API
...
@@ -410,7 +410,7 @@ If you are a developer and want to create a local image, run
...
@@ -410,7 +410,7 @@ If you are a developer and want to create a local image, run
### Publish new image
### Publish new image
If you are developer and want to publish a new image to DockerHub, run
If you are developer and want to publish a new image to DockerHub, run
```bash
```bash
# Releasing the image to DockerHub
# Releasing the image to DockerHub
...
@@ -427,7 +427,6 @@ The Engine API logs can be found under `./logs`.
...
@@ -427,7 +427,6 @@ The Engine API logs can be found under `./logs`.
Automated Radio (AURA) is a open source software suite for community radio stations. Learn more about AURA in the [Meta repository](https://gitlab.servus.at/aura/meta).
Automated Radio (AURA) is a open source software suite for community radio stations. Learn more about AURA in the [Meta repository](https://gitlab.servus.at/aura/meta).
