From dbd799b1036ff99ad12006123fe339fc57d7db04 Mon Sep 17 00:00:00 2001 From: David Trattnig <david.trattnig@o94.at> Date: Mon, 24 Aug 2020 13:39:32 +0200 Subject: [PATCH] Extended documation. --- README.md | 84 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 56 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index f9cd2d7..f476edb 100644 --- a/README.md +++ b/README.md @@ -6,33 +6,59 @@ The Project serves the Engine API and handles state management of multiple [Engi The Engine API stores and provides following information: -- Playlogs: A history of all audio titles being played by the Engine. This is used for detailed reports. +- Playlogs: A history of all audio titles being played by the Engine. This is used for example to generate detailed reports for regulartory purposes. - Track Service: Same as track service, but stripped-down information. Used for implementing a track service feature on the radio's website. -- Active Source: In redundant deployment scenarios the API stores and shares information on which engine instance is active. This could be extended to other audio sources. -- Health Information: In case of some critical issue, the health status of the respective engine is stored. +- Active Source: In redundant deployment scenarios the API stores and shares information on which engine instance is currently active. This could be extended to other audio sources. +- Health Information: In case of some critical issue affecting the functionality of AURA Engine, the history of health status records of the respective engine is stored. You can find details on the available API endpoints here: https://app.swaggerhub.com/apis/AURA-Engine/engine-api/1.0.0 ## Deployment Modes -AURA Engine allows single and redundant deployments. +AURA Engine allows single and redundant deployments for high availability scenarios. ### Single Deployment This is the most simple case. In that scenario the Engine API is deployed on the same host as the [Engine](https://gitlab.servus.at/aura/engine) itself. -> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like a reverse proxy. +> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX, +acting as a reverse proxy. <img src="docs/engine-api_single.png" width="500" /> ### Redundant Deployment -In this scenario there are two Engine instances. Here you will need deploy one Engine API on the host of each Engine instance. -Additionally you'll have to deploy a third, so-called "Syncronization Node" of the Engine API. The sync instanc of Engine API -is in charge of synchronizing playlogs, health information etc. +In this scenario there are two Engine instances involved. Here you will need to deploy one Engine API on the host of each Engine instance. Additionally you'll have to set up +a third, so-called "Syncronization Node" of the Engine API. This sync instance of Engine API is in charge of synchronizing playlogs and managing the active engine state. <img src="docs/engine-api_redundancy.png" width="800" /> +#### Managing Active Engine State + +In order to avoid duplicate playlog storage, the Synchronization Node requires to know what the currently active Engine is. This can be achieved by some external *Status Monitor* +component which tracks the heartbeat of both engines. In case the Status Monitor identifies one Engine as dysfunctional, it sends a REST request to the Sync Node, informing it +about the second, functional Engine instance being activated. + +The history of active Engine instances is stored in the database of the Sync Node. It is not only used for playlog syncing, but is also handy as an audit log. + +> At the moment AURA doesn't provide its own *Status Monitor* solution. You'll need to integrate your self knitted component which tracks the heartbeat of the engines and posts +the active engine state to the Sync Node. + +#### High Availability Playlog Synchronization + +Usually when some new audio source starts playing, AURA Engine logs it to its local Engine API instance via some REST call. Now, the local API server stores this information in its +local database. Next, it also performs a request to the Synchronization API Serve. The Sync Server checks if this request is coming from the currently active engine instance. +If yes, it stores this information in the playlog database. + +During the synchronization process between some Engine Node and the Synchronization Node, there might be the case, that the latter is not available e.g. due to network outage, +maintenance etc. In this situation the playlog obviously can not be synced. That means the local playlog is marked as "not synced". Whenever the Sync Node is up- and running again, +some automated job on the Sync Node is continuously checking for "unsynced" records on remote nodes. If there are such records pending to be synced, this job reads them as batches +from that Engine Node. To avoid this sync causing high traffic on any engine instance, these batches are read with some configured delay time (see `sync_interval`, `sync_batch_size`, +and `sync_step_sleep` in the Sync Node configuration; all values are in seconds). + +> In your live deployment you might not want to expose the API directly on the web. For security reasons it's highly recommended to guard it using something like NGINX, +acting as a reverse proxy. + ## Getting started ### Requirements @@ -63,7 +89,7 @@ sudo mysql -u root -p < contrib/mariadb-database.sql ### Configuration -Copy the the sample configuration file in `./config/sample/sample-production.engine-api` to `config` and edit the file. +Copy the sample configuration file in `./config/sample/sample-production.engine-api` to `config` and edit the file. First update the main configuration and then configure the type of federation. Depending on how you want to run your Engine API node and where it is deployed, you'll needed to uncomment one of these federation sections: @@ -73,41 +99,46 @@ Engine API node and where it is deployed, you'll needed to uncomment one of thes Use this section if you are running [AURA Engine](https://gitlab.servus.at/aura/engine) standalone or if this is the first API node in a redundant deployment. -Replace `localhost` and port number with the actual details of your sync node. +Replace `api.sync.local` with the actual host name or IP of your sync node. ```ini # NODE 1 host_id=1 -sync_host="http://localhost:8010" +sync_host="http://api.sync.local:8008" ``` #### Engine 2 Node In case this is the second API node in a redundant deployment. -Replace `localhost` and port number with the actual details of your sync node. +Replace `api.sync.local` with the actual host name or IP of your sync node. ```ini # NODE 2 host_id=2 -sync_host="http://localhost:8010" +sync_host="http://api.sync.local:8008" ``` #### Synchronization Node -This is the synchronization instance in a redundant setup. This instance combines all valid information from coming from Engine API 1 and 2. +This is the synchronization instance in a redundant setup. This instance combines all valid information coming from Engine API 1 and 2. -Replace `localhost` and port number with the actual details of your main nodes. +Replace `engine1.local` and `engine2.local` with the actual details of your main nodes. ```ini # NODE SYNC host_id=0 -main_host_1="http://localhost:8008" -main_host_2="http://localhost:8009" +main_host_1="http://engine1.local:8008" +main_host_2="http://engine2.local:8008" + +# The Engine which is seen as "active" as long no other information is received from the status monitor default_source=1 +# How often the Engine 1 and 2 nodes should be checked for unsynced records (in seconds) sync_interval=3600 +# How many unsynced records should be retrieved at once (in seconds) sync_batch_size=100 -sync_step_sleep=0.23 +# How long to wait until the next batch is requested (in seconds) +sync_step_sleep=2 ``` ## Running the Server @@ -192,17 +223,14 @@ Your OpenAPI definition lives here: http://localhost:8008/api/v1/openapi.json ``` -## Testing - -To launch the integration tests, use tox: +# About -```bash -sudo pip install tox -./run.sh test -``` +This project is based on a swagger-enabled Flask server using an *API First* approach. It also uses the [Connexion](https://github.com/zalando/connexion) library on top of Flask. -> Note, the test-cases are currently not functional. +<img src="https://gitlab.servus.at/autoradio/meta/-/raw/master/images/aura-logo.png" width="150" /> -# About +Aura Engine is the play-out engine of the Aura Radio Software Suite. Aura stands for Automated Radio and is a swiss army knife for community radio stations. Beside the Engine it provides Steering (Admin Interface for the radio station), Dashboard (Collaborative scheduling and programme coordination), Tank (Audio uploading, pre-processing and delivery). Read more in the [Aura Meta](https://gitlab.servus.at/autoradio/meta) repository or on the specific project pages. -This project is based on a swagger-enabled Flask server using an *API First* approach. It also uses the [Connexion](https://github.com/zalando/connexion) library on top of Flask. +| [<img src="https://gitlab.servus.at/autoradio/meta/-/raw/master/images/aura-steering.png" width="150" align="left" />](https://gitlab.servus.at/autoradio/pv) | [<img src="https://gitlab.servus.at/autoradio/meta/-/raw/master/images/aura-dashboard.png" width="150" align="left" />](https://gitlab.servus.at/autoradio/dashboard) | [<img src="https://gitlab.servus.at/autoradio/meta/-/raw/master/images/aura-tank.png" width="150" align="left" />](https://gitlab.servus.at/autoradio/tank) | [<img src="https://gitlab.servus.at/autoradio/meta/-/raw/master/images/aura-engine.png" width="150" align="left" />](https://gitlab.servus.at/autoradio/engine) | +|---|---|---|---| +| [Steering](https://gitlab.servus.at/autoradio/pv) | [Dashboard](https://gitlab.servus.at/autoradio/dashboard) | [Tank](https://gitlab.servus.at/autoradio/tank) | [Engine](https://gitlab.servus.at/autoradio/engine)<br/>[Engine](https://gitlab.servus.at/autoradio/engine-api) | \ No newline at end of file -- GitLab