[STORY] As a host, I want CBA Media URLs automatically resolve to the actual media file, when pasting into the import dialog

changed milestone to %1.2

added STORY label

This ticket originated from the report #174 (closed)

marked this issue as related to #174 (closed)

changed the description

Unorthodox idea: Let’s write a CBA plugin for yt-dlp. If tank requests the URL and it doesn’t directly resolve to an audio data stream, call yt-dlp --extract-audio "$URL" instead and see if that yields a valid audio stream.

That's an interesting and creative idea :) I'm not sure if it's worth the effort though, because the CBA API call is pretty simple actually.

All you need, is a fetch on https://cba.fro.at/wp-json/wp/v2/media pass the media ID and API key. And voilá, you have the media file. Since we need to store the CBA API Key anyway, in order to build the CBA Uploader, I think that's pretty straighforward.

And we can even do this client-site in Dashboard, when pasting the URL into the <input>. So we can leave any additional logic out of Tank.

@fm_margarethem As soon we have a place to store the CBA API Key in the Radio Station Settings, this story can be solved with relatively small effort. Do you think we should schedule this for one of the coming releases already?

in Rivendell at RH there is a URL resolver in place which does not only address CBA links, possibly @equinox can give more insights on its logic. posting a screenshot of the URL resolver UI here for reference.

At Radio Helsinki the import daemon uses youtube-dl to extract media files from all sites supported by youtube-dl. CBA is already kind-of supported by youtube-dl, as well as yt-dlp, because the generic extractor just looks for HTML-5 <audio> Tags and usually finds the correct files. However there is twist to that: CBA only provides files with reduced quality for copyrighted material. In order to get the file in full quality the CBA team gave us an API Key. This key works with the normal API as well as the RSS Feed. Back in the day i tried to get an info-extractor merged into youtube-dl that can leverage those keys but it was never merged. I doubt that yt-dlp will handle this request differently.

I never added the lookup to tank for two reasons:

At least back when i started to use youtube-dl in this way there was a new release every 2-3 days. I think development has slowed down since then but it will most likely still be quite a lot of work to keep it up to date. Especially if we need to maintain the patch to enable usage of the API Keys ourselves.
I don't think the import daemon is the right place for it. Occasionally we had the problem that the info extractor from youtube-dl yields the wrong file - especially if there are multiple files available. If the lookup is done inside of tank the user cannot really interact with that. A much nicer user experience would be if the Dashboard would do the lookup (potentially using a backend service) and in case more then one url is found lets the user select the correct one.

All in all i still think such a feature should be added to AURA. I would also prefer a solution that works with more sites than just CBA. For example the idea to add youtube-dl lookups to the Radio Helsinki import daemon came up because we wanted to support pasting links from Dropbox or WeTransfer.

And we can even do this client-site in Dashboard, when pasting the URL into the <input>. So we can leave any additional logic out of Tank.

As already said i would also like this feature be implemented in the Dashboard but we cannot do this completely client side because we would leak the CBA API key to all clients.

Many thanks for the detailed background info.

I absolutely agree to all of your points. Regarding youtube-dl, I've raised similar worries with multiple audio sources on one page.

We will providing a CBA Media URL resolution in alpha-3 already. It should be in Dashboard, so no Tank changes are involved. I am in favor of additional services too, but see them out of scope for the MVP. So more like post 1.0.

Do you think we (esp. Helsinki) need that sooner, @equinox and @fm_margarethem?

As already said i would also like this feature be implemented in the Dashboard but we cannot do this completely client side because we would leak the CBA API key to all clients.

I was in touch with CBA previously, asking about using the CBA key client-site. I got the okay that the API key can be exposed in the browser, confirming that other radios do the same.

Do you think we (esp. Helsinki) need that sooner, @equinox and @fm_margarethem?

No i don't think so. The version of youtube-dl we are currently using with the Radio Helsinki import daemon hasn't been updated for quite some time now. This means that the extraction of the links will only work with sites that haven't changed at all in the last 3-4 years. I think CBA is one of the few sites where this is the case. Long story short i don't think the feature is used for any other site than the CBA and if Aura has support for it we are good.

I was in touch with CBA previously, asking about using the CBA key client-site. I got the okay that the API key can be exposed in the browser, confirming that other radios do the same.

I think there is a misunderstanding here. The API key i am talking about can be used to access full quality versions of all audio files uploaded to CBA. This is a one-per-radiostation key that we must not leak to our users. I have the feeling you are talking about a personal API key that links a CBA user to the user who is logged into the Aura dashboard? Is this correct? If so i have questions:

What exactly can be accessed using this personalized keys? Only files that belong to the CBA user behind the API key? All the files?
If the dashboard is using the API key to extract the link to the file in full quality - can the tank import this URL without the need of the API key?

Nope, I'm talking about the radio-specific key. I also got a demo aura demo key for AURA. It allows accessing metadata of all resources.

CBA uses the API key mainly for statistics and individual radio-branding. CBA can still secure its usage using CORS. See the ticket description, which holds an example path which I got with the AURA demo key.

We at least should advise radio stations, not using the demo key, but request their individual radio-station key.

But I agree, it's probably not the most ideal implementation to keep keys in the browser, even when they are obscured in some way. At our current stage I would not worry too much, since we have to push things forward. At a later stage, I'd get in touch with CBA, suggesting them some security improvements, where we can collaborate on.

The confusion grows. What has CORS to do with any of this?

What keeps the user from pressing F12 and just reading the key?

Nothing keeps from reading the key, as I wrote above. We got the okay to expose it.

With CORS you can at least limit the usage of one key linked to a domain. Obviously this doesn't help, once you use the key server-side. And as I said, imo it doesn't hurt anyone at the current stage.

No you can't. CORS punches holes in the same-oigin-policy of the browser. So if anything they extend to usage of the key. And this is only even true if you are using a browser. They do jack shit if i copy the key from the browser debugger and use curl to send the request.

You might be talking about CSP? This would slightly increase security but again CSP is enforced by the browser and doesn't do anything if you use another client.

That's what I said, this is no real security. More security-by-obscurity. It only works when you are stay in the browser:

Obviously this doesn't help, once you use the key server-side.

You seem not to accept the point the we got the OK to do so? I suggest we move the discussion over to Matrix for stand-point arguing :-D

To be brutally honest given the discussion so far i'm not so sure whether they have fully grasped what is proposed here. I wouldn't even call this security-by-obscurity there is no obscurity and also no security what so ever. But sure we can move the discussion to Matrix.

Let me propose another way that i just thought of while typing my message in matrix:

Your suggested solution means that the browser somehow needs to know the API key to use for the request to CBA. Imho there are 2 possible ways to do this:

The key is somewhere hardcoded in the Javascript of the Dashboard
The client-side dashboard requests the API key from the server.

Option 1 means that not just authenticated users can see the key but everybody who can read the source code of the dashboard. For option 2 to be better then option 1 the call to the server needs to be authenticated. Otherwise anybody who can read the source code of the dashboard knows where to send the request to get the API key.

So i propose the following: If we already have to have an authenticated API endpoint for the retrieval of the API key why not change this endpoint to do the call to CBA server-side and pass the result to the requesting client. This way the CBA API key can stay on the server.

If time-crunch is the concern here i could implement such an endpoint in tank by the end of this week. Although i am not sure if tank is the right place for it - especially in the long run.

see: steering#165 (comment 11029)

marked this issue as related to #163 (moved)

mentioned in issue #163 (moved)

changed milestone to %1.0-alpha3 — Playful Platypus

added discuss label

changed the description

assigned to @kmohrf

added 1 deleted label

added To Do label

removed discuss label

mentioned in issue aura#221 (closed)

added 1 deleted label

added Doing label and removed To Do label

Re-adding the discuss label because the discussion around the CBA API key is not resolved.

I must agree with @equinox here. I don’t understand why we expose the API key to the dashboard. Or in other words: I don’t understand why the CBA makes the effort to secure its API with API keys and then allows us to expose these keys world-readable.

I also don’t understand the reason why this is supposed to be solved in the dashboard. Why go to the trouble of thinking about how we secure this key later on in the dashboard if we can just keep it from being exposed in the first place by implementing the resolver in tank or steering?

So… in terms of moving forward: Let’s implement the resolver in steering (or better: let tank do it automatically – do we have someone apart from @equinox who’s good with Go?) and let the dashboard use that resolver instead. Deal?

Previous version of this comment

Storing the API Key via the radio settings requires a schema first. This is a general problem with the radio settings implementation as outlined in aura#221 (closed) and steering#173 (closed). The schema should be defined in steering, but we haven’t done that on a technical level.
We haven’t defined permissions for the radio settings API so these keys would be world readable. I think that is a problem.

Personally I’d prefer a solution where tank does correct import itself or at least provides a mechanism for resolution.

removed Doing label

added discuss label

let's discuss how to proceed with this together with @chrizz (not exactly GO but C# expert who can maybe help) and @eigenwijsje (and maybe @david has some more insights, too?) in the upcoming sprint planning meeting!

changed milestone to %1.0-alpha5 — Capable Capybara

moving milestone because it still needs discussion.

added requires-refinement label

mentioned in issue steering#216

changed the description

marked this issue as related to steering#216

added size:s label

changed milestone to %1.0-alpha6 — Wondrous Whale

changed the description

mentioned in issue #150 (moved)

mentioned in issue aura#390 (closed)

changed the description

changed milestone to %1.0-alpha7 — Helpful Hedgehog

changed the description

added moveable label

mentioned in issue aura#440

changed milestone to %1.0-alpha8

changed the description

[STORY] As a host, I want CBA Media URLs automatically resolve to the actual media file, when pasting into the import dialog

Parent: [Epic] CBA Media URL Resolver (aura#440)

CBA API Request

Dependencies

Designs

Child items ...

Activity