New Video & TV Api

[keatontaylor]

https://developer.amazon.com/blogs/alexa/post/1314314b-9b98-4501-9032-f72c000a59b5/how-to-control-your-content-with-the-video-skill-api

Something to consider.

[trisk]

Awesome, I've been waiting for this!

[auchter]

Yeah, this is awesome.

I started playing around with it a bit last night, but wasn't very successful. I created a new Skill that implements the Video Skill API, and pointed it towards my existing haaska lambda. I was able to get Alexa to discover the media players via the Video Skill Discovery call, but after that I run into the same issues described here.

Seems like maybe an incomplete roll out on Amazon's part?

[keatontaylor]

I am still confused by how this is all supposed to interconnect. It looks like discovered video player devices should just act similar to any smart home devices, with the exception for the ingest of media content by content providers. Which I suspect will just have a UID passed along to the players themselves. With the video skills api and modifications you made to the discovery for haaska were you able to at least control playback via voice commands?

[keatontaylor]

Also @auchter could you submit your current code to a new branch so I could play around with it?

[keatontaylor]

@auchter apparently you need to add an icon and fill in all the publishing information (make sure publishing information green checkmark is checked) to have this properly show up under manage tv & video.

[keatontaylor]

Been playing around with this a bit more and here are my observations.

The documentation says that gracenote content ids will be provided, which would be perfect for auto-playing content on a roku device with the rest api they provide, but the request body payload does not contain the gracenote IDs.

I suppose that is fine though, since the title, season number and episode number are provided which could be good to link to the roku search function for voice based searching and playback control.

It seems the roku might actually be the most versatile media player to use this with.

I got playback controls setup, but they're not working, almost like alexa doesn't realize that the device I am attempting to query is a media player. Oh well, more playing around tomorrow...

[keatontaylor]

https://github.com/keatontaylor/haaska/tree/add-video-skills-api

WIP code, not conforming to the current class structure of haaska so I didn't make a pull request yet.

[auchter]

Here's what I've been playing around with: https://github.com/auchter/haaska/tree/video-skill

[keatontaylor]

As far as I can tell, either SearchAndPlay or SearchAndDisplayResults is required to place the echo into a state so that it will "forward" control commands to the lambda instance.

So something like: "Alexa, show me Star Trek: Voyager" will cause the linked echos to forward all Alexa, play; Alexa, pause; etc. commands to the haaska lambda.

This works well for the roku because the External Control API provided by roku can be used to search for content using the search keywords as parameters from alexa, but for other devices like the chromecast and fire tv there doesn't seem to be a way (yet).

So, that means for those media players those users would need to invoke a fake SearchAndPlay command such as: "Alexa play Game of Thrones" so that all commands for a few hours afterwards will be forwarded.

I hope that makes sense.

[auchter]

Yeah, I noticed this as well. "Alexa, change channel to 200" also seems to work to enable the forwarding, too.
I hope that the interaction model is improved to eliminate the need for this, but it's not too inconvenient for me yet

[keatontaylor]

So do the volume controls still work? I notice you're not excluding the media players from the standard Connected Home discovery.

[keatontaylor]

I also seem to be running into an issue where the RemoteVideoPlayer invocation isn't passing the cookie data set by discovery.

cookie is just blank causing an error, but works fine for the ChannelController

[auchter]

Yep, same exact problem here, which I assume is a bug on Amazon's end.

[Redferne]

Can't wait to try this with Kodi. Great work! Thank you!

[keatontaylor]

Been playing around with this more, adding in functionality for the roku search API and man does it work well. I've made a pull request for the python-roku library that adds the search support here: brentahughes/python-roku#10

and here for HA: keatontaylor/home-assistant@33e1575

One of the things I like most is you can choose one or more providers and it will choose the first one for the content that is found. So Game of Thrones will yield HBO, while House of Cards will yield Netflix if the providers are in the order "HBO GO, Amazon Video, Netflix"

[keatontaylor]

Something to consider is how supported features will need to be implemented. Roku doesn't support the play or pause service, just the play_pause service. So that will have to be accounted for. This is simple enough by getting the supported_features integer attribute and determining how to handle media features based on those supported features.

[auchter]

Roku doesn't support the play or pause service, just the play_pause service.

Ah, that's unfortunate, and from a quick glance at the roku component and python library it doesn't look like there's a good way to implement play or pause, either. I mean it makes sense from an interactive point of view, but state-dependent commands just kinda suck for automation purposes (especially when there's no way to query the state).

This is simple enough by getting the supported_features integer attribute and determining how to handle media features based on those supported features.

Sounds reasonable to me.

[auchter]

Been playing around with this more, adding in functionality for the roku search API and man does it work well.

That looks awesome. I might have to go pick up a Roku to play around with this...

and here for HA: keatontaylor/home-assistant@33e1575

This ties in with your point above with handling device-specific functionality. Having a service that's only implemented on certain types of media players (as implied by the name roku_search) is going to get a bit tricky to support.

What are your thoughts about proposing some search functionality (likely loosely based upon the parameters provided by the Video Skills API) as a media_player-level service in HA?

[keatontaylor]

Actually, thinking about it now I do like the idea of a more abstract media_player level service for doing search. haaska can provide the media content such as title, season, episode and HA can essentially implement a media_player level search.

Thinking ahead to how it would work with roku, I could change the HA roku component to accept config parameters for things like provider array and launch boolean.

However, of all the media player devices on HA, I am not sure which ones actually have any formal API for searching content besides the roku. So abstraction might not be necessary right now.

[auchter]

However, of all the media player devices on HA, I am not sure which ones actually have any formal API for searching content besides the roku. So abstraction might not be necessary right now.

Plex should be able to support this without issue. I'll take a look into this and see what sort of commonalities it has with your Roku implementation, and maybe we can converge on a reasonable service API to satisfy them both.

[keatontaylor]

Well I ran into an annoying issue, was playing around with the alexa video skill in the alexa developer console and now regardless of what I try and do, the video skill will not work. It says it is not enabled when clearly it is. I've tried everything.

[trisk]

I'm running your video-skill branch in AWS and have a video skill pointed at the existing Lambda function. Account linking stopped working last night (it would return a 400) but started working again today, so I have the skill enabled again.

Discovery seems to be working as I have two devices shown for all media_player entities. The smart home commands still work for those devices, but I can't figure out how to get the video playback controller commands to do anything ("Alexa, pause on TV" or "Alexa, pause TV" is just interpreted as a music playback command).

Should the smart home skill be disabled or media_player entities hidden from discovery there?

[auchter]

but I can't figure out how to get the video playback controller commands to do anything ("Alexa, pause on TV" or "Alexa, pause TV" is just interpreted as a music playback command).

If you try "Alexa, change channel to 200" or similar, does "Alexa, pause" start working?

[keatontaylor]

I find it remarkable just how well the Video Skill API complements the Roku ECP. I have mapped the change channel command to channels on the Roku. It does mean the user has to "learn" what numbers correspond to what app since I am just grabbing the app source list from HA and using each item in the list as a number starting at 1 and going to how many items are in the source list. This is like any other provider though, since you have to learn what channel numbers correspond to what.

The downside is HA alphabetizes the source list so numbers will potentially change when new apps are added. I could map the channel number to the IDs for the channels provided by roku, but some of those IDs are really high values. Such as DirecTV Now with ID 140474.

Still it does seem to work well if we can find a way to map the channels to numbers effectively.

[trisk]

I tried "Alexa, change channel to 100", but I'm getting "I couldn't find any enabled video skills" like @keatontaylor even though the skill is linked and marked enabled 😢

[keatontaylor]

I banged my head against a wall on this one and eventually set up a new lambda secondary to haaska that made it all work. Not sure why.

[auchter]

@keatontaylor Took a first pass at implementing search for Kodi, based off your Roku implementation. Right now it only supports videos, but it seems to work pretty well.

https://github.com/auchter/home-assistant/pull/1
and
https://github.com/auchter/haaska/tree/video-skill

(haaska right now has the entity_id for SearchAndPlay hardcoded to media_player.kodi, since the cookie in the payload is empty... I should really bring that up on Amazon's support forums.)

[keatontaylor]

My solution for the cookie issue was to generate the endpointID by converting the entity_id into a base64 string. Provides uniqueness since HA cannot have duplicate entity IDs so that seems to work well.

[keatontaylor]

Gracenote TMSIDs are now being included in the SearchAndPlay and SearchAndDisplayResults queries. I cannot seem to match them up correctly for more refined searches on the Roku, but I'll keep working. I'd also be curious if any other platform could take advantage of this.

[keatontaylor]

https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/smart-home-skill-api-reference#entertainment-control-messages

:)

[keatontaylor]

Man, looking at this code, it looks like a total mess. Part of the skill will be associated with the smart home skill api, while another part will be part of the video skill api.

[maddox]

Adopting media player control would be a good first iteration. I think we've all wanted media player controls from Alexa for a long time, falling back to hacks like "dim" for volume, etc.