Skip to content

Commit

Permalink
Share SLU Stream Between Contexts (#4)
Browse files Browse the repository at this point in the history 
* Refactor SluClient to keep connected between contexts

* Refactor UI to match new naming

* Update README.md

* Streamline button delegate

* Generate docs, fix links
  • Loading branch information
@langma
langma authored Apr 9, 2021
1 parent 65a8e21 commit 27b554f
Show file tree
Hide file tree
Showing 41 changed files with 674 additions and 561 deletions.
1 change: 1 addition & 0 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ clean:

$(DOCSPATH): $(SOURCES)
$(SWIFT) doc generate ./Sources/ --module-name Speechly --output $(DOCSPATH) --base-url ""
@sed -i.bak -E 's/(\[.+\])\((.+)\)/\1(\2.md)/g' docs/*.md && rm docs/*.md.bak

$(RELEASEBUILD): $(SOURCES) Package.swift
$(XCODE) archive $(BUILDFLAGS) -archivePath "$(ARCHPATH)/release" -configuration Release
Expand Down
120 changes: 66 additions & 54 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Complete your touch user interface with voice

This repository contains the source code for the iOS client for [Speechly](https://www.speechly.com/?utm_source=github&utm_medium=ios-client&utm_campaign=text) SLU API. Speechly allows you to easily build applications with voice-enabled UIs.

## Usage
## Installation

### Swift package dependency

Expand Down Expand Up @@ -47,19 +47,21 @@ And then running `swift package resolve`.

If you are using Xcode, check out the [official tutorial for adding package dependencies to your app](https://developer.apple.com/documentation/xcode/adding_package_dependencies_to_your_app).

### Client usage
## Client Usage

The client exposes methods for starting and stopping the recognition, as well as a delegate protocol to implement for receiving recognition results:
The client exposes methods for starting and stopping the recognition, as well as a delegate protocol to implement for receiving recognition results. The `startContext()` method will open the microphone device and stream audio to the API, and the `stopContext()` method will close the audio context and the microphone.

Note: the application's `Info.plist` needs to include key `NSMicrophoneUsageDescription` to actually enable microphone access. The value is a string that iOS presents to the user when requesting permission to access the microphone.

```swift
import Foundation
import Speechly

class SpeechlyManager {
let client: SpeechClient
let client: Speechly.Client

public init() {
client = try! SpeechClient(
client = try! Speechly.Client(
// Specify your Speechly application's identifier here.
appId: UUID(uuidString: "your-speechly-app-id")!,
// or, if you want to use the project based login, set projectId.
Expand All @@ -72,30 +74,32 @@ class SpeechlyManager {
public func start() {
// Use this to unmute the microphone and start recognising user's voice input.
// You can call this when e.g. a button is pressed.
client.start()
// startContext accepts an optional `appId` parameter, if you need to specify it
// per context.
client.startContext()
}

public func stop() {
// Use this to mute the microphone and stop recognising user's voice input.
// You can call this when e.g. a button is depressed.
client.stop()
client.stopContext()
}
}

// Implement the `Speechly.SpeechClientDelegate` for reacting to recognition results.
extension SpeechlyManager: SpeechClientDelegate {
// Implement the `Speechly.SpeechlyDelegate` for reacting to recognition results.
extension SpeechlyManager: SpeechlyDelegate {
// (Optional) Use this method for telling the user that recognition has started.
func speechlyClientDidStart(_: SpeechClientProtocol) {
func speechlyClientDidStartContext(_: SpeechlyProtocol) {
print("Speechly client has started an audio stream!")
}

// (Optional) Use this method for telling the user that recognition has finished.
func speechlyClientDidStop(_: SpeechClientProtocol) {
func speechlyClientDidStopContext(_: SpeechlyProtocol) {
print("Speechly client has finished an audio stream!")
}

// Use this method for receiving recognition results.
func speechlyClientDidUpdateSegment(_ client: SpeechClientProtocol, segment: SpeechSegment) {
func speechlyClientDidUpdateSegment(_ client: SpeechlyProtocol, segment: Segment) {
print("Received a new recognition result from Speechly!")

// What the user wants the app to do, (e.g. "book" a hotel).
Expand All @@ -111,87 +115,95 @@ extension SpeechlyManager: SpeechClientDelegate {
}
```

Check out the [ios-repo-filtering](https://github.com/speechly/ios-repo-filtering) repository for a demo app built using this client.

### User interface components
## User Interface Components

The client library also includes a couple of ready-made UI components which can be used together with `SpeechClient`.
The client library also includes a couple of ready-made UI components which can be used together with `Speechly.Client`.

`SpeechButton` starts and stops voice recognition automatically when pressed and released, using build-in icons and visual effects which you can replace with your own if needed.
`MicrophoneButtonView` presents a microphone button using build-in icons and visual effects which you can replace with your own if needed. The microphone button protocol can be forwarded to `Speechly.Client` instance easily.

`SpeechTranscriptView` visualizes the transcripts received in the `speechlyClientDidUpdateSegment` callback, automatically highlighting recognized entities.
`TranscriptView` visualizes the transcripts received in the `speechlyClientDidUpdateSegment` callback, automatically highlighting recognized entities. For other callbacks, see [the protocol docs](docs/SpeechlyProtocol.md).

These can be used, for example, in the following way:
These can be used, for example, in the following way (`UIKit`):

```swift
import UIKit
import SnapKit
import Speechly

class ViewController: UIViewController, SpeechClientDelegate, SpeechButtonDelegate {

private let client: SpeechClient

private lazy var speechButton = SpeechButton(delegate: self)

private let transcriptView = SpeechTranscriptView()

init() {
client = try! SpeechClient(
appId: UUID(uuidString: "your-speechly-app-id")!
)

client.delegate = self
}

class ViewController: UIViewController {
private let manager = SpeechlyManager()

override func viewDidLoad() {
super.viewDidLoad()

view.backgroundColor = UIColor.white

manager.addViews(view: view)
}
}

class SpeechlyManager {
private let client: Speechly.Client
private let transcriptView = TranscriptView()

private lazy var speechButton = MicrophoneButtonView(delegate: self)

public init() {
client = try! Speechly.Client(
appId: UUID(uuidString: "your-speechly-app-id")!
)
client.delegate = self
speechButton.holdToTalkText = "Hold to talk"
speechButton.pressedScale = 1.5

transcriptView.autohideInterval = 3

}

public func addViews(view: UIView) {
view.addSubview(transcriptView)
view.addSubview(speechButton)

transcriptView.snp.makeConstraints { (make) in
make.top.left.equalTo(view.safeAreaLayoutGuide).inset(20)
make.right.lessThanOrEqualTo(view.safeAreaLayoutGuide).inset(20)
}

speechButton.snp.makeConstraints { (make) in
make.centerX.equalToSuperview()
make.bottom.equalTo(view.safeAreaLayoutGuide).inset(20)
}
}
}

extension ViewController: SpeechClientDelegate {

func speechlyClientDidUpdateSegment(_ speechlyClient: SpeechClientProtocol, segment: SpeechSegment) {
DispatchQueue.main.async {
self.transcriptView.configure(segment: segment, animated: true)
}
public func start() {
client.startContext()
}

public func stop() {
client.stopContext()
}
}

extension ViewController: SpeechButtonDelegate {

func clientForSpeechButton(_ button: SpeechButton) -> SpeechClient? {
return client
extension SpeechlyManager: MicrophoneButtonDelegate {
func didOpenMicrophone(_ button: MicrophoneButtonView) {
self.start()
}
func didCloseMicrophone(_ button: MicrophoneButtonView) {
self.stop()
}
}

extension SpeechlyManager: SpeechlyDelegate {
func speechlyClientDidUpdateSegment(_ client: SpeechlyProtocol, segment: Segment) {
DispatchQueue.main.async {
self.transcriptView.configure(segment: segment, animated: true)
}
}
}
```

For a `SwiftUI` example, check out the [ios-repo-filtering](https://github.com/speechly/ios-repo-filtering) demo app.

## Documentation

Check out [official Speechly documentation](https://docs.speechly.com/client-libraries/ios/) for tutorials and guides on how to use this client.

You can also find the [API documentation in the repo](docs/Home.md).
You can also find the [speechly-ios-client documentation in the repo](docs/Home.md).

## Contributing

Expand Down
38 changes: 19 additions & 19 deletions Sources/Speechly/Speech/SpeechContext.swift → Sources/Speechly/AudioContext.swift
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@ import Foundation
///
/// A single context aggregates messages from SLU API, which correspond to the audio portion
/// sent to the API within a single recognition stream.
public struct SpeechContext: Hashable, Identifiable {
private var _segments: [SpeechSegment] = []
private var _indexedSegments: [Int:SpeechSegment] = [:]
public struct AudioContext: Hashable, Identifiable {
private var _segments: [Segment] = []
private var _indexedSegments: [Int:Segment] = [:]
private var _segmentsAreDirty: Bool = false

/// The ID of the segment, assigned by the API.
public let id: String

/// The segments belonging to the segment, can be empty if there was nothing recognised from the audio.
public var segments: [SpeechSegment] {
public var segments: [Segment] {
mutating get {
if self._segmentsAreDirty {
self._segments = Array(self._indexedSegments.values).sorted()
Expand All @@ -27,7 +27,7 @@ public struct SpeechContext: Hashable, Identifiable {

set(newValue) {
self._segments = newValue.sorted()
self._indexedSegments = newValue.reduce(into: [Int:SpeechSegment]()) { (acc, segment) in
self._indexedSegments = newValue.reduce(into: [Int:Segment]()) { (acc, segment) in
acc[segment.segmentId] = segment
}
}
Expand All @@ -49,60 +49,60 @@ public struct SpeechContext: Hashable, Identifiable {
/// - Important: this initialiser does not check whether `segments` have `id` set as their `contextId` values,
/// so it is possible to pass segments to this initialiser that don't belong to this context
/// according to the identifiers.
public init(id: String, segments: [SpeechSegment]) {
public init(id: String, segments: [Segment]) {
self.init(id: id)
self.segments = segments
}
}

// MARK: - Comparable protocol conformance.

extension SpeechContext: Comparable {
public static func < (lhs: SpeechContext, rhs: SpeechContext) -> Bool {
extension AudioContext: Comparable {
public static func < (lhs: AudioContext, rhs: AudioContext) -> Bool {
return lhs.id < rhs.id
}

public static func <= (lhs: SpeechContext, rhs: SpeechContext) -> Bool {
public static func <= (lhs: AudioContext, rhs: AudioContext) -> Bool {
return lhs.id <= rhs.id
}

public static func >= (lhs: SpeechContext, rhs: SpeechContext) -> Bool {
public static func >= (lhs: AudioContext, rhs: AudioContext) -> Bool {
return lhs.id >= rhs.id
}

public static func > (lhs: SpeechContext, rhs: SpeechContext) -> Bool {
public static func > (lhs: AudioContext, rhs: AudioContext) -> Bool {
return lhs.id > rhs.id
}
}

// MARK: - Parsing logic implementation.

extension SpeechContext {
mutating func addTranscripts(_ value: [SpeechTranscript], segmentId: Int) throws -> SpeechSegment {
extension AudioContext {
mutating func addTranscripts(_ value: [Transcript], segmentId: Int) throws -> Segment {
return try self.updateSegment(id: segmentId, transform: { segment in
for t in value {
try segment.addTranscript(t)
}
})
}

mutating func addEntities(_ value: [SpeechEntity], segmentId: Int) throws -> SpeechSegment {
mutating func addEntities(_ value: [Entity], segmentId: Int) throws -> Segment {
return try self.updateSegment(id: segmentId, transform: { segment in
for e in value {
try segment.addEntity(e)
}
})
}

mutating func addIntent(_ value: SpeechIntent, segmentId: Int) throws -> SpeechSegment {
mutating func addIntent(_ value: Intent, segmentId: Int) throws -> Segment {
return try self.updateSegment(id: segmentId, transform: { segment in try segment.setIntent(value) })
}

mutating func finaliseSegment(segmentId: Int) throws -> SpeechSegment {
mutating func finaliseSegment(segmentId: Int) throws -> Segment {
return try self.updateSegment(id: segmentId, transform: { segment in try segment.finalise() })
}

mutating func finalise() throws -> SpeechContext {
mutating func finalise() throws -> AudioContext {
for (k, v) in self._indexedSegments {
if !v.isFinal {
self._indexedSegments.removeValue(forKey: k)
Expand All @@ -114,8 +114,8 @@ extension SpeechContext {
return self
}

private mutating func updateSegment(id: Int, transform: (inout SpeechSegment) throws -> Void) rethrows -> SpeechSegment {
var segment = self._indexedSegments[id] ?? SpeechSegment(segmentId: id, contextId: self.id)
private mutating func updateSegment(id: Int, transform: (inout Segment) throws -> Void) rethrows -> Segment {
var segment = self._indexedSegments[id] ?? Segment(segmentId: id, contextId: self.id)

try transform(&segment)

Expand Down
Loading

0 comments on commit 27b554f

Please sign in to comment.