Support MacBook Touch Bar API

[kevinsawicki]

Tracking issue for adding touch bar API support to Electron on macOS 👉 ⌨️ 💻 🍎

• Chrome issue
• NSTouchBar docs
• Design video
• NSTouchBar catalog
• Touch bar example app

Please feel free to leave comments here about API ideas, usage patterns, etc.

[jakubzitny]

It should probably have a similar API as menu has

• importing TouchBar from electron in main process and from remote in renderer
• creating a custom instance with new TouchBar()
• setting it with static method TouchBar.setApplicationTouchBar(touchBar)

The custom instance should have setters for customizationIdentifier, defaultItemIdentifiers, customizationAllowedItemIdentifiers and principalItemIdentifier.

I am not sure how to add items from the Catalog. Maybe something like MenuItem's types such as checkbox or separator it should just be enumerated with string ids.

What do you think?

[HFreni]

I think that you should also consider adding a setter for NSTouchBarItemIdentifierOtherItemsProxy to allow for nested TouchBar MenuItems.

In terms of adding items from the Catalog, I think that they should be accessible as objects so that a developer can just instantiate a new MenuItem() that takes in the array of defaultItemIdentifiers

My 2 cents.

[NetOpWibby]

We're just deleting comments now? Cool.

Any idea on how to add a screenshot of a <webview> to the Touch Bar?

[anaisbetts]

@NetOperatorWibby We asked around and I don't think any of the Electron maintainers deleted your comment, if they did then it was accidental. Regardless of how it happened, sorry about the missing message!

[HFreni]

I was just reading the TouchBar documentation some more, and it seems as if macOS does all of the spacing and rendering, which means that there may not be a possibility to display a <webview> container or screenshot within the Touch Bar.

I'm wondering what kind of effort it would require for someone to create a method of displaying webview content within the Touch Bar.
In the event that it's possible, it creates an even larger number of use cases for the Touch Bar, including but not limited to games and notifications, plus it would allow developers to unify their user interface and to develop custom MenuItems for their applications.

[bengotow]

@HFreni I've been reading on Twitter that the TouchBar actually runs WatchOS (which is crazy and super cool.) It seems it's receiving framebuffer data at a low level though, so I imagine we could create a custom TouchBar view and forward it data!

Source: https://twitter.com/stroughtonsmith/status/791872723681239040

[davidhariri]

The main use cases I see are these two types of views:

1. ButtonView(onPress:Function, textContent:String/null, graphicContent:URLString/64Blob/null)
2. SimpleView(textContent:String/null, graphicContent:URLString/64Blob/null)

If we could have some sort of simple interface for alignment/justification within the parent view (similar to FlexBox?) than that would be a great start for most devs I think!

Side note: It would be so cool to see Git status in Atom in the TouchBar...

[rootscript]

@kevinsawicki @bengotow - I found this an interesting read: https://www.raywenderlich.com/147118/use-nstouchbar-macos

Visual design guide: https://developer.apple.com/library/content/documentation/UserExperience/Conceptual/OSXHIGuidelines/VisualDesign.html#//apple_ref/doc/uid/20000957-CH106-SW1

List of system icons: https://developer.apple.com/library/content/documentation/UserExperience/Conceptual/OSXHIGuidelines/Icons.html#//apple_ref/doc/uid/20000957-CH107-SW1

Standard controls: https://developer.apple.com/library/content/documentation/UserExperience/Conceptual/OSXHIGuidelines/ControlsandViews.html#//apple_ref/doc/uid/20000957-CH108-SW1

[mydigitalself]

http://www.iclarified.com/57924/doom-on-the-macbook-pro-touch-bar-video

#justsayin' (and apologies for bringing down the tone)

[bengotow]

Hey folks - not sure if anyone's working on this yet. I've been reading through the macOS docs, and I think implementing this well will be quite a bit of work, but it could probably follow the pattern set by Electron's Menu module. Here's an example of what I think the API could look like, accounting for touch bar popovers and groups, as well as a few must-haves like customization (default true or false) and custom text / button colors.

const bar = TouchBar.buildFromTemplate([
  {
    customizationLabel: 'Skip Track'
    identifier: 'com.myapp.skip-track',
    image: <NativeImage>,
    default: true,
  },
  {
    identifier: TouchBar.ItemIdentifierFixedSpaceLarge,
  },
  {
    customizationLabel: 'Play Controls'
    identifier: 'com.myapp.play-controls',
    type: 'group',
    default: true,
    items: [
      {
        customizationLabel: 'Previous'
        identifier: 'com.myapp.play-controls.prev',
        title: 'Prev',
        labelColor: '#fff',
        bezelColor: 'red',
        click: () => { ... }
      },
      {
        customizationLabel: 'Next'
        identifier: 'com.myapp.play-controls.next',
        title: 'Next',
        click: () => { ... }
      },
    ]
  },
  {
    customizationLabel: 'Advanced'
    identifier: 'com.myapp.advanced',
    type: 'popover',
    default: false,
    items: [
      {
        customizationLabel: 'Advanced - Mute Audio'
        identifier: 'com.myapp.advanced.mute',
        title: 'Mute Audio',
        click: () => { ... }
      },
      {
        customizationLabel: 'Advanced - Apply Crossfade'
        identifier: 'com.myapp.advanced.crossfade',
        title: 'Apply Crossfade',
        click: () => { ... }
      },
    ]
  }
])

myEditorArea.addEventListener('focus', () => {
  TouchBar.setActiveTouchBar(bar);
});
myEditorArea.addEventListener('blur', () => {
  TouchBar.setActiveTouchBar(null);
});

I think the trickiest part is that macOS composes the final touch bar by combining every touch bar in the responder chain, so theoretically if you had something like this (below) where both my-video-editor and my-video-caption declared touch bars, you'd see the combination of both bars when focused in the caption area:

<div id="my-video-editor">
    <textarea id="my-video-caption">
       <!-- user is focused here -->
    </textarea>
</div>

I think it'd be fine to leave this as an exercise to the reader, but the macOS implementation is pretty sophisticated and "whitespace-aware". Doing it right would require more than Object.assign(barA, barB). I think in a first pass, just providing TouchBar.setActiveTouchBar(bar) would be good enough, though?

Also, anyone know if Chromium is planning to support this on the broader web? Would be a bummer to implement it, only to have them come up with a nice general solution.

[MarshallOfSound]

I got my new MacBook Pro 2016 yesterday and because I have nothing better to do 😆 I started looking into this. Initial results are promising but there are a few things that still need to be figured out.

I'm not even close to an API yet but on the JS side I think it will be best to end up with something like.

import { TouchBarButton, TouchBarColorPicker, TouchBarSlider, TouchBarLabel } from 'electron';

mainWindow.setTouchBar([
  new TouchBarButton({ label: 'My Button', onClick: fn }),
  new TouchBarSlider({ onChange: fn });
]);

Now to answer a few things:

Also, anyone know if Chromium is planning to support this on the broader web? Would be a bummer to implement it, only to have them come up with a nice general solution.

Last I checked Chromium wanted to keep the TouchBar a trusted space. I.e. No website content / extension content. So I doubt a web friendly API is coming from that direction any time soon 👍

I think in a first pass, just providing TouchBar.setActiveTouchBar(bar) would be good enough, though?

Read up above as well, but we need to be setting the active touch bar at a window level not an App level. Also I think that providing DOM level TouchBar cascading is out of scope of the first run (as you said) and maybe even something that should be handled on the JS side by apps rather than offered by Electron. There would be many ways users would want it to work so would probably be better off just letting them manipulate an array how they want to.

image: <NativeImage>,

I haven't even tried to passing a NativeImage to a custom item. Could be fun 😆

few must-haves like customization

Customization is gonna be a fun one (and this is also gonna pop a hole in your API idea). Basically to create in order to generate the TouchBar a method is called on your NSTouchBarDelegate called makeItemForIdentifier. You get passed an identifier and have to generate a NSTouchBarItem. This means we can't let the users assign their own identifiers. The workaround I have is assigning each TouchBarItem created on the JS side a unique JS id and then attending that to standard identifiers. So for instance at the moment the button identifier is com.electron.tb.button, when a new button is created it is assigned the identifier com.electron.tb.button.1. This allows us to correctly create the NSTouchBarItem and also use that number from the identifier to fetch the extra settings (color, text, size, Etc.).

It's definitely going to be a long road to get this working nicely though 👍

EDIT: I've gotten to the point where this is working

mainWindow.setTouchBar(new TouchBar([
  new (TouchBar.Button)({
    label: 'Foo bar',
    click: () => {
      console.log('Foo bar clicked')
    }
  })
]))

[davidhariri]

Wow, nice work @MarshallOfSound! Looking forward to playing around with this.

[MarshallOfSound]

Current (working) API

mainWindow.setTouchBar(new TouchBar([
  new (TouchBar.Button)({
    label: 'Hello World!',
    // image: 'path/to/image',
    backgroundColor: "FF0000",
    labelColor: "0000FF",
    click: () => {
      console.log('Hello World Clicked')
    }
  }),
  new (TouchBar.Label)({
    label: 'This is a Label'
  }),
  new (TouchBar.ColorPicker)({
    change: (newColor) => {
      console.log('Color was changed', newColor)
    }
  }),
  new (TouchBar.Slider)({
    label: 'Slider 123',
    minValue: 50,
    maxValue: 1000,
    initialValue: 300,
    change: (newVal) => {
      console.log('Slider was changed', newVal)
    }
  }),
]))

I'll push my branch soon as a WIP PR so that people can give some implementation feedback.

[MarshallOfSound]

Just pushed my WIP PR up to #8095

Feedback is appreciated

[MiracleBlue]

Wow. Just wow. I'm so excited for this. I'll see if I can give any useful feedback to your PR, really hoping to use this as soon as possible :D just got my new MBP yesterday, the potential for this touch bar thing is huge.

[sunjay]

Not sure if this is being considered, but other laptop manufacturers might go copy Apple's touchbar and add their own spin to it. In case that happens, we should try to keep that in mind when designing these APIs so a bunch of code doesn't eventually break. (Unless the plan is to keep an OSX specific TouchBar API and then add a new API for every touch bar)

[MarshallOfSound]

@sunjay The TouchBarAPI is currently in Objective C which will only ever run or compile on macOS devices. If another manufacturer adds a device similar to the touch bar their API would be completely different and almost certainly in a different language so it would not causes any clashes 👍

[mrjacobbloom]

I think @sunjay means to keep the JS side generalized so that we can use the same code if other manufacturers create similar products, even if the low-level stuff that Electron does is different

[pozylon]

@Airhogs777 @sunjay I suppose adding an abstraction layer is fine if other operating systems jump on the TouchBar train. But at the moment, there's no need for abstraction.

Why?
Watching the Microsoft strategy clearly shows us that they are pushing for "all-touch" desktop computing. Universal OS for desktop and tablets, Surface , Surface Studio, ... I don't see a TouchBar fit in there nicely.

The Linux/Unix based desktop environment maintainers would have to add TouchBar support for their OS running on Apple hardware. If they are clever (and they are) they are certainly going to avoid using any other API method signatures than Apple.

As long Microsoft does not add OS-level support for TouchBar + SDK, no other hardware producer but Apple will release any device.

[omartan]

Hmm, here's a simple suggestion for Touchbar v1, it'll probably look like how Safari Touchbar work where you can press the button to enter URL except this will access the command palette and once clicked, it'll display suggestions based on last/common used commands?

Longer down the road, probably able to use Git to commit?

[koenpunt]

Also, anyone know if Chromium is planning to support this on the broader web? Would be a bummer to implement it, only to have them come up with a nice general solution.

There's this: https://bugs.chromium.org/p/chromium/issues/detail?id=660126

[davecarlson]

any further progress on this ?

[kevinsawicki]

This is currently blocked on Chrome not supporting building against macOS SDK 10.12 which this pull request requires. Currently pinch zooming breaks in Chrome if you build it with SDK 10.11+.

It looks like it is actively being worked on: https://bugs.chromium.org/p/chromium/issues/detail?id=680927

[jpsim]

Touch bar support was added to Chromium according to https://bugs.chromium.org/p/chromium/issues/detail?id=660126.

[kevinsawicki]

This has landed on master and will be included in the next Electron beta release, 1.6.3, see the new docs for more details.

[etiennepinchon]

Wait a minute! Is that it? :( I just tried the Touchbar api, at first I was excited, but quickly realized it is far from being usable! Sure you can create a few buttons, sliders and stuff but what about the deeper stuff?

For instance being able to trigger a color picker so that it opens without the user to press the button or all the default buttons like the emoji bar? the suggestions bar?
A LOT is missing! :( I know I know haha this is experimental but there is so much possibilities!
It would be sad to stop here!

[MarshallOfSound]

For instance being able to trigger a color picker so that it opens without the user to press the button

There is not way to programattically trigger a touch bar item, check out the color picker docs.
We can change some options but we can't trigger it.
https://developer.apple.com/documentation/appkit/nscolorpickertouchbaritem?language=objc

the emoji bar

You would implement this yourself using the TouchBarScrubber item. There isn't a built in "emoji picker" for the touch bar

the suggestions bar

I assume you mean typing suggestions, this is dependent on Chromium's implementation of that feature as the text views in Chromium aren't native text views.

[patrikholcak]

I’ve always felt like the electron’s TouchBar api itself is highly developer unfriendly. TouchBar feels more like a UI element that you want to change depending on the state of your app and respond to it (dispatch actions, set state,…) and communicating via ipc, having to mount and dismount events might be a bit confusing for an inexperienced dev and makes it harder to have more layouts.

Defining it in components of your app would allow it to respond to the state more easier. I made a small library (shameless plug) that allows developers to define TouchBar using React components, but I’m thinking that it might be easier to just port the current api to renderer process (which would also make it library agnostic) and have it communicate via ipc for you, AKA:

// ui-component.js
class App {
  constructor() {
    new TouchBar({
      items: [
        new TouchBarButton({
          click: this.doSomething
        }),
        new TouchBarSlider({}),
        …
      ],
      escapeItem: new TouchBarButton({…})
    })
  }
}

Could this be interesting enough for a library/PR?

[MarshallOfSound]

Renderer wrappers for the touch bar API are definitely things that belong in userland. The JS API simply gives you maximum access to the native apis. What you do with that / how you model it is entirely up to you.

[etiennepinchon]

@MarshallOfSound Thank you for your details response!

There is not way to programatically trigger a touch bar item

So let's say I have an non-native color picker in my app, there is not way to open the touchbar color picker to selection mode when my color picker is open ?

I think this is really the missing key element to get a responsive feeling (UX <-> Touchbar) .