Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add texture support for macOS shell. #8507

Merged
merged 40 commits into from
Oct 10, 2019
Merged
Changes from 16 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
5302cf8
Merge pull request #1 from flutter/master
cloudwebrtc Mar 28, 2019
2953e13
Merge remote-tracking branch 'upstream/master'
cloudwebrtc Apr 9, 2019
4b422b3
Add texture support for macOS shell.
cloudwebrtc Apr 9, 2019
fa3f8b0
code format.
cloudwebrtc Apr 9, 2019
a222a4a
Add missing file.
cloudwebrtc Apr 9, 2019
668e656
Update FLETexture.h
cloudwebrtc Apr 9, 2019
97f1e91
Update FLEPluginRegistrar.h
cloudwebrtc Apr 10, 2019
cd636b4
Restore header file order.
cloudwebrtc Apr 10, 2019
799e9c1
Update code styles and comments.
cloudwebrtc Apr 10, 2019
52b1417
Rename textureId to textureID.
cloudwebrtc Apr 10, 2019
e49c549
Update comment for FLEExternalTextureGL.h.
cloudwebrtc Apr 10, 2019
7589f46
Update comments and partial code.
cloudwebrtc Apr 17, 2019
2c60631
Rename the texture in some methods to disambiguate.
cloudwebrtc Apr 17, 2019
42b7502
Merge remote-tracking branch 'upstream/master'
cloudwebrtc Apr 18, 2019
dbcdf16
Merge branch 'master' into macos_texture
cloudwebrtc Apr 18, 2019
9b15427
Fix errors caused by merge master conflicts.
cloudwebrtc Apr 18, 2019
e4ff652
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc May 19, 2019
ba2efa3
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Jul 11, 2019
9c19bac
clang format.
cloudwebrtc Jul 16, 2019
6ff0a76
gn format.
cloudwebrtc Jul 16, 2019
ec8083a
Add files to licenses_flutter.
cloudwebrtc Jul 16, 2019
ff49e14
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Sep 24, 2019
ee09e69
clang-format.
cloudwebrtc Sep 24, 2019
0f06004
Update.
cloudwebrtc Sep 24, 2019
09156bf
Use shared FlutterTexture.h.
cloudwebrtc Sep 25, 2019
235314a
clang-format.
cloudwebrtc Sep 25, 2019
650c00a
Fixed FlutterTexture.h import path for ios.
cloudwebrtc Sep 25, 2019
d4c0be8
Update.
cloudwebrtc Sep 26, 2019
cda947c
clang-format.
cloudwebrtc Sep 26, 2019
e78d2e1
Change _pixelBuffer to a local variable.
cloudwebrtc Sep 27, 2019
eb3bf39
Add missing static on a local constant
stuartmorgan Sep 27, 2019
d622f01
Fix variable naming inconsistency
stuartmorgan Sep 27, 2019
ec70d4a
Merge branch 'master' into macos_texture
cloudwebrtc Oct 5, 2019
2871956
Use drawImageRect to draw textures of undesired size.
cloudwebrtc Oct 5, 2019
9b63c65
Merge branch 'macos_texture' of https://github.com/cloudwebrtc/flutte…
cloudwebrtc Oct 5, 2019
4962f91
clang-format.
cloudwebrtc Oct 5, 2019
b20be2c
Fixed CI errors.
cloudwebrtc Oct 5, 2019
067e8a1
update.
cloudwebrtc Oct 7, 2019
27c6f94
clang-format.
cloudwebrtc Oct 7, 2019
5f6833f
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Oct 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions shell/platform/darwin/macos/BUILD.gn
Original file line number Diff line number Diff line change
@@ -31,6 +31,7 @@ _flutter_framework_dir = "$root_out_dir/$_flutter_framework_filename"
# the Flutter engine source root.
_flutter_framework_headers = [
"framework/Headers/FlutterMacOS.h",
"framework/Headers/FLETexture.h",
cloudwebrtc marked this conversation as resolved.
Show resolved Hide resolved
"framework/Headers/FLEOpenGLContextHandling.h",
"framework/Headers/FLEPlugin.h",
"framework/Headers/FLEPluginRegistrar.h",
@@ -48,6 +49,8 @@ shared_library("create_flutter_framework_dylib") {
output_name = "$_flutter_framework_name"

sources = [
"framework/Source/FLEExternalTextureGL.h",
"framework/Source/FLEExternalTextureGL.mm",
"framework/Source/FLETextInputModel.h",
"framework/Source/FLETextInputModel.mm",
"framework/Source/FLETextInputPlugin.h",
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@
#import <Cocoa/Cocoa.h>

#import "FLEPlugin.h"
#import "FLETexture.h"

#if defined(FLUTTER_FRAMEWORK)
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterBinaryMessenger.h"
@@ -34,6 +35,11 @@ FLUTTER_EXPORT
*/
@property(nonnull, readonly) id<FlutterBinaryMessenger> messenger;

/**
* Returns a `FLETextureRegistrar` for registering textures provided by the plugin.
*/
@property(nonnull, readonly) id<FLETextureRegistrar> textures;
cloudwebrtc marked this conversation as resolved.
Show resolved Hide resolved

/**
* The view displaying Flutter content.
*
43 changes: 43 additions & 0 deletions shell/platform/darwin/macos/framework/Headers/FLETexture.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be identical to FlutterTexture.h other than the prefixes; now that macOS is in the same repo as iOS the goal is to share code whenever possible (and move away from the FLE prefix). You can just add that file to the frameworkSshared.gni list, and use the existing protocols in the macOS code.

(If you want to add these comments to FlutterTexture.h, ideally as a separate PR, that would be great, since the existing file isn't commented at all.)

Copy link
Contributor Author

@cloudwebrtc cloudwebrtc Apr 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chinmaygarde @stuartmorgan
I found in the macos test that CVOpenGLTextureCacheCreateTextureFromImage creates a GL_TEXTURE_RECTANGLE object instead of GL_TEXTURE_2D, which causes the texture to not be properly scaled to fill the flutter widget when rendering, so I used width/height in texture->copyPixelBuffer when the plugin receives the request, it scales the texture to the corresponding size according to the widget size provided by the engine, and then becomes normal when rendering.
I'm not sure to add width/height to ios/framework/Headers/FlutterTexture.h, or continue to use FLETexture.h, or should report a new issue in flutter engine.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to specify that the texture target is GL_TEXTURE_RECTANGLE by specifying the target property of FlutterOpenGLTexture. All the copying ought to be unnecessarily inefficient.

// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import <CoreVideo/CoreVideo.h>
#import <Foundation/Foundation.h>

/**
* Implement a texture object for the FLE plugin side.
*/
@protocol FLETexture <NSObject>

/**
* When the texture on the platform side is ready,
* the flutter engine will copy the texture buffer
* using copyPixelBuffer.
*/
- (nullable CVPixelBufferRef)copyPixelBuffer:(size_t)width height:(size_t)height;

@end

/**
* The protocol for an object managing registration for texture.
*/
@protocol FLETextureRegistrar <NSObject>

/**
* Register a |texture| object and return a textureID.
*/
- (int64_t)registerTexture:(nonnull id<FLETexture>)texture;

/**
* Mark a texture buffer is ready.
*/
- (void)textureFrameAvailable:(int64_t)textureID;

/**
* Unregister an existing Texture object.
*/
- (void)unregisterTexture:(int64_t)textureID;

@end
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@
#import "FLEOpenGLContextHandling.h"
#import "FLEPluginRegistrar.h"
#import "FLEReshapeListener.h"
#import "FLETexture.h"

#if defined(FLUTTER_FRAMEWORK)
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterBinaryMessenger.h"
@@ -37,8 +38,10 @@ typedef NS_ENUM(NSInteger, FlutterMouseTrackingMode) {
FLUTTER_EXPORT
@interface FLEViewController : NSViewController <FlutterBinaryMessenger,
FLEPluginRegistrar,
FLEReshapeListener,
FLEPluginRegistry,
FLEReshapeListener>
FLEReshapeListener,
FLETextureRegistrar>

/**
* The view this controller manages when launched in interactive mode (headless set to false). Must
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
#import "FLEPlugin.h"
#import "FLEPluginRegistrar.h"
#import "FLEReshapeListener.h"
#import "FLETexture.h"
#import "FLEView.h"
#import "FLEViewController.h"
#import "FlutterBinaryMessenger.h"
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import <Foundation/Foundation.h>

#import "flutter/shell/platform/darwin/macos/framework/Headers/FLETexture.h"
#import "flutter/shell/platform/embedder/embedder.h"
cloudwebrtc marked this conversation as resolved.
Show resolved Hide resolved

/**
* Used to bridge FLETexture object and handle the texture copy request the
* flutter engine.
*/
@interface FLEExternalTextureGL : NSObject

/**
* Initializes a texture adapter with |texture| return a instance.
*/
- (nonnull instancetype)initWithFLETexture:(nonnull id<FLETexture>)texture;

/**
* Accepts texture buffer copy request from the flutter engine.
* When the user side marks the textureId as available, the flutter engine will
* callback to this method and ask for populate the |openGLTexture| object,
* such as the texture type and the format of the pixel buffer and the texture object.
*/
- (BOOL)populateTextureWithWidth:(size_t)width
height:(size_t)height
openGLTexture:(nonnull FlutterOpenGLTexture*)openGLTexture;

/**
* Returns the ID for the FLETexture instance.
*/
- (int64_t)textureID;

@end
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import "flutter/shell/platform/darwin/macos/framework/Source/FLEExternalTextureGL.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're moving away from using FLE as a prefix, and this code isn't part of an unstable public API surface, you can use Flutter rather than FLE as the prefix for this class, so that it doesn't need to be renamed later.


#import <AppKit/AppKit.h>
#import <CoreVideo/CoreVideo.h>
#import <OpenGL/gl.h>

static void OnGLTextureRelease(CVPixelBufferRef pixelBuffer) {
CVPixelBufferRelease(pixelBuffer);
}

@implementation FLEExternalTextureGL {
/**
* OpenGL texture cache.
*/
CVOpenGLTextureCacheRef _openGLTextureCache;
/**
* The pixel buffer copied from the user side will be released
* when the flutter engine renders it.
*/
CVPixelBufferRef _pixelBuffer;
/**
* User side texture object, used to copy pixel buffer.
*/
id<FLETexture> _texture;
cloudwebrtc marked this conversation as resolved.
Show resolved Hide resolved
}

- (instancetype)initWithFLETexture:(id<FLETexture>)texture {
self = [super init];
if (self) {
_texture = texture;
}
return self;
}

- (int64_t)textureID {
return reinterpret_cast<int64_t>(self);
}

- (BOOL)populateTextureWithWidth:(size_t)width
height:(size_t)height
openGLTexture:(FlutterOpenGLTexture*)openGLTexture {
// Copy the pixel buffer from the FLETexture instance implemented on the user side.
_pixelBuffer = [_texture copyPixelBuffer:width height:height];

if (!_pixelBuffer) {
return NO;
}

// Create the opengl texture cache if necessary.
if (!_openGLTextureCache) {
CGLContextObj context = [NSOpenGLContext currentContext].CGLContextObj;
CGLPixelFormatObj format = CGLGetPixelFormat(context);
if (CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, context, format, NULL,
&_openGLTextureCache) != kCVReturnSuccess) {
NSLog(@"Could not create texture cache.");
CVPixelBufferRelease(_pixelBuffer);
return NO;
}
}

// Try to clear the cache of OpenGL textures to save memory.
CVOpenGLTextureCacheFlush(_openGLTextureCache, 0);

CVOpenGLTextureRef cvOpenGLTexture = NULL;
if (CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _openGLTextureCache,
_pixelBuffer, NULL,
&cvOpenGLTexture) != kCVReturnSuccess) {
CVPixelBufferRelease(_pixelBuffer);
return NO;
}
CVPixelBufferRelease(_pixelBuffer);
cloudwebrtc marked this conversation as resolved.
Show resolved Hide resolved

openGLTexture->target = static_cast<uint32_t>(CVOpenGLTextureGetTarget(cvOpenGLTexture));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloudwebrtc Are you saying that CVOpenGLTextureGetTarget is not returning GL_TEXTURE_RECTANGLE? I am a little bit confused. There should be no issues with having to convert to GL_TEXTURE_2D here.

Copy link
Contributor Author

@cloudwebrtc cloudwebrtc Apr 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chinmaygarde See CVOpenGLTextureGetTarget from the code context
  It should be the returned GL_TEXTURE_RECTANGLE, but the problem is with the engine's texture rendering.
The whole picture of the problem is:

  • 1, when I patched Texture for flutter/engine to macOS 10.14.4 (xcode 10.2)
    The following error will appear in the console at runtime, without displaying any texture frames.
    image
  • 2, so I tried to find the reason
    I solved it by patching skia
diff --git a/src/sksl/SkSLGLSLCodeGenerator.cpp b/src/sksl/SkSLGLSLCodeGenerator.cpp
index d003dc9397..04357e2a4c 100644
--- a/src/sksl/SkSLGLSLCodeGenerator.cpp
+++ b/src/sksl/SkSLGLSLCodeGenerator.cpp
@@ -656,7 +656,7 @@ void GLSLCodeGenerator::writeFunctionCall(const FunctionCall& c) {
                         proj = false;
                         break;
                     case SpvDimRect:
-                        dim = "Rect";
+                        dim = "2DRect";
                         proj = false;
                         break;
                     case SpvDimBuffer:

Then I can see the texture display, but the picture is not correct.
image

  • 3, then I use the width/height in TextureFrameCallback to resized the video frame to this size on the plugin side, then get the texture display of the correct zoom size, and feel free to stretch ExampleEmbedder.app to display correctly.

shell/platform/embedder/embedder.h#L214

Typedef bool (*TextureFrameCallback)(void* /* user data */,
                                      Int64_t /* texture identifier */,
                                      Size_t /* width */,
                                      Size_t /* height */,
                                      FlutterOpenGLTexture* /* texture out */);

Looks like this:
plugins/flutter_webrtc/macos/FlutterRTCVideoRenderer.m#L52

I am not sure that the type of dim in SpvDimRect is a bug in skia on macos.

openGLTexture->name = static_cast<uint32_t>(CVOpenGLTextureGetName(cvOpenGLTexture));
openGLTexture->format = static_cast<uint32_t>(GL_RGBA8);
openGLTexture->destruction_callback = (VoidCallback)OnGLTextureRelease;
openGLTexture->user_data = cvOpenGLTexture;
return YES;
}

- (void)dealloc {
CVOpenGLTextureCacheRelease(_openGLTextureCache);
}

@end
Original file line number Diff line number Diff line change
@@ -9,6 +9,7 @@
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterCodecs.h"
#import "flutter/shell/platform/darwin/macos/framework/Headers/FLEReshapeListener.h"
#import "flutter/shell/platform/darwin/macos/framework/Headers/FLEView.h"
#import "flutter/shell/platform/darwin/macos/framework/Source/FLEExternalTextureGL.h"
#import "flutter/shell/platform/darwin/macos/framework/Source/FLETextInputPlugin.h"
#import "flutter/shell/platform/embedder/embedder.h"

@@ -91,6 +92,13 @@ - (void)dispatchMouseEvent:(nonnull NSEvent*)event phase:(FlutterPointerPhase)ph
*/
- (void)dispatchKeyEvent:(NSEvent*)event ofType:(NSString*)type;

/**
* Forwards texture copy request to the corresponding texture via |textureID|.
*/
- (BOOL)populateTextureWithIdentifier:(int64_t)textureID
width:(size_t)width
height:(size_t)height
openGLTexture:(FlutterOpenGLTexture*)openGLTexture;
@end

#pragma mark - Static methods provided to engine configuration
@@ -142,6 +150,20 @@ static bool OnMakeResourceCurrent(FLEViewController* controller) {
return true;
}

/**
* Dispatching the texture copy request forward to the controller.
*/
static bool OnAcquireExternalTexture(FLEViewController* controller,
stuartmorgan marked this conversation as resolved.
Show resolved Hide resolved
int64_t texture_identifier,
size_t width,
size_t height,
FlutterOpenGLTexture* openGLTexture) {
return [controller populateTextureWithIdentifier:texture_identifier
width:width
height:height
openGLTexture:openGLTexture];
}

#pragma mark Static methods provided for headless engine configuration

static bool HeadlessOnMakeCurrent(FLEViewController* controller) {
@@ -181,6 +203,9 @@ @implementation FLEViewController {
// A message channel for passing key events to the Flutter engine. This should be replaced with
// an embedding API; see Issue #47.
FlutterBasicMessageChannel* _keyEventChannel;

// A mapping of textureID to internal FLEExternalTextureGL adapter.
NSMutableDictionary<NSNumber*, FLEExternalTextureGL*>* _textures;
}

@dynamic view;
@@ -191,6 +216,7 @@ @implementation FLEViewController {
static void CommonInit(FLEViewController* controller) {
controller->_messageHandlers = [[NSMutableDictionary alloc] init];
controller->_additionalKeyResponders = [[NSMutableOrderedSet alloc] init];
controller->_textures = [[NSMutableDictionary alloc] init];
}

- (instancetype)initWithCoder:(NSCoder*)coder {
@@ -367,7 +393,9 @@ + (FlutterRendererConfig)createRenderConfigHeadless:(BOOL)headless {
.open_gl.clear_current = (BoolCallback)OnClearCurrent,
.open_gl.present = (BoolCallback)OnPresent,
.open_gl.fbo_callback = (UIntCallback)OnFBO,
.open_gl.make_resource_current = (BoolCallback)OnMakeResourceCurrent};
.open_gl.make_resource_current = (BoolCallback)OnMakeResourceCurrent,
.open_gl.gl_external_texture_frame_callback =
(TextureFrameCallback)OnAcquireExternalTexture};
return config;
}
}
@@ -472,6 +500,32 @@ - (void)dispatchKeyEvent:(NSEvent*)event ofType:(NSString*)type {
}];
}

#pragma mark - FlutterTextureRegistrar

- (BOOL)populateTextureWithIdentifier:(int64_t)textureID
width:(size_t)width
height:(size_t)height
openGLTexture:(FlutterOpenGLTexture*)openGLTexture {
return [_textures[@(textureID)] populateTextureWithWidth:width height:height openGLTexture:openGLTexture];
}

- (int64_t)registerTexture:(id<FLETexture>)texture {
FLEExternalTextureGL* fleTexture = [[FLEExternalTextureGL alloc] initWithFLETexture:texture];
int64_t textureID = [fleTexture textureID];
FlutterEngineRegisterExternalTexture(_engine, textureID);
_textures[@(textureID)] = fleTexture;
return textureID;
}

- (void)textureFrameAvailable:(int64_t)textureID {
FlutterEngineMarkExternalTextureFrameAvailable(_engine, textureID);
}

- (void)unregisterTexture:(int64_t)textureID {
FlutterEngineUnregisterExternalTexture(_engine, textureID);
[_textures removeObjectForKey:@(textureID)];
}

#pragma mark - FLEReshapeListener

/**
@@ -515,6 +569,10 @@ - (void)setMessageHandlerOnChannel:(nonnull NSString*)channel
return self;
}

- (id<FLETextureRegistrar>)textures {
return self;
}

- (void)addMethodCallDelegate:(nonnull id<FLEPlugin>)delegate
channel:(nonnull FlutterMethodChannel*)channel {
[channel setMethodCallHandler:^(FlutterMethodCall* call, FlutterResult result) {