Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Language-Understanding LSP #1711

Merged
merged 102 commits into from
Jan 20, 2020
Merged
Show file tree
Hide file tree
Changes from 100 commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
a3be983
Add LSP of LG
cosmicshuai Sep 26, 2019
967c133
change folder name
cosmicshuai Sep 26, 2019
50eead9
change folder location
cosmicshuai Sep 26, 2019
9e4da28
seperate server and client demo
cosmicshuai Sep 26, 2019
1963f3a
add completion and hover for builtin-functions
cosmicshuai Sep 26, 2019
4ed777e
change file names
cosmicshuai Sep 26, 2019
ba5b6d4
change dependency
cosmicshuai Sep 27, 2019
c58ae8c
refactor the package
cosmicshuai Sep 27, 2019
85a2523
update demo readme
cosmicshuai Sep 27, 2019
415a11c
merge changes from master
cosmicshuai Sep 29, 2019
934211d
make some refine
Danieladu Oct 3, 2019
e825a94
Merge branch 'master' into shuwan/AddingLG_LSP
Danieladu Oct 3, 2019
6699260
remove npm lock file
Danieladu Oct 3, 2019
2a3957b
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 9, 2019
f4dd101
add demos to workspace and run test in 1 command
cosmicshuai Oct 9, 2019
4e34574
remove declaration files
cosmicshuai Oct 9, 2019
615266e
change tsc outDir to dist, simplify test command
cosmicshuai Oct 9, 2019
b24276c
Merge branch 'master' into shuwan/AddingLG_LSP
boydc2014 Oct 9, 2019
7a792ed
add syntax highlight in demo and new API
cosmicshuai Oct 11, 2019
70729c3
Merge branch 'shuwan/AddingLG_LSP' of https://github.com/microsoft/Bo…
cosmicshuai Oct 11, 2019
79264f0
change naming of the project and move the demo
cosmicshuai Oct 12, 2019
e0927c6
merge from master and resolve confilcts
cosmicshuai Oct 12, 2019
3e35a12
remove package.json in demo
cosmicshuai Oct 12, 2019
1ec29ec
Merge branch 'shuwan/AddingLG_LSP' of https://github.com/microsoft/Bo…
cosmicshuai Oct 12, 2019
ed2a6d7
change the content in readme
cosmicshuai Oct 12, 2019
d3eea2d
fix lg-lsp package publish problem
cosmicshuai Oct 14, 2019
e368b33
fix build command and redundent d.ts files
cosmicshuai Oct 14, 2019
f4e384d
integrate LG LSP server to composer server
cosmicshuai Oct 14, 2019
0fda7c6
change api in demo
cosmicshuai Oct 14, 2019
f8d78a2
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 14, 2019
a8835be
Merge branch 'master' into shuwan/AddingLG_LSP
boydc2014 Oct 14, 2019
862ddc8
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 15, 2019
f561ee1
change the order of commands in build:prod
cosmicshuai Oct 15, 2019
67726d1
Merge branch 'shuwan/AddingLG_LSP' of https://github.com/microsoft/Bo…
cosmicshuai Oct 15, 2019
5d58244
delete redundent files generated from build
cosmicshuai Oct 15, 2019
6f03b83
change lg-lsp-server api to attachLSPServer
cosmicshuai Oct 15, 2019
98bb820
remove gitignore in lg-lsp-server demo
cosmicshuai Oct 15, 2019
26d6ae0
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 17, 2019
57a7e47
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 22, 2019
54662d7
remove attachLSPServer in server
cosmicshuai Oct 22, 2019
ebc3218
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Oct 29, 2019
9e36bd3
Merge branch 'master' into shuwan/AddingLG_LSP
zhixzhan Nov 1, 2019
f4732c2
Merge branch 'master' into shuwan/AddingLG_LSP
zhixzhan Nov 1, 2019
96739a5
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Nov 5, 2019
78a3529
fix token rules and suggestion context awareness
cosmicshuai Nov 5, 2019
31753fa
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Nov 5, 2019
9575102
Merge branch 'master' into shuwan/AddingLG_LSP
cosmicshuai Nov 14, 2019
474511f
init of LU LSP
cosmicshuai Nov 18, 2019
352a703
add diagnostic in LSP
cosmicshuai Nov 19, 2019
413132f
add auto suggestions
cosmicshuai Nov 21, 2019
b0f74c3
fix
cosmicshuai Nov 22, 2019
512ca2c
remove redundent in client
cosmicshuai Nov 22, 2019
fb0fcc8
add doc on type config
cosmicshuai Nov 24, 2019
d429fd9
fix
cosmicshuai Nov 24, 2019
67edb14
modify doc type format
cosmicshuai Nov 25, 2019
5eb3d05
fix
cosmicshuai Nov 25, 2019
8e2784b
change version of bf-lu
cosmicshuai Nov 25, 2019
57f955a
fix diagnostic
cosmicshuai Nov 26, 2019
5323b40
refactor getRangeAtPosition method
cosmicshuai Nov 26, 2019
99f2141
fix tokens
cosmicshuai Nov 26, 2019
068b090
remove black line
cosmicshuai Nov 26, 2019
9508fa4
fix tokens
cosmicshuai Nov 26, 2019
c57ce7d
add roles suggestions
cosmicshuai Nov 28, 2019
4791d96
add regex entity seperated line definition
cosmicshuai Dec 2, 2019
37b58cd
add suggestions for ml entity
cosmicshuai Dec 2, 2019
61891a1
matching only ML entities
cosmicshuai Dec 2, 2019
28481da
fix
cosmicshuai Dec 2, 2019
a362a32
update documents
cosmicshuai Dec 6, 2019
5734502
fix
cosmicshuai Dec 27, 2019
ef16147
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Dec 27, 2019
1324176
integrate lu-lsp to composer
cosmicshuai Dec 27, 2019
83b31ae
Merge branch 'shuwan/AddingLG_LSP' into shuwan/AddLu_LSP
cosmicshuai Dec 27, 2019
ac0eeab
upgrade botbuilder-expressions
zhixzhan Dec 28, 2019
522e228
clean up
zhixzhan Dec 28, 2019
d9178d1
update sample
zhixzhan Dec 28, 2019
32e45be
Merge branch 'master' into shuwan/AddLu_LSP
zhixzhan Dec 28, 2019
4bb6108
add test
zhixzhan Dec 28, 2019
67faf9f
add syntax highlight
cosmicshuai Dec 31, 2019
0dbcb31
add token rule for {@ expr
cosmicshuai Dec 31, 2019
1d1d42a
add a case in token
cosmicshuai Dec 31, 2019
b33c89d
fix import statement token
cosmicshuai Dec 31, 2019
c7a5ea4
fix composite entity auto complete
cosmicshuai Dec 31, 2019
499fb49
fix entity suggestion and tokens
cosmicshuai Jan 6, 2020
93e94c7
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 7, 2020
08e5115
add labeling experience of add unlabeled entity
cosmicshuai Jan 9, 2020
9c3553e
Merge branch 'shuwan/AddLu_LSP' of https://github.com/microsoft/BotFr…
cosmicshuai Jan 9, 2020
589a13a
fix typos
cosmicshuai Jan 9, 2020
e1f402e
fix lint
cosmicshuai Jan 9, 2020
09b2028
fix redundent edits
cosmicshuai Jan 9, 2020
fa49d8b
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 9, 2020
363f926
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 10, 2020
54404a8
fix unnecessary space
cosmicshuai Jan 12, 2020
481ec44
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 13, 2020
1296d71
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 14, 2020
3b2bc7d
fix naming and role suggestions
cosmicshuai Jan 14, 2020
46a4cbc
fix labeling and error postion
cosmicshuai Jan 14, 2020
fcf6d9b
fix insert text in wrong line, intent usesFeature
cosmicshuai Jan 15, 2020
0320505
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 15, 2020
ab4d6ad
update bf-lu version
cosmicshuai Jan 15, 2020
7691994
change find a valid luisJson and suggest composite
cosmicshuai Jan 15, 2020
01041b2
Merge branch 'master' into shuwan/AddLu_LSP
cosmicshuai Jan 16, 2020
8a3976d
add unit test for LU lsp basic funcionalities
cosmicshuai Jan 16, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,23 @@
"webRoot": "${workspaceFolder}"
},
{
"name": "LSP Server",
"name": "LU LSP Server",
"type": "node",
"request": "launch",
"args": [
"${workspaceFolder}/Composer/packages/tools/language-servers/language-understanding/demo/src/server.ts"
],
"runtimeArgs": [
"--nolazy",
"-r",
"${workspaceFolder}/Composer/node_modules/ts-node/register"
],
"sourceMaps": true,
"cwd": "${workspaceFolder}/Composer/packages/tools/language-servers/language-understanding/demo/src",
"protocol": "inspector",
},
{
"name": "LG LSP Server",
"type": "node",
"request": "launch",
"args": [
Expand Down
2 changes: 1 addition & 1 deletion Composer/packages/client/setupProxy.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ const paths = require('./config/paths');
const proxySetting = require(paths.appPackageJson).proxy;

module.exports = function(app) {
const wsProxy = proxy('/lg-language-server', {
const wsProxy = proxy(['/lg-language-server', '/lu-language-server'], {
target: proxySetting.replace('/^http/', 'ws'),
changeOrigin: true,
ws: true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ export default function CodeEditor(props) {
minimap: 'on',
lineDecorationsWidth: undefined,
lineNumbersMinChars: false,
glyphMargin: true,
autoClosingBrackets: 'always',
wordBasedSuggestions: false,
autoIndent: true,
formatOnType: true,
lightbulb: {
enabled: true,
},
}}
errorMsg={errorMsg}
value={content}
Expand Down
4 changes: 2 additions & 2 deletions Composer/packages/lib/code-editor/demo/src/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
import React from 'react';
import ReactDOM from 'react-dom';

import App from './lgEditor';
// import App from './lgEditor';
// import App from './jsonEditor';
// import App from './luEditor';
import App from './luEditor';
// import App from './lgJsonEditor';
// import App from './inlineEditor';
// import App from './multiEditors';
Expand Down
4 changes: 4 additions & 0 deletions Composer/packages/lib/code-editor/demo/src/luEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,10 @@ export default function App() {
const props = {
value,
onChange,
languageServer: {
port: 5003,
path: '/lu-language-server',
},
};
return <LuEditor {...props} />;
}
127 changes: 123 additions & 4 deletions Composer/packages/lib/code-editor/src/LuEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,131 @@
// Licensed under the MIT License.

import React from 'react';
import { listen, MessageConnection } from 'vscode-ws-jsonrpc';
import * as monacoEditor from '@bfcomposer/monaco-editor/esm/vs/editor/editor.api';
import * as monacoCore from 'monaco-editor-core';
import get from 'lodash/get';
import { MonacoServices, MonacoLanguageClient } from 'monaco-languageclient';

import { registerLULanguage } from './languages';
import { createUrl, createWebSocket, createLanguageClient } from './utils/lspUtil';
import { RichEditor, RichEditorProps } from './RichEditor';

const LU_HELP = 'https://aka.ms/lu-file-format';
const placeholder = `> See ${LU_HELP} to learn about supported LU concepts.`;
const LU_HELP = 'https://github.com/microsoft/botframework-cli/blob/master/packages/lu/docs/lu-file-format.md';
const placeholder = `> To learn more about the LU file format, read the documentation at
> ${LU_HELP}`;

export function LuEditor(props: RichEditorProps) {
return <RichEditor placeholder={placeholder} helpURL={LU_HELP} {...props} />;
export interface LUOption {
inline: boolean;
content: string;
template?: {
name: string;
parameters?: string[];
body: string;
};
}

export interface LULSPEditorProps extends RichEditorProps {
luOption?: LUOption;
languageServer?:
| {
host?: string;
hostname?: string;
port?: number | string;
path: string;
}
| string;
}

const defaultLUServer = {
path: '/lu-language-server',
};
declare global {
interface Window {
monacoServiceInstance: MonacoServices;
monacoLUEditorInstance: MonacoLanguageClient;
}
}

function convertEdit(serverEdit: any) {
cosmicshuai marked this conversation as resolved.
Show resolved Hide resolved
return {
range: {
startLineNumber: serverEdit.range.start.line,
startColumn: serverEdit.range.start.character,
endLineNumber: serverEdit.range.end.line,
endColumn: serverEdit.range.end.character,
},
text: serverEdit.newText,
forceMoveMarkers: true,
};
}

export function LuEditor(props: LULSPEditorProps) {
const options = {
quickSuggestions: true,
wordBasedSuggestions: false,
formatOnType: true,
};

const { languageServer, ...restProps } = props;
const luServer = languageServer || defaultLUServer;

const editorWillMount = (monaco: typeof monacoEditor) => {
registerLULanguage(monaco);
if (typeof props.editorWillMount === 'function') {
return props.editorWillMount(monaco);
}
};
const editorDidMount = (editor: monacoEditor.editor.IStandaloneCodeEditor, monaco: typeof monacoEditor) => {
if (!window.monacoServiceInstance) {
window.monacoServiceInstance = MonacoServices.install(editor as monacoCore.editor.IStandaloneCodeEditor | any);
}

if (!window.monacoLUEditorInstance) {
const uri = get(editor.getModel(), 'uri._formatted', '');
const url = createUrl(luServer);
const webSocket: WebSocket = createWebSocket(url);
listen({
webSocket,
onConnection: (connection: MessageConnection) => {
const languageClient = createLanguageClient('LU Language Client', ['botframeworklu'], connection);
if (!window.monacoLUEditorInstance) {
window.monacoLUEditorInstance = languageClient;
}

editor.addCommand(monaco.KeyMod.Shift | monaco.KeyCode.Enter, function() {
const position = editor.getPosition();
languageClient.sendRequest('labelingExperienceRequest', { uri, position });
});
languageClient.onReady().then(() =>
languageClient.onNotification('addUnlabelUtterance', result => {
const edits = result.edits.map(e => {
return convertEdit(e);
});
editor.executeEdits(uri, edits);
})
);
const disposable = languageClient.start();
connection.onClose(() => disposable.dispose());
},
});
}

if (typeof props.editorDidMount === 'function') {
return props.editorDidMount(editor, monaco);
}
};

return (
<RichEditor
placeholder={placeholder}
helpURL={LU_HELP}
{...restProps}
theme={'lutheme'}
cosmicshuai marked this conversation as resolved.
Show resolved Hide resolved
language={'botframeworklu'}
cosmicshuai marked this conversation as resolved.
Show resolved Hide resolved
options={options}
editorWillMount={editorWillMount}
editorDidMount={editorDidMount}
/>
);
}
1 change: 1 addition & 0 deletions Composer/packages/lib/code-editor/src/languages/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
// Licensed under the MIT License.

export * from './lg';
export * from './lu';
77 changes: 77 additions & 0 deletions Composer/packages/lib/code-editor/src/languages/lu.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

import * as monacoEditor from '@bfcomposer/monaco-editor/esm/vs/editor/editor.api';

export function registerLULanguage(monaco: typeof monacoEditor) {
monaco.languages.setMonarchTokensProvider('botframeworklu', {
tokenizer: {
root: [
[/^\s*#/, { token: 'intent', next: '@intent' }],
[/^\s*@/, { token: 'entity-identifier', goBack: 1, next: '@entityMode' }],
[/^\s*>\s*[\s\S]*$/, { token: 'comments' }],
],

intent: [
[/^\s*#/, { token: 'intent', next: '@intent' }],
[/^\s*-/, { token: 'utterrance-indentifier', next: '@utterrance' }],
[/^\s*>\s*[\s\S]*$/, { token: 'comments' }],
[/^\s*@/, { token: 'entity-identifier', goBack: 1, next: '@entityMode' }],
[/.*$/, 'intent'],
],
utterrance: [
[/^\s*#/, { token: 'intent', next: '@intent' }],
[/^\s*>\s*[\s\S]*$/, { token: 'comments' }],
[/^\s*-/, { token: 'utterrance-indentifier', next: 'utterrance' }],
[/^\s*@/, { token: 'entity-identifier', goBack: 1, next: '@entityMode' }],
[/({)(\s*[\w.:\s]*\s*)(=)(\s*[\w.]*\s*)(})/, ['lb', 'pattern', 'equal', 'entity-name', 'rb']],
[/({\s*@)(\s*[\w.]*\s*)(})/, ['lb', 'entity-name', 'rb']],
// eslint-disable-next-line security/detect-unsafe-regex
[/\s*\[[\w\s.]+\]\(.{1,2}\/[\w.*]+(#[\w.?]+)?\)/, 'import-desc'],
[/./, 'utterance-other'],
],
entityMode: [
[/^\s*#/, { token: 'intent', next: '@intent' }],
[/^\s*>\s*[\s\S]*$/, { token: 'comments' }],
[/^\s*-/, { token: 'utterrance-indentifier', next: 'utterrance' }],
[
/(@\s*)(ml|prebuilt|regex|list|composite|patternany|phraselist)(\s*\w*)/,
['intent-indentifier', 'entity-type', 'entity-name'],
],
[/(@\s*)(\s*\w*)/, ['intent-indentifier', 'entity-name']],
[/\s*(hasRoles|useFeature)\s*/, 'keywords'],
[/.*$/, 'entity-other', '@pop'],
],
},
});

monaco.languages.register({
id: 'botframeworklu',
cosmicshuai marked this conversation as resolved.
Show resolved Hide resolved
extensions: ['.lu'],
aliases: ['LU', 'language-understanding'],
mimetypes: ['application/lu'],
});

monaco.languages.setLanguageConfiguration('botframeworklu', {
autoClosingPairs: [
{ open: '{', close: '}' },
{ open: '[', close: ']' },
{ open: '(', close: ')' },
],
});

monaco.editor.defineTheme('lutheme', {
cosmicshuai marked this conversation as resolved.
Show resolved Hide resolved
base: 'vs',
inherit: false,
colors: {},
rules: [
{ token: 'intent', foreground: '0000FF' },
{ token: 'pattern', foreground: '00B7C3' },
{ token: 'entity-name', foreground: '038387' },
{ token: 'comments', foreground: '7A7A7A' },
{ token: 'import-desc', foreground: '00A32B' },
{ token: 'entity-type', foreground: 'DF2C2C' },
{ token: 'keywords', foreground: '0078D7' },
],
});
}
1 change: 1 addition & 0 deletions Composer/packages/server/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
"@azure/ms-rest-js": "^1.8.7",
"@bfc/indexers": "*",
"@bfc/lg-languageserver": "*",
"@bfc/lu-languageserver": "*",
"@bfc/shared": "*",
"@bfcomposer/lubuild": "1.1.2-preview",
"archiver": "^3.0.0",
Expand Down
20 changes: 20 additions & 0 deletions Composer/packages/server/src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import * as ws from 'ws';
import * as rpc from 'vscode-ws-jsonrpc';
import { IConnection, createConnection } from 'vscode-languageserver';
import { LGServer } from '@bfc/lg-languageserver';
import { LUServer } from '@bfc/lu-languageserver';

import { BotProjectService } from './services/project';
import { getAuthProvider } from './router/auth';
Expand Down Expand Up @@ -124,6 +125,14 @@ function launchLanguageServer(socket: rpc.IWebSocket) {
server.start();
}

function launchLuLanguageServer(socket: rpc.IWebSocket) {
const reader = new rpc.WebSocketMessageReader(socket);
const writer = new rpc.WebSocketMessageWriter(socket);
const connection: IConnection = createConnection(reader, writer);
const server = new LUServer(connection);
server.start();
}

attachLSPServer(wss, server, '/lg-language-server', webSocket => {
// launch language server when the web socket is opened
if (webSocket.readyState === webSocket.OPEN) {
Expand All @@ -134,3 +143,14 @@ attachLSPServer(wss, server, '/lg-language-server', webSocket => {
});
}
});

attachLSPServer(wss, server, '/lu-language-server', webSocket => {
// launch language server when the web socket is opened
if (webSocket.readyState === webSocket.OPEN) {
launchLuLanguageServer(webSocket);
} else {
webSocket.on('open', () => {
launchLuLanguageServer(webSocket);
});
}
});
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"dependencies": {
"@bfc/indexers": "*",
"botbuilder-lg": "4.7.0-preview.93464",
"botframework-expressions": "4.7.0-preview.93464",
"request-light": "^0.2.2",
"vscode-languageserver": "^5.3.0-next"
},
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/lib
/demo/lib
/demo/dist
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Language-understanding language server demo



## goto language-understanding directory, install packages, run
```
npm install
```

## How to start the demo

#### 1. under language-understanding directory, run
```
yarn start
```

### 2. go to code-editor directory

start luEditor demo, connect :5003/lu-language-server

## Features

### 1.Auto-suggestions for entity types and defined entities

### 2.Syntax and semantic diagonostics

### Auto-copletion for ml entities and list entities
Loading