Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Divide the package manager into separate components #2188

Merged
merged 66 commits into from
Apr 24, 2018
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
6205e1d
use async await
akshita31 Apr 10, 2018
c632ab2
async await
akshita31 Apr 10, 2018
01f8cb1
remove promise
akshita31 Apr 10, 2018
56675b9
remove packagejson
akshita31 Apr 10, 2018
19ee685
resolve conflicts
akshita31 Apr 10, 2018
400ba5e
Feature tests running with refactored package manager
akshita31 Apr 10, 2018
5c4bfc0
trial to mock server
akshita31 Apr 11, 2018
3968900
package add
akshita31 Apr 11, 2018
cbf50af
package manager test not running
akshita31 Apr 11, 2018
8c09369
Mocking the server running
akshita31 Apr 11, 2018
c74aecf
Refactoring packages-1
akshita31 Apr 12, 2018
3e58f13
Test for the downloader running using the mock server
akshita31 Apr 13, 2018
5565a2e
Using network settings
akshita31 Apr 13, 2018
c8135b5
Changing the package path
akshita31 Apr 13, 2018
9afc874
Dividing packages.ts into separate components
akshita31 Apr 13, 2018
9cc0016
modification for network settings
akshita31 Apr 13, 2018
98e7e74
Clean up the downloaders
akshita31 Apr 13, 2018
e20a434
launch.json
akshita31 Apr 13, 2018
97e83ec
Clean up
akshita31 Apr 13, 2018
e7b520a
do not use filter
akshita31 Apr 13, 2018
c6bee5a
use filter async
akshita31 Apr 13, 2018
2dac16c
Use tmp file interface
akshita31 Apr 13, 2018
85d867f
Use tmpfile interface
akshita31 Apr 13, 2018
348e03a
merge master
akshita31 Apr 13, 2018
57d4bff
Check for the event stream
akshita31 Apr 14, 2018
9afa848
package json
akshita31 Apr 14, 2018
e5b0d65
Changes
akshita31 Apr 16, 2018
7e77a6e
Using FilePathResolver
akshita31 Apr 16, 2018
76e9780
Remove using
akshita31 Apr 16, 2018
47a18d1
Testing for normal and fallback case working
akshita31 Apr 16, 2018
441153a
Resolve the paths
akshita31 Apr 16, 2018
df2f874
Remove failing case
akshita31 Apr 16, 2018
4c2c15c
package installer test-1
akshita31 Apr 17, 2018
7e44a06
Add package installer test
akshita31 Apr 17, 2018
e218cb4
Create tmp asset
akshita31 Apr 17, 2018
b96fb22
Package Downloader test refactored
akshita31 Apr 17, 2018
10b6073
Rename files
akshita31 Apr 17, 2018
b9ac792
resolve compile issues
akshita31 Apr 17, 2018
10c6811
Clean up installer
akshita31 Apr 17, 2018
053efe3
Clean up the tests
akshita31 Apr 17, 2018
286fa7c
Nits
akshita31 Apr 18, 2018
12dfc3f
Rename packages
akshita31 Apr 18, 2018
24a5c03
Package installer test
akshita31 Apr 18, 2018
9e26124
PR feedback
akshita31 Apr 18, 2018
bccfa13
Package Filter test
akshita31 Apr 19, 2018
2b6ba72
Remove yauzl.d.ts
akshita31 Apr 19, 2018
7b2586f
Filter test
akshita31 Apr 19, 2018
df111f6
Added test for invalid zip file
akshita31 Apr 19, 2018
3ff44bb
method for getting the request options
akshita31 Apr 19, 2018
ebe6a2a
remove imports
akshita31 Apr 19, 2018
3a97968
please resolve the path
akshita31 Apr 20, 2018
93ea65c
remove latest in settings
akshita31 Apr 20, 2018
cc4a782
trial for travis
akshita31 Apr 20, 2018
13ad87d
Test run
akshita31 Apr 20, 2018
a6ed9a0
Package Manager test executing
akshita31 Apr 21, 2018
fdc9e0c
Use free port in package manager test
akshita31 Apr 21, 2018
16e1c88
Package Manager (using a https server running)
akshita31 Apr 23, 2018
1fe14a5
using http mock server
akshita31 Apr 23, 2018
bde8b49
Downloader test running using free port
akshita31 Apr 23, 2018
7abd6a6
Nits
akshita31 Apr 23, 2018
3e323af
clean up
akshita31 Apr 24, 2018
41b7b87
Making fallback url optional
akshita31 Apr 24, 2018
28fab5a
using else if
akshita31 Apr 24, 2018
45300ac
PR feedback
akshita31 Apr 24, 2018
573ca83
Package Error
akshita31 Apr 24, 2018
87fff8b
dispose if present
akshita31 Apr 24, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
649 changes: 649 additions & 0 deletions package-lock.json

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
"https-proxy-agent": "^2.1.1",
"jsonc-parser": "^1.0.0",
"mkdirp": "^0.5.1",
"node-filter-async": "0.0.4",
"open": "*",
"request-light": "^0.2.0",
"rxjs": "^5.5.6",
Expand All @@ -80,6 +81,7 @@
"yauzl": "^2.5.0"
},
"devDependencies": {
"@types/archiver": "^2.1.1",
"@types/chai": "^4.1.2",
"@types/chai-arrays": "1.0.2",
"@types/chai-as-promised": "^7.1.0",
Expand All @@ -95,6 +97,7 @@
"@types/node": "^9.4.7",
"@types/semver": "^5.5.0",
"@types/tmp": "0.0.33",
"archiver": "^2.1.1",
"async-child-process": "^1.1.1",
"async-file": "^2.0.2",
"async-shelljs": "^0.1.2",
Expand All @@ -118,6 +121,7 @@
"minimist": "^1.2.0",
"mocha": "^5.0.4",
"mocha-typescript": "^1.1.12",
"mock-http-server": "^0.2.0",
"npm-run-all": "^4.1.2",
"nyc": "^11.6.0",
"plist": "^3.0.1",
Expand Down
48 changes: 21 additions & 27 deletions src/CSharpExtDownloader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,68 +4,62 @@
*--------------------------------------------------------------------------------------------*/

import * as util from './common';
import { GetNetworkConfiguration, defaultPackageManagerFactory, IPackageManagerFactory } from './downloader.helper';
import { PackageManager } from './packages';
import { PlatformInformation } from './platform';
import { PackageInstallation, LogPlatformInfo, InstallationSuccess, InstallationFailure } from './omnisharp/loggingEvents';
import { EventStream } from './EventStream';
import { vscode } from './vscodeAdapter';
import { DownloadAndInstallPackages } from './packageManager/PackageManager';
import { Package } from './packageManager/Package';
import { NetworkSettingsProvider } from './NetworkSettings';
import { ResolveFilePaths } from './packageManager/PackageFilePathResolver';

/*
* Class used to download the runtime dependencies of the C# Extension
*/
export class CSharpExtDownloader {
private proxy: string;
private strictSSL: boolean;
private packageManager: PackageManager;

public constructor(
vscode: vscode,
private provider: NetworkSettingsProvider,
private eventStream: EventStream,
private packageJSON: any,
private platformInfo: PlatformInformation,
packageManagerFactory: IPackageManagerFactory = defaultPackageManagerFactory) {

let networkConfiguration = GetNetworkConfiguration(vscode);
this.proxy = networkConfiguration.Proxy;
this.strictSSL = networkConfiguration.StrictSSL;
this.packageManager = packageManagerFactory(this.platformInfo, this.packageJSON);
private platformInfo: PlatformInformation) {
}

public async installRuntimeDependencies(): Promise<boolean> {
this.eventStream.post(new PackageInstallation("C# dependencies"));
let installationStage = 'touchBeginFile';
let success = false;
let installationStage = '';

try {
await util.touchInstallFile(util.InstallFileType.Begin);

await util.touchInstallFile(util.InstallFileType.Begin);
// Display platform information and RID
this.eventStream.post(new LogPlatformInfo(this.platformInfo));

installationStage = 'downloadPackages';
await this.packageManager.DownloadPackages(this.eventStream, this.proxy, this.strictSSL);

installationStage = 'installPackages';
await this.packageManager.InstallPackages(this.eventStream);

let runTimeDependencies = this.GetRunTimeDependenciesPackages();
runTimeDependencies.forEach(pkg => ResolveFilePaths(pkg));
installationStage = 'downloadAndInstallPackages';
await DownloadAndInstallPackages(runTimeDependencies, this.provider, this.platformInfo, this.eventStream);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since c#extdl'er assumes that DownloadAndInstallPacakges returning means the lockfile can be changed, maybe we should just move lockfile manipulation into the downloader component?

Copy link
Contributor Author

@akshita31 akshita31 Apr 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are planning to remove the Lockfile concept(in a subsequent PR) entirely and using the InstallTestPath to check whether the packages are installed or not.

installationStage = 'touchLockFile';
await util.touchInstallFile(util.InstallFileType.Lock);

success = true;
this.eventStream.post(new InstallationSuccess());
}
catch (error) {
this.eventStream.post(new InstallationFailure(installationStage, error));
}
finally {
// We do this step at the end so that we clean up the begin file in the case that we hit above catch block
// Attach a an empty catch to this so that errors here do not propogate
try {
util.deleteInstallFile(util.InstallFileType.Begin);
}
catch (error) { }
return success;
}
}
}

private GetRunTimeDependenciesPackages(): Package[] {
if (this.packageJSON.runtimeDependencies) {
return JSON.parse(JSON.stringify(<Package[]>this.packageJSON.runtimeDependencies));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand.
JSON.stringify turns an object into JSON. We then turn that JSON back into an object? What's wrong with the original packageJson.RunTimeDependencies object?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By this we are copying these packages by 'value' using this method

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the common practice to "deep copy" json object in js... it is awkward but it is what it is

}

return null;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the .foreach on line 38 do if we return null here? How would that even happen? User deleted/edited package.json?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes right. Changed it to throw an error instead. However I am not sure what the scenario would be.
We had this check in the packages.ts , hence I used the same here - https://github.com/OmniSharp/omnisharp-vscode/pull/2188/files#diff-b28d79ec3e8b65c75f9ee73da2cf52d4L73

Do you think we should remove the check ?

}
}
59 changes: 59 additions & 0 deletions src/CreateTmpAsset.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as tmp from 'tmp';
import { NestedError } from './packageManager/Package';
import { rimraf } from 'async-file';

export async function createTmpFile(): Promise<TmpAsset> {
const tmpFile = await new Promise<tmp.SynchrounousResult>((resolve, reject) => {
tmp.file({ prefix: 'package-' }, (err, path, fd, cleanupCallback) => {
if (err) {
return reject(new NestedError('Error from tmp.file', err));
}
if (fd == 0) {
return reject(new NestedError("Temporary package file unavailable"));
}

resolve(<tmp.SynchrounousResult>{ name: path, fd: fd, removeCallback: cleanupCallback });
});
});

return {
fd: tmpFile.fd,
name: tmpFile.name,
dispose: () => tmpFile.removeCallback()
};
}

export async function createTmpDir(unsafeCleanup: boolean): Promise<TmpAsset> {
const tmpDir = await new Promise<tmp.SynchrounousResult>((resolve, reject) => {
tmp.dir({ unsafeCleanup }, (err, path, cleanupCallback) => {
if (err) {
return reject(new NestedError('Error from tmp.dir', err));
}

resolve(<tmp.SynchrounousResult>{ name: path, removeCallback: cleanupCallback });
});
});

return {
fd: tmpDir.fd,
name: tmpDir.name,
dispose: () => {
if (unsafeCleanup) {
rimraf(tmpDir.name);//to delete directories that have folders inside them
}
else {
tmpDir.removeCallback();
}
}
};
}

export interface TmpAsset {
fd: number;
name: string;
dispose: () => void;
}
25 changes: 25 additions & 0 deletions src/NetworkSettings.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/

import { vscode } from "./vscodeAdapter";

export default class NetworkSettings {
constructor(public readonly proxy: string, public readonly strictSSL: boolean) {
}
}

export interface NetworkSettingsProvider {
//to do: make this async
(): NetworkSettings;
}

export function vscodeNetworkSettingsProvider(vscode: vscode): NetworkSettingsProvider {
return () => {
const config = vscode.workspace.getConfiguration();
const proxy = config.get<string>('http.proxy');
const strictSSL = config.get('http.proxyStrictSSL', true);
return new NetworkSettings(proxy, strictSSL);
};
}
19 changes: 0 additions & 19 deletions src/downloader.helper.ts

This file was deleted.

12 changes: 7 additions & 5 deletions src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import TelemetryReporter from 'vscode-extension-telemetry';
import { addJSONProviders } from './features/json/jsonContributions';
import { ProjectStatusBarObserver } from './observers/ProjectStatusBarObserver';
import CSharpExtensionExports from './CSharpExtensionExports';
import { vscodeNetworkSettingsProvider, NetworkSettingsProvider } from './NetworkSettings';

export async function activate(context: vscode.ExtensionContext): Promise<CSharpExtensionExports> {

Expand Down Expand Up @@ -90,10 +91,11 @@ export async function activate(context: vscode.ExtensionContext): Promise<CSharp
let telemetryObserver = new TelemetryObserver(platformInfo, () => reporter);
eventStream.subscribe(telemetryObserver.post);

let runtimeDependenciesExist = await ensureRuntimeDependencies(extension, eventStream, platformInfo);

let networkSettingsProvider = vscodeNetworkSettingsProvider(vscode);
let runtimeDependenciesExist = await ensureRuntimeDependencies(extension, eventStream, platformInfo, networkSettingsProvider);

// activate language services
let omniSharpPromise = OmniSharp.activate(context, eventStream, extension.packageJSON, platformInfo);
let omniSharpPromise = OmniSharp.activate(context, eventStream, extension.packageJSON, platformInfo, networkSettingsProvider);

// register JSON completion & hover providers for project.json
context.subscriptions.push(addJSONProviders());
Expand All @@ -116,11 +118,11 @@ export async function activate(context: vscode.ExtensionContext): Promise<CSharp
};
}

async function ensureRuntimeDependencies(extension: vscode.Extension<CSharpExtensionExports>, eventStream: EventStream, platformInfo: PlatformInformation): Promise<boolean> {
async function ensureRuntimeDependencies(extension: vscode.Extension<CSharpExtensionExports>, eventStream: EventStream, platformInfo: PlatformInformation, networkSettingsProvider : NetworkSettingsProvider): Promise<boolean> {
return util.installFileExists(util.InstallFileType.Lock)
.then(exists => {
if (!exists) {
const downloader = new CSharpExtDownloader(vscode, eventStream, extension.packageJSON, platformInfo);
const downloader = new CSharpExtDownloader(networkSettingsProvider, eventStream, extension.packageJSON, platformInfo);
return downloader.installRuntimeDependencies();
} else {
return true;
Expand Down
2 changes: 1 addition & 1 deletion src/observers/CsharpLoggerObserver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/

import { PackageError } from "../packages";
import { BaseLoggerObserver } from "./BaseLoggerObserver";
import * as Event from "../omnisharp/loggingEvents";
import { PackageError } from "../packageManager/Package";
export class CsharpLoggerObserver extends BaseLoggerObserver {
private dots: number;

Expand Down
2 changes: 1 addition & 1 deletion src/observers/TelemetryObserver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/

import { PackageError } from "../packages";
import { PlatformInformation } from "../platform";
import { BaseEvent, PackageInstallation, InstallationFailure, InstallationSuccess, OmnisharpDelayTrackerEventMeasures, OmnisharpStart, TestExecutionCountReport, TelemetryEventWithMeasures } from "../omnisharp/loggingEvents";
import { PackageError } from "../packageManager/Package";

export interface ITelemetryReporter {
sendTelemetryEvent(eventName: string, properties?: { [key: string]: string }, measures?: { [key: string]: number }): void;
Expand Down
59 changes: 26 additions & 33 deletions src/omnisharp/OmnisharpDownloader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,24 @@
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/

import { GetNetworkConfiguration, IPackageManagerFactory, defaultPackageManagerFactory } from '../downloader.helper';
import { GetPackagesFromVersion, GetVersionFilePackage } from './OmnisharpPackageCreator';
import { Package, PackageManager } from '../packages';
import * as fs from 'fs';
import { GetPackagesFromVersion } from './OmnisharpPackageCreator';
import { PlatformInformation } from '../platform';
import { PackageInstallation, LogPlatformInfo, InstallationSuccess, InstallationFailure, InstallationProgress } from './loggingEvents';
import { EventStream } from '../EventStream';
import { vscode } from '../vscodeAdapter';

import { NetworkSettingsProvider } from '../NetworkSettings';
import { DownloadAndInstallPackages } from '../packageManager/PackageManager';
import { createTmpFile, TmpAsset } from '../CreateTmpAsset';
import { DownloadFile } from '../packageManager/FileDownloader';
import { ResolveFilePaths } from '../packageManager/PackageFilePathResolver';

export class OmnisharpDownloader {
private proxy: string;
private strictSSL: boolean;
private packageManager: PackageManager;

public constructor(
vscode: vscode,
private provider: NetworkSettingsProvider,
private eventStream: EventStream,
private packageJSON: any,
private platformInfo: PlatformInformation,
packageManagerFactory: IPackageManagerFactory = defaultPackageManagerFactory) {

let networkConfiguration = GetNetworkConfiguration(vscode);
this.proxy = networkConfiguration.Proxy;
this.strictSSL = networkConfiguration.StrictSSL;
this.packageManager = packageManagerFactory(this.platformInfo, this.packageJSON);
private platformInfo: PlatformInformation) {
}

public async DownloadAndInstallOmnisharp(version: string, serverUrl: string, installPath: string) {
Expand All @@ -36,18 +29,11 @@ export class OmnisharpDownloader {

try {
this.eventStream.post(new LogPlatformInfo(this.platformInfo));

installationStage = 'getPackageInfo';
let packages: Package[] = GetPackagesFromVersion(version, this.packageJSON.runtimeDependencies, serverUrl, installPath);

installationStage = 'downloadPackages';
// Specify the packages that the package manager needs to download
this.packageManager.SetVersionPackagesForDownload(packages);
await this.packageManager.DownloadPackages(this.eventStream, this.proxy, this.strictSSL);

installationStage = 'installPackages';
await this.packageManager.InstallPackages(this.eventStream);

let packages = GetPackagesFromVersion(version, this.packageJSON.runtimeDependencies, serverUrl, installPath);
packages.forEach(pkg => ResolveFilePaths(pkg));
installationStage = 'downloadAndInstallPackages';
await DownloadAndInstallPackages(packages, this.provider, this.platformInfo, this.eventStream);
this.eventStream.post(new InstallationSuccess());
}
catch (error) {
Expand All @@ -56,18 +42,25 @@ export class OmnisharpDownloader {
}
}

public async GetLatestVersion(serverUrl: string, latestVersionFileServerPath: string): Promise<string> {
public async GetLatestVersion(serverUrl: string, latestVersionFileServerPath: string) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why did we remove the return type annotation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed by mistake. Thanks for pointing it out.

let installationStage = 'getLatestVersionInfoFile';
let description = "Latest Omnisharp Version Information";
let url = `${serverUrl}/${latestVersionFileServerPath}`;
let tmpFile: TmpAsset;
try {
this.eventStream.post(new InstallationProgress(installationStage, 'Getting latest build information...'));
//The package manager needs a package format to download, hence we form a package for the latest version file
let filePackage = GetVersionFilePackage(serverUrl, latestVersionFileServerPath);
//Fetch the latest version information from the file
return await this.packageManager.GetLatestVersionFromFile(this.eventStream, this.proxy, this.strictSSL, filePackage);
tmpFile = await createTmpFile();
await DownloadFile(tmpFile.fd, description, url, "", this.eventStream, this.provider);
return fs.readFileSync(tmpFile.name, 'utf8');
}
catch (error) {
this.eventStream.post(new InstallationFailure(installationStage, error));
throw error;
}
finally {
if (tmpFile) {
tmpFile.dispose();
}
}
}
}
}
Loading