Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add OnnxTransform for scoring Onnx 1.2 models - integrates Microsoft.ML.Scoring/Sonoma Library #942

Merged
merged 32 commits into from
Sep 26, 2018
Merged

Add OnnxTransform for scoring Onnx 1.2 models - integrates Microsoft.ML.Scoring/Sonoma Library #942

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
d67eb57
initial checkin
jignparm Sep 18, 2018
1402d19
added 1 file
jignparm Sep 19, 2018
d7aefce
Added nugets and project so compile successfully
jignparm Sep 19, 2018
b5482df
Added unit tests -- need further debugging
jignparm Sep 21, 2018
40f857e
Merge branch 'master' into jignparm/onnxtransform
jignparm Sep 21, 2018
46d7821
Fixed schema propagation tests
jignparm Sep 24, 2018
4bf00de
Merge remote-tracking branch 'upstream/master' into jignesh/onnx
Sep 24, 2018
dd956be
Merge branch 'jignparm/onnxtransform' of https://github.com/jignparm/…
jignparm Sep 24, 2018
e61b730
Cleaned up inconsistent multiple/single IO code. Now OnnxTransform is…
Sep 24, 2018
6fdd04e
Some minor fixes
jignparm Sep 24, 2018
e597068
Merge branch 'jignparm/onnxtransform' of https://github.com/jignparm/…
jignparm Sep 24, 2018
fd27eec
Updated tests
Sep 24, 2018
2778577
moved tests to different project to prevent tensorflow.dll from crashing
jignparm Sep 25, 2018
27de2b0
unit tests passing
jignparm Sep 25, 2018
4d94c43
relaxed range differences
jignparm Sep 25, 2018
84334ed
Made factory create() methods private
Sep 25, 2018
4b958b1
Addressed some PR comments
Sep 25, 2018
7ff3012
Moved IdvToTensorAdapter to OnnxUtils
Sep 25, 2018
d47516e
Fixed blanks line
Sep 25, 2018
51b3c42
Merge branch 'master' into jignparm/onnxtransform
jignparm Sep 25, 2018
b9af85e
fixed merge conflicts
jignparm Sep 25, 2018
8c880f0
Small clean up
Sep 25, 2018
d554822
added PR comments
jignparm Sep 26, 2018
3069de6
Fixed PR code reviews
jignparm Sep 26, 2018
ca61e26
more PR comments
jignparm Sep 26, 2018
d95e323
Addressed Ivans PR comments
jignparm Sep 26, 2018
24d34b4
added back onnx proj, removed onnxtransform proj
jignparm Sep 26, 2018
0182ca1
small edit to kick off build
jignparm Sep 26, 2018
cc14268
doc changes
jignparm Sep 26, 2018
aa17a35
fixed gt to gt;
jignparm Sep 26, 2018
4c7821f
small change to kick off build
jignparm Sep 26, 2018
316a5a1
create IDV from array instead of list of array
jignparm Sep 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion Microsoft.ML.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,11 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.Analyzer", "sr
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.StaticPipelineTesting", "test\Microsoft.ML.StaticPipelineTesting\Microsoft.ML.StaticPipelineTesting.csproj", "{8B38BF24-35F4-4787-A9C5-22D35987106E}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Microsoft.ML.DnnAnalyzer", "src\Microsoft.ML.DnnAnalyzer\Microsoft.ML.DnnAnalyzer\Microsoft.ML.DnnAnalyzer.csproj", "{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.OnnxTransform", "src\Microsoft.ML.OnnxTransform\Microsoft.ML.OnnxTransform.csproj", "{8C05642D-C3AA-4972-B02C-93681161A6BC}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.DnnAnalyzer", "src\Microsoft.ML.DnnAnalyzer\Microsoft.ML.DnnAnalyzer\Microsoft.ML.DnnAnalyzer.csproj", "{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Microsoft.ML.OnnxTransformTest", "test\Microsoft.ML.OnnxTransformTest\Microsoft.ML.OnnxTransformTest.csproj", "{49D03292-8AFE-4B82-823C-D047BF8420F7}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down Expand Up @@ -421,6 +425,14 @@ Global
{8B38BF24-35F4-4787-A9C5-22D35987106E}.Release|Any CPU.Build.0 = Release|Any CPU
{8B38BF24-35F4-4787-A9C5-22D35987106E}.Release-Intrinsics|Any CPU.ActiveCfg = Release|Any CPU
{8B38BF24-35F4-4787-A9C5-22D35987106E}.Release-Intrinsics|Any CPU.Build.0 = Release|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Debug-Intrinsics|Any CPU.ActiveCfg = Debug|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Debug-Intrinsics|Any CPU.Build.0 = Debug|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Release|Any CPU.Build.0 = Release|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Release-Intrinsics|Any CPU.ActiveCfg = Release|Any CPU
{8C05642D-C3AA-4972-B02C-93681161A6BC}.Release-Intrinsics|Any CPU.Build.0 = Release|Any CPU
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Debug|Any CPU.Build.0 = Debug|Any CPU
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Debug-Intrinsics|Any CPU.ActiveCfg = Debug|Any CPU
Expand All @@ -429,6 +441,14 @@ Global
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Release|Any CPU.Build.0 = Release|Any CPU
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Release-Intrinsics|Any CPU.ActiveCfg = Release|Any CPU
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA}.Release-Intrinsics|Any CPU.Build.0 = Release|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Debug|Any CPU.Build.0 = Debug|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Debug-Intrinsics|Any CPU.ActiveCfg = Debug|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Debug-Intrinsics|Any CPU.Build.0 = Debug|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Release|Any CPU.ActiveCfg = Release|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Release|Any CPU.Build.0 = Release|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Release-Intrinsics|Any CPU.ActiveCfg = Release|Any CPU
{49D03292-8AFE-4B82-823C-D047BF8420F7}.Release-Intrinsics|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -476,7 +496,9 @@ Global
{570A0B8A-5463-44D2-8521-54C0CA4CACA9} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
{6DEF0F40-3853-47B3-8165-5F24BA5E14DF} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
{8B38BF24-35F4-4787-A9C5-22D35987106E} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
{8C05642D-C3AA-4972-B02C-93681161A6BC} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
{73DAAC82-D308-48CC-8FFE-3B037F8BBCCA} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
{49D03292-8AFE-4B82-823C-D047BF8420F7} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {41165AF1-35BB-4832-A189-73060F82B01D}
Expand Down
1 change: 1 addition & 0 deletions build/Dependencies.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
<MicrosoftCodeAnalysisCSharpVersion>2.9.0</MicrosoftCodeAnalysisCSharpVersion>
<MicrosoftCSharpVersion>4.5.0</MicrosoftCSharpVersion>
<SystemCompositionVersion>1.2.0</SystemCompositionVersion>
<MicrosoftMLScoring>1.0.4-dev47509</MicrosoftMLScoring>
<SystemIOFileSystemAccessControl>4.5.0</SystemIOFileSystemAccessControl>
<SystemSecurityPrincipalWindows>4.5.0</SystemSecurityPrincipalWindows>
</PropertyGroup>
Expand Down
13 changes: 13 additions & 0 deletions pkg/Microsoft.ML.OnnxTransform/Microsoft.ML.OnnxTransform.nupkgproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk" DefaultTargets="Pack">

<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<PackageDescription>ML.NET component for Sonoma</PackageDescription>
Copy link
Contributor

@TomFinley TomFinley Sep 25, 2018 •
edited by jignparm
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sonoma [](start = 45, length = 6)

Is Sonoma a public name? #Resolved

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with microsoft.ml.scoring

In reply to: 220280263 [](ancestors = 220280263)

</PropertyGroup>
Copy link
Member

@eerhardt eerhardt Sep 25, 2018 •
edited by jignparm
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in the standup meeting, we want to keep Microsoft.ML.OnnxTransform from being a stable package. A way to do that is to add this property here:

<VersionSuffix Condition="'$(StabilizePackageVersion)' == 'true'">$(PreReleaseLabel)</VersionSuffix>

Which will ensure that the Version will still have -preview in it. #Resolved

Copy link
Member

@eerhardt eerhardt Sep 25, 2018 •
edited by jignparm
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See

machinelearning/Directory.Build.props

Lines 61 to 66 in 9258be2

<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">false</StabilizePackageVersion>
<IncludeBuildNumberInPackageVersion Condition="'$(StabilizePackageVersion)' != 'true'">true</IncludeBuildNumberInPackageVersion>
<VersionSuffix></VersionSuffix>
<VersionSuffix Condition="'$(StabilizePackageVersion)' != 'true'">$(PreReleaseLabel)</VersionSuffix>
<VersionSuffix Condition="'$(IncludeBuildNumberInPackageVersion)' == 'true'">$(VersionSuffix)-$(BuildNumberMajor)-$(BuildNumberMinor)</VersionSuffix>
for how this is determined in the root props file. When we want to build stable packages for release, we set $(StabilizePackageVersion) to true for the build. #Resolved

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was another discussion post this mornings call.

The plan is to use this dev nuget for the bug-bash, and assuming bugs are fixed by Thursday and sonoma goes live before tuesday, we'll switch to point to the stable release of the nuget.

build/Dependencies.props: 1.0.4-dev48825

In reply to: 220290306 [](ancestors = 220290306)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was another discussion post this mornings call.

The plan is to use this dev nuget for the bug-bash, and assuming bugs are fixed by Thursday and sonoma goes live before tuesday, we'll switch to point to the stable release of the nuget.

build/Dependencies.props: 1.0.4-dev48825

In reply to: 220290881 [](ancestors = 220290881)


<ItemGroup>
<ProjectReference Include="../Microsoft.ML/Microsoft.ML.nupkgproj" />
<PackageReference Include="Microsoft.ML.Scoring" Version="$(MicrosoftMLScoring)"/>
</ItemGroup>

</Project>
5 changes: 5 additions & 0 deletions pkg/Microsoft.ML.OnnxTransform/Microsoft.ML.OnnxTransform.symbols.nupkgproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
<Project DefaultTargets="Pack">

<Import Project="Microsoft.ML.OnnxTransform.nupkgproj" />

</Project>
22 changes: 22 additions & 0 deletions src/Microsoft.ML.OnnxTransform/Microsoft.ML.OnnxTransform.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<IncludeInPackage>Microsoft.ML.OnnxTransform</IncludeInPackage>
<DefineConstants>CORECLR</DefineConstants>
Copy link
Contributor

@Ivanidzo4ka Ivanidzo4ka Sep 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is unnecessary #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

In reply to: 220400768 [](ancestors = 220400768)

<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\Microsoft.ML.Core\Microsoft.ML.Core.csproj" />
<ProjectReference Include="..\Microsoft.ML.Data\Microsoft.ML.Data.csproj" />
<!--ProjectReference Include="..\Microsoft.ML.ImageAnalytics\Microsoft.ML.ImageAnalytics.csproj" /-->
Copy link
Contributor

@TomFinley TomFinley Sep 25, 2018 •
edited by jignparm
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[](start = 4, length = 101)

Is it possible to just clean things up like this? #Resolved

Copy link
Contributor

@Ivanidzo4ka Ivanidzo4ka Sep 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[](start = 3, length = 102)

Delete it. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

In reply to: 220397176 [](ancestors = 220397176)

<PackageReference Include="Microsoft.ML.Scoring" Version="$(MicrosoftMLScoring)" />
</ItemGroup>


<ItemGroup>
<Service Include="{B5B31292-0FED-4ACC-9879-329728881CF3}" />
</ItemGroup>
jignparm marked this conversation as resolved.
Show resolved Hide resolved

</Project>
Loading