feat: support direct pip install #1223

serena-ruan · 2021-10-21T09:44:33Z

No description provided.

serena-ruan · 2021-10-21T09:47:06Z

/azp run

azure-pipelines · 2021-10-21T09:47:17Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov-commenter · 2021-10-21T09:53:57Z

Codecov Report

Merging #1223 (2daed28) into master (ea91189) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1223   +/-   ##
=======================================
  Coverage   83.69%   83.69%           
=======================================
  Files         264      264           
  Lines       12911    12911           
  Branches      637      637           
=======================================
  Hits        10806    10806           
  Misses       2105     2105

Impacted Files	Coverage Δ
...m/microsoft/azure/synapse/ml/codegen/CodeGen.scala	`98.36% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2771853...2daed28. Read the comment docs.

mhamilton723

Youre just blasting through these tasks, true MVP status. left a few minor comments

mhamilton723 · 2021-10-21T16:40:18Z

pipeline.yaml

+    - bash: echo "##vso[task.prependpath]$CONDA/bin"
+      condition: startsWith(variables['tag'], 'v')
+      displayName: Add conda to PATH
+    - bash: conda info


Do we need this conda info part?

Nope, it just displays the conda information, like all other jobs so I kept it.

mhamilton723 · 2021-10-21T16:42:50Z

project/Secrets.scala

@@ -59,5 +59,6 @@ object Secrets {
    sys.env.getOrElse("PGP-PRIVATE", getSecret("pgp-private")).getBytes("UTF-8")))
  lazy val pgpPassword: String = sys.env.getOrElse("PGP-PW", getSecret("pgp-pw"))
  lazy val storageKey: String = sys.env.getOrElse("STORAGE_KEY", getSecret("storage-key"))
+  lazy val pypiAPIToken: String = sys.env.getOrElse("PYPI-API-TOKEN", getSecret("pypi-api-token"))


might be good to use this var in the build sbt step so no env vars are needed if properly authenticated

serena-ruan · 2021-10-22T08:56:54Z

/azp run

azure-pipelines · 2021-10-22T08:57:04Z

Azure Pipelines successfully started running 1 pipeline(s).

…nto serena/supportPip

serena-ruan · 2021-10-25T06:17:47Z

/azp run

azure-pipelines · 2021-10-25T06:17:58Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T06:39:30Z

/azp run

azure-pipelines · 2021-10-25T06:39:43Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T06:58:50Z

/azp run

azure-pipelines · 2021-10-25T06:59:02Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T07:15:02Z

/azp run

azure-pipelines · 2021-10-25T07:15:15Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T07:37:33Z

/azp run

serena-ruan · 2021-10-25T08:14:40Z

/azp run

azure-pipelines · 2021-10-25T08:14:51Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T08:45:26Z

/azp run

azure-pipelines · 2021-10-25T08:45:37Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T09:02:16Z

/azp run

azure-pipelines · 2021-10-25T09:02:26Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T09:22:19Z

/azp run

azure-pipelines · 2021-10-25T09:22:33Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-25T09:51:40Z

/azp run

azure-pipelines · 2021-10-25T09:51:50Z

Azure Pipelines successfully started running 1 pipeline(s).

mhamilton723

Content looks great, just a nit about replicated code

mhamilton723 · 2021-10-25T16:29:31Z

build.sbt

+  writeSetupFileToTarget(dir)
+  runCmd(
+    activateCondaEnv.value ++
+      Seq(s"python", "setup.py", "bdist_wheel", "--universal", "-d", packageDir),


A lot of the code added is replicated in the project/codegen. Perhaps we can abstract this into functions and call twice?

mhamilton723 · 2021-10-25T16:34:09Z

build.sbt

+
+val publishPypi = TaskKey[Unit]("publishPypi", "publish synapseml python wheel to pypi")
+publishPypi := {
+  packageSynapseML.value


some duplicated version getting code here.

SynapseML/project/CodegenPlugin.scala

Line 38 in bbd8744

val pythonizedVersion = settingKey[String]("Pythonized version")

Might want to put the implementation code in an object and use it in all these diff places

serena-ruan · 2021-10-26T04:00:51Z

/azp run

azure-pipelines · 2021-10-26T04:01:03Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2021-10-26T04:44:27Z

/azp run

azure-pipelines · 2021-10-26T04:44:39Z

Azure Pipelines successfully started running 1 pipeline(s).

mhamilton723 · 2021-10-26T22:28:10Z

/azp run

azure-pipelines · 2021-10-26T22:28:22Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan requested a review from mhamilton723 as a code owner October 21, 2021 09:44

feat: add publish python packages to pypi task

c706c2d

serena-ruan force-pushed the serena/supportPip branch from d205de8 to c706c2d Compare October 21, 2021 09:46

serena-ruan changed the title ~~support direct pip install~~ feat: support direct pip install Oct 21, 2021

mhamilton723 requested changes Oct 21, 2021

View reviewed changes

serena-ruan and others added 3 commits October 22, 2021 16:53

add task to pack all projects into one wheel file and test in testpypi

839c938

address comments

ca7a20a

Merge branch 'master' into serena/supportPip

addc69c

serena-ruan and others added 3 commits October 22, 2021 17:28

Merge branch 'master' into serena/supportPip

4bacc30

fix sbt task execution order

24118e1

Merge branch 'serena/supportPip' of github.com:serena-ruan/mmlspark i…

e915225

…nto serena/supportPip

fix job env variable

86282eb

test

6b502d0

fix env variable name

8263029

serena-ruan added 2 commits October 25, 2021 15:35

add twineAuthenticate task

27af520

update task

8bd7071

add pypiApiToken into Secrets -- this is needed to fetch it in sbt

17b683b

test pipeline job

62c38dd

fixing..

fc8243b

update pipeline

da87dff

mhamilton723 requested changes Oct 25, 2021

View reviewed changes

serena-ruan added 2 commits October 26, 2021 11:11

address comments

1ba91f6

factor out packagePythonWheel and pyVersion

5ddd2f8

fix .value usage error

fea82e2

mhamilton723 approved these changes Oct 26, 2021

View reviewed changes

Merge branch 'master' into serena/supportPip

2daed28

mhamilton723 merged commit 3d92dd7 into microsoft:master Oct 27, 2021

serena-ruan deleted the serena/supportPip branch November 9, 2021 05:59

TomFinley mentioned this pull request Dec 20, 2022

Where can I get the mmlspark python whl file？ #1768

Open

feat: support direct pip install #1223

feat: support direct pip install #1223

Conversation

serena-ruan commented Oct 21, 2021

serena-ruan commented Oct 21, 2021

azure-pipelines bot commented Oct 21, 2021

codecov-commenter commented Oct 21, 2021 • edited Loading

Codecov Report

mhamilton723 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serena-ruan commented Oct 22, 2021

azure-pipelines bot commented Oct 22, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

serena-ruan commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

mhamilton723 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serena-ruan commented Oct 26, 2021

azure-pipelines bot commented Oct 26, 2021

serena-ruan commented Oct 26, 2021

azure-pipelines bot commented Oct 26, 2021

mhamilton723 commented Oct 26, 2021

azure-pipelines bot commented Oct 26, 2021

codecov-commenter commented Oct 21, 2021 •

edited

Loading