Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build issue - createVirtualEnvironment -> pygradleBootstrap #351

Open
mjothiram opened this issue Aug 18, 2021 · 8 comments
Open

build issue - createVirtualEnvironment -> pygradleBootstrap #351

mjothiram opened this issue Aug 18, 2021 · 8 comments

Comments

@mjothiram
Copy link

I see this issue in my build from this morning. And trying to see this PyPi in LinkedIn Artifactory, but could not see that. Is it removed or any issue?

Thanks

Execution failed for task ':airflow:createVirtualEnvironment'.

Could not resolve all files for configuration ':airflow:pygradleBootstrap'.
Could not resolve pypi:virtualenv:16.0.0.
Required by:
project :airflow
> Could not resolve pypi:virtualenv:16.0.0.
> Could not get resource 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'.
> Could not GET 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'. Received status code 401 from server: Unauthorized

@rafishaiks03
Copy link

Hi

Can anyone please address the above issue, iam also facing the same.

I'm not able to get a solution....

Thanks in Advance..!!
image

@sudhir418
Copy link

Even we are facing the same issue , any alternatives?

@byblakeorriver
Copy link

Same here.

@rafishaiks03
Copy link

I'm still facing the same issue and didn't find any alternatives also..
still looking for help.. if not dropping Pygradle is the only option left...

@griffinptg
Copy link

griffinptg commented Aug 19, 2021

I was able to resolve by downloading dependencies from pypi and uploading to my company's Artifactory. I had to make an .ivy file for each. It was a pain!

For example:
Download the .tar.gz here: https://pypi.org/project/virtualenv/16.0.0/
Upload that to your own Artifactory (assuming you have one) with an ivy file, virtualenv-16.0.0.ivy:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="virtualenv" revision="16.0.0" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="virtualenv" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default" />
</ivy-module>

There are a bunch more after this FYI. And if you get a missing Python import instead of the 401, do the same, but also add it as a dependency to the lib that needs it, for example:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="requests" revision="2.18.4" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="requests" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default">
    <dependency org="pypi" name="urllib3" rev="1.26.6" conf="default" />
    <dependency org="pypi" name="chardet" rev="4.0.0" conf="default" />
    <dependency org="pypi" name="certifi" rev="2021.5.30" conf="default" />
    <dependency org="pypi" name="idna" rev="3.2" conf="default" />
  </dependencies>
</ivy-module>

Just posting this as a workaround. It would still be good to see a fix for this.

@rafishaiks03
Copy link

Hi @griffinmm,

It seems you have posted in maven dependency format could you please help me out how and where to add the above code in gradle build. May be you can share your build.gradle file with the above things.

Thanks in Advance...!

@griffinptg
Copy link

@rafishaiks03 Here's my build.gradle, sanitized a bit:

buildscript {
    repositories {
        ivy {
            url 'http://artifactory.yourdomain.com/artifactory/plugins-release'
            credentials {
                username = "${artifactory_user}"
                password = "${artifactory_password}"
            }
        }
    }
    dependencies {
        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.15.1"
    }
}

plugins {
    id 'com.linkedin.python' version '0.9.11'
}

allprojects {
    apply plugin: "com.jfrog.artifactory"
}

artifactory {
    contextUrl = "https://artifactory.yourdomain.com/artifactory"
    resolve {
        repository {
            repoKey = 'pypi-local'
            username = "${artifactory_user}"
            password = "${artifactory_password}"
            maven = false
            ivy {
                ivyLayout =  "[organisation]/[module]/[revision]/[module]-[revision].ivy"  
                artifactLayout =  "[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
                mavenCompatible = false
            }
        }
    }
}

python {
   testDir = project.file('integration-test').path
   
   pythonEnvironment = loadEnvFile()

   details {
       pythonVersion = '3.6'
   }

   forceVersion('pypi', 'flake8', '3.2.1')
}

repositories {
   pyGradlePyPi()
}

pytest {
  additionalArguments = ["--junitxml=test-results/results.xml"]
}

dependencies {
  setupRequires 'pypi:jedi:0.11.1'
  setupRequires 'pypi:autopep8:1.3.3'
  setupRequires 'pypi:rope:0.10.7'
  setupRequires 'pypi:yapf:0.20.0'

  test 'pypi:requests:2.18.4'
  test('pypi:boto3:1.5.18') {
    exclude module: 'docutils'
  }
}

def loadEnvFile() {
  def m = [:]
  if(project.hasProperty('configFile')) {
    file('integration-test/config/' + configFile).eachLine { line ->
      def tokens = line.tokenize('=')
      println "Setting property: " + tokens
      m.put(tokens[0], tokens[1])
    }
  }
  return m
}

@alghoshal
Copy link

alghoshal commented Aug 19, 2024

For anyone else facing these pygradle issue, been able to build on Python-3.8. Like @griffinmm had to download modules from pypi, with some differences:

  • Downloaded them using the pivy-importer (see: https://github.com/linkedin/pygradle.git), which also prepares the ivy files automatically
  • Downloaded modules to a local folder (specified in build.gradle) & used that as a local repository in place of Artifactory.
  • Had to fix issues with name, case, dependencies, etc in some of modules (see Script-2 below: fixDownloadedIvyModules.sh)

Details shared here:
Project: https://github.com/alghoshal/pygradle_python3_example/blob/main/README.md

Build file: https://github.com/alghoshal/pygradle_python3_example/blob/main/build.gradle

Script-1 (Downloader): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/downloadPyGradleDependecies.sh

Script-2 (Fixes name, etc): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/fixDownloadedIvyModules.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants