Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: add debug command for creating envs #3237

Merged
merged 93 commits into from
Nov 29, 2018
Merged

Conversation

msarahan
Copy link
Contributor

@msarahan msarahan commented Oct 28, 2018

The purpose of this feature is to aid in creating environments and activation scripts so that users can quickly create an environment just like conda-build creates for build/hot or test. This feature creates the envs, splits the env activation stuff into a separate script that can be sourced without running a build, and outputs a line that can be copy-pasted to get into an activated shell that works the same as a build inside conda-build.

examples:

# create build/host envs and activation script
conda-debug my-test-recipe
# create test env and activation script
conda-debug /home/msarahan/miniconda/conda-bld/linux-64/somepkg-1.0-0.tar.bz2

Note that you have to have the dash between conda and debug. That's because conda has those executables hard-coded, and conda debug may not work until we add an entry for it in conda.

fixes #1751

@msarahan msarahan force-pushed the add_debug_cmd branch 2 times, most recently from 46130eb to d6961f8 Compare November 13, 2018 19:16
@msarahan msarahan force-pushed the add_debug_cmd branch 2 times, most recently from 4497282 to f56bea0 Compare November 14, 2018 13:55
@msarahan msarahan force-pushed the add_debug_cmd branch 2 times, most recently from 1320243 to 9601554 Compare November 14, 2018 18:11
@soapy1
Copy link
Contributor

soapy1 commented Nov 14, 2018

I haven't tested out everything yet but here are my initial notes:

  • Initially had trouble getting it working - was my fault, didn’t have lib archive installed

  • Tried debugging a package I had locally

    • conda-debug ~/miniconda3/pkgs/alabaster-0.7.11-py35_0.tar.bz2
  • Did the step where you go into the environment

    • cd /Users/scastellarin/miniconda3/conda-bld/debug_1542233048885/test_tmp && source /Users/scastellarin/miniconda3/conda-bld/debug_1542233048885/test_tmp/conda_test_env_vars.sh
    • Ran the test with ./run_test.sh and this killed my shell. Was somewhat unexpected. I guess the intended way to run it is bash run_test.sh, that works fine
  • Tried debugging a recipe I had locally

    • conda-debug gdata-feedstock
  • The first time I ran it it was good

  • Ran it with —test flag and it killed my shell

  • Ran it again with —test flag and it gave me an error Debugging for test mode is only supported for package files that already exist. Please build your package first, then use it to create the debugging environment.

  • Tried it with —path that did and did not exist

Questions:

  • the provided debug directory has 3 different testing files that seem to do the same thing. Eg. conda_test_runner.sh, run_test.py, run_test.sh. What is the purpose of having this duplication?
  • Is there a way to purge the environment when the user is done?
  • Do you think it would be beneficial if using condo-debug on a recipe also gave the user test runners

@msarahan
Copy link
Contributor Author

msarahan commented Nov 15, 2018

hmm... yes, this is a little rough. Thanks for poking at it!

  1. I removed the --test flag completely. I originally thought that you could create build/host envs from existing packages. With recipes that use outputs, that's a terrible idea. Such recipes can't have the source section in them, so creating those envs isn't going to be all that useful. Instead, when you pass a recipe, you get build/host envs and the work dir. When you pass a path to an existing package, you get the test env and the test_tmp dir.

the provided debug directory has 3 different testing files that seem to do the same thing. Eg. conda_test_runner.sh, run_test.py, run_test.sh. What is the purpose of having this duplication?

Is the content of those files all the same? I think they are different.

  • conda_test_runner.sh: this one actually runs the test. This is the one that conda-build calls when it runs the tests. It then calls any run_test.* scripts.
  • run_test.sh: this can be manually created as part of the recipe, or might be created if the test section contains "commands" entries. It is not called directly - only via conda_test_runner.sh. See https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#test-section
  • conda_test_env_vars.sh this is a new file as of this PR. All of this used to be lumped into conda_test_runner.sh. The purpose of splitting it out is that you can set up a shell env just like the one inside the conda-build call.

It's a bug if any of these are the same as any other.

Is there a way to purge the environment when the user is done?

Good point. I'll have to think about this. I think conda-build purge will clean them up as long as --path is not specified, but I need to check. If --path is used, then it's all under user control and I don't want to touch it.

Do you think it would be beneficial if using condo-debug on a recipe also gave the user test runners

Sure, perhaps as additional text shown at the end. I'm definitely assuming some expert-level users here, but making the tool more friendly is never a bad thing.

@msarahan msarahan force-pushed the add_debug_cmd branch 2 times, most recently from 835c45a to ef1d3f9 Compare November 15, 2018 14:45
@msarahan
Copy link
Contributor Author

Is there a way to purge the environment when the user is done?

Verified that conda build purge does clean these up

Do you think it would be beneficial if using condo-debug on a recipe also gave the user test runners

Added text like:

################################################################################
Test environment created for debugging.  To enter a debugging environment:

cd /Users/msarahan/miniconda3/conda-bld/conda-build_1542302975704/work && source /Users/msarahan/miniconda3/conda-bld/conda-build_1542302975704/work/build_env_setup.sh

To run your tests, you might want to start with running the conda_test_runner.sh file.
################################################################################

after the path activation command printout.

@soapy1
Copy link
Contributor

soapy1 commented Nov 15, 2018

Did a bit more testing on this:

  • Installed locally creates package

    • conda-debug ~/miniconda3/pkgs/alabaster-0.7.11-py35_0.tar.bz2
  • Tried using -a flag

    • conda-debug ~/miniconda3/pkgs/alabaster-0.7.11-py35_0.tar.bz2 -a
  • Nice!

  • Tried setting path to a folder that already exists with stuff in it

$ ls test
Vagrantfile

$ conda-debug ~/miniconda3/pkgs/alabaster-0.7.11-py35_0.tar.bz2 -a -p test
  • Then I did it again (to the same folder)
  • Looks like it will overwrite it
  • Tried with another package (a folder, not .tar.gz)
    • The output of this was a bit confusing
  • Tried with another package (.tar.gz, not a folder)
    • Added the artifact

Steps to reproduce bug:

  1. Run conda debug give it a path. Eg conda-debug ~/miniconda3/pkgs/filelock-3.0.4-py36_0.tar.bz2 -p test
  2. Run conda debug for another package and give it the same path eg. conda-debug ~/miniconda3/pkgs/alabaster-0.7.11-py35_0.tar.bz2 -p test
  • Gave it a directory that exists but is fulll of not packages
    • Returns on OSError OSError: No meta.yaml or conda.yaml files found in /Users/scastellarin/Desktop/test
  • Gave it a directory filled with packages + recipes OSError: More than one meta.yaml files found in /Users/scastellarin/miniconda3/pkgs
  • did some more things kind of like this and got the same error message

@minrk
Copy link
Contributor

minrk commented Nov 16, 2018

This is great! I'm not sure if it's the right thing to do, but I'd probably usually want to debug with a not-long host prefix if I could (parent/_host instead of parent/_h_placehold...). I know that would technically not test the issues that the long path is needed for, but the long paths genuinely make things harder to debug.

@msarahan
Copy link
Contributor Author

Thanks @minrk, that's a good idea, and should be easy to implement.

I had a couple of other ideas this morning:

  1. for build/host debugging, after provisioning the source, try to git init/add files in the work dir. Multiple sources will make this complicated, but the payoff would be huge in streamlining development of patches.
  2. In addition to 1 here, add in commands or flags to conda-debug that streamline translation of changes in the debug work dir into patches that are entered into the original meta.yaml file. This would be done with git format-patch and some parsing of the existing meta.yaml to understand what the next patch number should be. It assumes that the conda-debug user would use "git commit" in the work dir for each thing that should be a patch.

@minrk
Copy link
Contributor

minrk commented Nov 16, 2018

In addition to 1 here, add in commands or flags to conda-debug that streamline translation of changes in the debug work dir into patches that are entered into the original meta.yaml file

I would love this. I regularly do git init; git add .; git commit -m 'starting point' so I can use format-patch, since I can't seem to ever remember the incantation to manually create patches. Making this easier would help me a ton.

btw, I've already used this to work out some issues in the petsc and openmpi repos, and it helped! A couple small weirdnesses I ran into:

  • I did source build_env_setup.sh followed by bash conda_build.sh. The build failed because conda_build.sh sources the same env file, and apparently some of the flags being added twice caused builds to fail (I didn't bother debugging this too deeply)
  • I messed myself up by debugging two builds from the same terminal session, so lots of flags inherited from the first debugging session were inherited in the second (I managed to accidentally compile petsc against the openmpi build I was debugging, which petsc caught in its configuration checks). Not sure what to do about that. Does conda-build normally reset things like CFLAGS, or does it always append? If it resets, then these env scripts should perhaps reset some envs explicitly as well. Alternately, a hint that starting a dedicated shell per debugging session is a good idea might be appropriate.

@embray
Copy link

embray commented Nov 23, 2018

Thank you for working on this. I'll try it out in a bit. +1 to Min's comments that making builds easier for debugging (e.g. removing the long host prefix) would be best, as long as it is understood by the user that this is for creating debug builds of packages only, and not builds that should be pushed to users.

@msarahan msarahan force-pushed the add_debug_cmd branch 2 times, most recently from afc359e to 273561b Compare November 29, 2018 03:46
@msarahan
Copy link
Contributor Author

Going ahead with this. There's one failing test:

tests/test_subpackages.py::test_subpackage_independent_hash

That test passes locally. I'm not concerned about it now, though it would be nice to have green icons again...

@msarahan msarahan merged commit ea99e2d into conda:master Nov 29, 2018
@msarahan msarahan deleted the add_debug_cmd branch November 29, 2018 03:50
@ihnorton
Copy link

ihnorton commented Dec 3, 2018

Thank you for implementing this! 👍

I was a little bit surprised that I still needed to run conda activate /path/to/_h_env/ to get CC, CPPFLAGS, etc. from the activation script for a package that is specified in my build requirements (gxx_linux-64). Would it make sense to add that activation as well, or at least a note in the documentation?

@msarahan
Copy link
Contributor Author

msarahan commented Dec 3, 2018

I think you might be getting caught with #3289, which should be fixed by #3288

@github-actions
Copy link

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!

@github-actions github-actions bot added the locked [bot] locked due to inactivity label Apr 17, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked [bot] locked due to inactivity
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add an interactive shell debug option for conda build
7 participants