Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logger rework. Package local logger. #1583

Merged
merged 3 commits into from
Dec 20, 2016
Merged

logger rework. Package local logger. #1583

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0fe1a13
logger rework. Package local logger.
msarahan Dec 17, 2016
53e2ff4
restore debug logger functionality, implement context manager
msarahan Dec 17, 2016
4de418b
revert caplog text attribute calls
msarahan Dec 19, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions conda_build/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
__version__ = get_versions()['version']
del get_versions

import logging

# Sub commands added by conda-build to the conda command
sub_commands = [
'build',
Expand All @@ -21,3 +23,13 @@
'sign',
'skeleton',
]

# Set default logging handler to avoid "No handler found" warnings.
try: # Python 2.7+
from logging import NullHandler
except ImportError:
class NullHandler(logging.Handler):
def emit(self, record):
pass

logging.getLogger(__name__).addHandler(NullHandler())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you're not done with this PR yet, so this could be a premature comment.
Using NullHandler is exactly the right thing to do for a library. It makes your library a good logging citizen. Also, the package's root-level __init__.py file is almost always the right place to invoke the basic logging configuration. (In conda's case I'm not, because I don't want to bog down the ..activate etc cli stuff with logger initialization. It's way too slow already.)

However, unless you add another handler somewhere else, the problem with conda-build logs will remain. I'm guessing you're just not done with the PR though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment. I'm not sure why, but things seem to be working with the code as is. Any insight?

I have a package on my channel, intentionally built with an 80-character prefix. This recipe will try to bring it in as a dependency:

package:
  name: test-pkg
  version: 1.0

requirements:
  build:
    - conda-build-test-has-prefix-files

Now, if you build that recipe (conda build -c msarahan .), you see the logger output, as I would hope. is there something else that isn't showing because of the way I have things now?

 msarahan@0109-msarahan-2  ~/code/conda-build/tests/test_pkg   logger  conda build -c msarahan .
BUILD START: test-pkg-1.0-0
updating index in: /Users/msarahan/miniconda2/conda-bld/osx-64
updating index in: /Users/msarahan/miniconda2/conda-bld/noarch

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-build-test-has-prefix-files-1.0|                0           3 KB  local

The following NEW packages will be INSTALLED:

    conda-build-test-has-prefix-files: 1.0-0 local

WARNING conda_build.build:create_env(648): Build prefix failed with prefix length 255
WARNING conda_build.build:create_env(649): Error was:
WARNING conda_build.build:create_env(650): Placeholder of length '80' too short in package local::conda-build-test-has-prefix-files-1.0-0.
The package must be rebuilt with conda-build > 2.0.
WARNING conda_build.build:create_env(651): One or more of your package dependencies needs to be rebuilt with a longer prefix length.
WARNING conda_build.build:create_env(653): Falling back to legacy prefix length of 80 characters.
WARNING conda_build.build:create_env(654): Your package will not install into prefixes > 80 characters.

157 changes: 78 additions & 79 deletions conda_build/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@
from conda_build.post import (post_process, post_build,
fix_permissions, get_build_metadata)
from conda_build.utils import (rm_rf, _check_call, copy_into, on_win, get_build_folders,
silence_loggers, path_prepended, create_entry_points,
path_prepended, create_entry_points,
prepend_bin_path, codec, root_script_dir, print_skip_message,
ensure_list, get_lock, ExitStack, get_recipe_abspath, tmp_chdir,
expand_globs)
expand_globs, LoggingContext)
from conda_build.metadata import build_string_from_metadata
from conda_build.index import update_index
from conda_build.create_test import (create_files, create_shell_files,
Expand All @@ -75,8 +75,6 @@
else:
shell_path = '/bin/bash'

log = logging.getLogger(__file__)


def prefix_files(prefix):
'''
Expand Down Expand Up @@ -593,82 +591,78 @@ def create_env(prefix, specs, config, clear_cache=True):
Create a conda envrionment for the given prefix and specs.
'''
if config.debug:
logging.getLogger("conda").setLevel(logging.DEBUG)
logging.getLogger("binstar").setLevel(logging.DEBUG)
logging.getLogger("install").setLevel(logging.DEBUG)
logging.getLogger("conda.install").setLevel(logging.DEBUG)
logging.getLogger("fetch").setLevel(logging.DEBUG)
logging.getLogger("print").setLevel(logging.DEBUG)
logging.getLogger("progress").setLevel(logging.DEBUG)
logging.getLogger("dotupdate").setLevel(logging.DEBUG)
logging.getLogger("stdoutlog").setLevel(logging.DEBUG)
logging.getLogger("requests").setLevel(logging.DEBUG)
logging.getLogger("conda_build").setLevel(logging.DEBUG)
external_logger_context = LoggingContext(logging.DEBUG)
else:
silence_loggers(show_warnings_and_errors=True)

if os.path.isdir(prefix):
rm_rf(prefix)
logging.getLogger("conda_build").setLevel(logging.INFO)
external_logger_context = LoggingContext(logging.ERROR)

specs = list(specs)
for feature, value in feature_list:
if value:
specs.append('%s@' % feature)
with external_logger_context:
log = logging.getLogger(__name__)

if specs: # Don't waste time if there is nothing to do
log.debug("Creating environment in %s", prefix)
log.debug(str(specs))
if os.path.isdir(prefix):
rm_rf(prefix)

with path_prepended(prefix):
locks = []
try:
cc.pkgs_dirs = cc.pkgs_dirs[:1]
locked_folders = cc.pkgs_dirs + list(config.bldpkgs_dirs)
for folder in locked_folders:
if not os.path.isdir(folder):
os.makedirs(folder)
lock = get_lock(folder, timeout=config.timeout)
if not folder.endswith('pkgs'):
update_index(folder, config=config, lock=lock, could_be_mirror=False)
locks.append(lock)

with ExitStack() as stack:
for lock in locks:
stack.enter_context(lock)
index = get_build_index(config=config, clear_cache=True)

actions = plan.install_actions(prefix, index, specs)
if config.disable_pip:
actions['LINK'] = [spec for spec in actions['LINK'] if not spec.startswith('pip-')] # noqa
actions['LINK'] = [spec for spec in actions['LINK'] if not spec.startswith('setuptools-')] # noqa
plan.display_actions(actions, index)
if on_win:
for k, v in os.environ.items():
os.environ[k] = str(v)
plan.execute_actions(actions, index, verbose=config.debug)
except (SystemExit, PaddingError, LinkError) as exc:
if (("too short in" in str(exc) or
'post-link failed for: openssl' in str(exc) or
isinstance(exc, PaddingError)) and
config.prefix_length > 80):
if config.prefix_length_fallback:
log.warn("Build prefix failed with prefix length %d", config.prefix_length)
log.warn("Error was: ")
log.warn(str(exc))
log.warn("One or more of your package dependencies needs to be rebuilt "
"with a longer prefix length.")
log.warn("Falling back to legacy prefix length of 80 characters.")
log.warn("Your package will not install into prefixes > 80 characters.")
config.prefix_length = 80

# Set this here and use to create environ
# Setting this here is important because we use it below (symlink)
prefix = config.build_prefix

create_env(prefix, specs, config=config,
clear_cache=clear_cache)
else:
raise
warn_on_old_conda_build(index=index)
specs = list(specs)
for feature, value in feature_list:
if value:
specs.append('%s@' % feature)

if specs: # Don't waste time if there is nothing to do
log.debug("Creating environment in %s", prefix)
log.debug(str(specs))

with path_prepended(prefix):
locks = []
try:
cc.pkgs_dirs = cc.pkgs_dirs[:1]
locked_folders = cc.pkgs_dirs + list(config.bldpkgs_dirs)
for folder in locked_folders:
if not os.path.isdir(folder):
os.makedirs(folder)
lock = get_lock(folder, timeout=config.timeout)
if not folder.endswith('pkgs'):
update_index(folder, config=config, lock=lock, could_be_mirror=False)
locks.append(lock)

with ExitStack() as stack:
for lock in locks:
stack.enter_context(lock)
index = get_build_index(config=config, clear_cache=True)

actions = plan.install_actions(prefix, index, specs)
if config.disable_pip:
actions['LINK'] = [spec for spec in actions['LINK'] if not spec.startswith('pip-')] # noqa
actions['LINK'] = [spec for spec in actions['LINK'] if not spec.startswith('setuptools-')] # noqa
plan.display_actions(actions, index)
if on_win:
for k, v in os.environ.items():
os.environ[k] = str(v)
plan.execute_actions(actions, index, verbose=config.debug)
except (SystemExit, PaddingError, LinkError) as exc:
if (("too short in" in str(exc) or
'post-link failed for: openssl' in str(exc) or
isinstance(exc, PaddingError)) and
config.prefix_length > 80):
if config.prefix_length_fallback:
log.warn("Build prefix failed with prefix length %d", config.prefix_length)
log.warn("Error was: ")
log.warn(str(exc))
log.warn("One or more of your package dependencies needs to be rebuilt "
"with a longer prefix length.")
log.warn("Falling back to legacy prefix length of 80 characters.")
log.warn("Your package will not install into prefixes > 80 characters.")
config.prefix_length = 80

# Set this here and use to create environ
# Setting this here is important because we use it below (symlink)
prefix = config.build_prefix

create_env(prefix, specs, config=config,
clear_cache=clear_cache)
else:
raise
warn_on_old_conda_build(index=index)

# ensure prefix exists, even if empty, i.e. when specs are empty
if not isdir(prefix):
Expand All @@ -685,7 +679,7 @@ def get_installed_conda_build_version():
vers_inst = [dist.split('::', 1)[-1].rsplit('-', 2)[1] for dist in root_linked
if dist.split('::', 1)[-1].rsplit('-', 2)[0] == 'conda-build']
if not len(vers_inst) == 1:
log.warn("Could not detect installed version of conda-build")
logging.getLogger(__name__).warn("Could not detect installed version of conda-build")
return None
return vers_inst[0]

Expand All @@ -696,7 +690,8 @@ def get_conda_build_index_versions(index):
try:
pkgs = r.get_pkgs(MatchSpec('conda-build'))
except (NoPackagesFound, NoPackagesFoundError):
log.warn("Could not find any versions of conda-build in the channels")
logging.getLogger(__name__).warn("Could not find any versions of conda-build "
"in the channels")
return [pkg.version for pkg in pkgs]


Expand Down Expand Up @@ -835,6 +830,8 @@ def build(m, config, post=None, need_source_download=True, need_reparse_in_env=F
print_skip_message(m)
return []

log = logging.getLogger(__name__)

with path_prepended(config.build_prefix):
env = environ.get_dict(config=config, m=m)
env["CONDA_BUILD_STATE"] = "BUILD"
Expand Down Expand Up @@ -1097,6 +1094,7 @@ def clean_pkg_cache(dist, timeout):
for pkg_id in [dist, 'local::' + dist]:
assert pkg_id not in package_cache()
except AssertionError:
log = logging.getLogger(__name__)
log.debug("Conda caching error: %s package remains in cache after removal", dist)
log.debug("Clearing package cache to compensate")
cache = package_cache()
Expand Down Expand Up @@ -1499,7 +1497,8 @@ def handle_pypi_upload(f, config):
try:
subprocess.check_call()
except:
log.warn("wheel upload failed - is twine installed? Is this package registered?")
logging.getLogger(__name__).warn("wheel upload failed - is twine installed?"
" Is this package registered?")


def print_build_intermediate_warning(config):
Expand Down
14 changes: 7 additions & 7 deletions conda_build/cli/main_build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
get_render_parser, bldpkg_path)
from conda_build.conda_interface import cc, add_parser_channels, url_path
import conda_build.source as source
from conda_build.utils import silence_loggers, print_skip_message
from conda_build.utils import print_skip_message, LoggingContext
from conda_build.config import Config

on_win = (sys.platform == 'win32')
Expand Down Expand Up @@ -221,12 +221,12 @@ def parse_args(args):


def output_action(recipe, config):
silence_loggers(show_warnings_and_errors=False)
metadata, _, _ = api.render(recipe, config=config)
if metadata.skip():
print_skip_message(metadata)
else:
print(bldpkg_path(metadata))
with LoggingContext(logging.CRITICAL + 1):
metadata, _, _ = api.render(recipe, config=config)
if metadata.skip():
print_skip_message(metadata)
else:
print(bldpkg_path(metadata))


def source_action(metadata, config):
Expand Down
16 changes: 7 additions & 9 deletions conda_build/cli/main_render.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from conda_build.completers import (RecipeCompleter, PythonVersionCompleter, RVersionsCompleter,
LuaVersionsCompleter, NumPyVersionCompleter)
from conda_build.config import Config
from conda_build.utils import silence_loggers
from conda_build.utils import LoggingContext

on_win = (sys.platform == 'win32')

Expand Down Expand Up @@ -127,14 +127,12 @@ def execute(args):
config = Config()
set_language_env_vars(args, p, config)

metadata, _, _ = render_recipe(args.recipe, no_download_source=args.no_source, config=config)
if args.output:
logging.basicConfig(level=logging.ERROR)
silence_loggers(show_warnings_and_errors=False)
print(bldpkg_path(metadata))
else:
logging.basicConfig(level=logging.INFO)
print(output_yaml(metadata, args.file))
with LoggingContext(logging.CRITICAL + 1):
metadata, _, _ = render_recipe(args.recipe, no_download_source=args.no_source, config=config)
if args.output:
print(bldpkg_path(metadata))
else:
print(output_yaml(metadata, args.file))


def main():
Expand Down
4 changes: 2 additions & 2 deletions conda_build/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@

from .utils import get_build_folders, rm_rf

log = logging.getLogger(__file__)
on_win = (sys.platform == 'win32')
DEFAULT_PREFIX_LENGTH = 255

Expand Down Expand Up @@ -404,7 +403,8 @@ def __enter__(self):

def __exit__(self, e_type, e_value, traceback):
if not getattr(self, 'dirty') and not getattr(self, 'keep_old_work') and e_type is None:
log.info("--keep-old-work flag not specified. Removing source and build files.\n")
logging.getLogger(__name__).info("--keep-old-work flag not specified. "
"Removing source and build files.\n")
self.clean()


Expand Down
4 changes: 2 additions & 2 deletions conda_build/environ.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@
from conda_build.features import feature_list
from conda_build.utils import prepend_bin_path, ensure_list

log = logging.getLogger(__file__)


def get_perl_ver(config):
return str(config.CONDA_PERL)
Expand Down Expand Up @@ -50,6 +48,7 @@ def get_lua_include_dir(config):

def verify_git_repo(git_dir, git_url, config, expected_rev='HEAD'):
env = os.environ.copy()
log = logging.getLogger(__name__)
if config.verbose:
stderr = None
else:
Expand Down Expand Up @@ -138,6 +137,7 @@ def get_git_info(repo, config):
:return:
"""
d = {}
log = logging.getLogger(__name__)

if config.verbose:
stderr = None
Expand Down
8 changes: 3 additions & 5 deletions conda_build/inspect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@
from conda_build.utils import groupby, getter, comma_join, rm_rf


log = logging.getLogger(__file__)


def which_prefix(path):
"""
given the path (to a (presumably) conda installed file) return the
Expand Down Expand Up @@ -144,6 +141,7 @@ def replace_path(binary, path, prefix):

def test_installable(channel='defaults'):
success = True
log = logging.getLogger(__name__)
has_py = re.compile(r'py(\d)(\d)')
for platform in ['osx-64', 'linux-32', 'linux-64', 'win-32', 'win-64']:
log.info("######## Testing platform %s ########", platform)
Expand Down Expand Up @@ -238,8 +236,8 @@ def inspect_linkages(packages, prefix=sys.prefix, untracked=False,
if path.startswith(prefix):
deps = list(which_package(path))
if len(deps) > 1:
log.warn("Warning: %s comes from multiple packages: %s", path,
comma_join(deps))
logging.getLogger(__name__).warn("Warning: %s comes from multiple "
"packages: %s", path, comma_join(deps))
if not deps:
if exists(path):
depmap['untracked'].append((lib, path.split(prefix +
Expand Down
Loading