hjoliver · wxtim · Aug 1, 2022 · Aug 1, 2022 · Aug 2, 2022 · Aug 2, 2022
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,27 +1,18 @@
-<!-- Complete this Pull Request template. -->
+<!--
+Thanks for your contribution:
+* Please list any related issues with a "closes" or "addresses" tag.
+* Please add a helpful description.
+* For bugfixes we have maintenance branches e.g. `8.0.x`, please raise separate
+  pull requests against master and the maintenance release branches as appropriate.
+-->
 
-<!-- Significant PRs should address an existing Issue. Choose one: -->
+**Check List**
 
-These changes partially address #xxxx
-These changes close #xxxx
-This is a small change with no associated Issue.
-
-<!-- The following requirements must be satisfied (with "[x]"). -->
-<!-- Mark the PR as a Draft if all requirements are not yet satisfied. -->
-
-**Requirements check-list**
 - [ ] I have read `CONTRIBUTING.md` and added my name as a Code Contributor.
 - [ ] Contains logically grouped changes (else tidy your branch by rebase).
 - [ ] Does not contain off-topic changes (use other PRs for other changes).
 - [ ] Applied any dependency changes to both `setup.cfg` and `conda-environment.yml`.
-<!-- choose one: -->
-- [ ] Appropriate tests are included (unit and/or functional).
-- [ ] Already covered by existing tests.
-- [ ] Does not need tests (why?).
-<!-- choose one: -->
-- [ ] Appropriate change log entry included.
-- [ ] No change log entry required (why? e.g. invisible to users).
-<!-- choose one: -->
-- [ ] (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
-- [ ] (7.8.x branch) I have updated the documentation in this PR branch.
-- [ ] No documentation update required.
+- [ ] Tests are included (or explain why tests are not needed).
+- [ ] `CHANGES.md` entry included if this is a change that can affect users
+- [ ] [Cylc-Doc](https://github.com/cylc/cylc-doc) pull request opened if required at cylc/cylc-doc/pull/XXXX.
+- [ ] If this is a bug fix, PRs raised to both master and the relevant maintenance branch.
diff --git a/CHANGES.md b/CHANGES.md
@@ -28,13 +28,26 @@ creating a new release entry be sure to copy & paste the span tag with the
 `actions:bind` attribute, which is used by a regex to find the text to be
 updated. Only the first match gets replaced, so it's fine to leave the old
 ones in. -->
+
+-------------------------------------------------------------------------------
+## __cylc-8.1.0 (<span actions:bind='release-date'>Upcoming</span>)__
+
+### Enhancements
+
+[#5032](https://github.com/cylc/cylc-flow/pull/5032) - set a default limit of
+100 for the "default" queue.
+
+
 -------------------------------------------------------------------------------
 ## __cylc-8.0.1 (<span actions:bind='release-date'>Upcoming</span>)__
 
 Maintenance release.
 
 ### Fixes
 
+[#5066](https://github.com/cylc/cylc-flow/pull/5066) - Fix bug where
+.cylcignore only found if `cylc install` is run in source directory.
+
 [#5031](https://github.com/cylc/cylc-flow/pull/5031) - Fix bug where
 specifying multiple datetime offsets (e.g. `final cycle point = +P1M-P1D`)
 would not obey the given order.

diff --git a/README.md b/README.md
@@ -14,49 +14,51 @@
 
 </div>
 
-Cylc (pronounced silk) is a general purpose workflow engine
-that specialises in cycling workflows and has strong scaling characteristics.
+Cylc (pronounced silk) is a general purpose workflow engine that also
+manages cycling systems very efficiently. It is used in production weather,
+climate, and environmental forecasting on HPC, but is not specialized to those
+domains.
 
-Cylc was originally developed to meet the challenges of production weather
-forecasting - which is notorious for the size and complexity of its workflows.
+### Quick Start
+
+
+[Installation](https://cylc.github.io/cylc-doc/stable/html/installation.html) |
+[Documentation](https://cylc.github.io/cylc-doc/stable/html/index.html)
+
+```bash
+# install cylc
+conda install cylc-flow
+
+# write your first workflow
+mkdir -p ~/cylc-src/example
+cat > ~/cylc-src/example/flow.cylc <<__CONFIG__
+[scheduling]
+    initial cycle point = 1
+    cycling mode = integer
+    [[graph]]
+        P1 = """
+            a => b => c & d
+            b[-P1] => b
+        """
+[runtime]
+    [[a, b, c, d]]
+        script = echo "Hello $CYLC_TASK_NAME"
+__CONFIG__
+
+# install and run it
+cylc install example
+cylc play example
+
+# watch it run
+cylc tui example
+```
 
 ### Citations & Publications
 
 [![DOI](https://zenodo.org/badge/1836229.svg)](https://zenodo.org/badge/latestdoi/1836229)
 [![JOSS](http://joss.theoj.org/papers/10.21105/joss.00737/status.svg)](https://doi.org/10.21105/joss.00737)
 [![CISE](https://img.shields.io/website/https/ieeexplore.ieee.org/document/8675433.svg?color=orange&label=CISE&up_message=10.1109%2FMCSE.2019.2906593)](https://ieeexplore.ieee.org/document/8675433)
 
-### Cylc 7 (legacy)
-
-![python](https://img.shields.io/badge/python-2.6%20%7C%202.7-orange)
-[![Documentation](https://img.shields.io/website?label=documentation&up_message=live&url=https%3A%2F%2Fcylc.github.io%2Fcylc-doc%2F7.9.3%2Fhtml%2Findex.html)](https://cylc.github.io/cylc-doc/7.9.3/html/index.html)
-
-* HTTPS network layer.
-* PyGTK GUI.
-* On the `7.8.x` branch in the source code.
-* 7.8 - Python 2.6
-* 7.9 - Python 2.7
-
-[Installation](https://github.com/cylc/cylc-flow/blob/7.8.x/INSTALL.md) |
-[Documentation](https://cylc.github.io/documentation)
-
-### Cylc 8 (production)
-
-![PyPI](https://img.shields.io/pypi/pyversions/cylc-flow.svg?color=green)
-[![PyPI](https://img.shields.io/pypi/v/cylc-flow.svg?color=yellow)](https://pypi.org/project/cylc-flow/)
-[![Anaconda-Server Badge](https://anaconda.org/conda-forge/cylc-flow/badges/version.svg)](https://anaconda.org/conda-forge/cylc-flow)
-[![Documentation](https://img.shields.io/website?label=documentation&up_message=live&url=https%3A%2F%2Fcylc.github.io%2Fcylc-doc%2Fstable%2Fhtml%2Findex.html)](https://cylc.github.io/cylc-doc/latest/html/index.html)
-
-* ZMQ (TCP) network layer.
-* Text-based terminal user interface (TUI).
-* Optional web-based graphical user interface (GUI) provided by ([cylc-uiserver](https://github.com/cylc/cylc-uiserver)).
-* On the `master` branch in the source code.
-
-Cylc 8 is now production-ready.
-
-[Installation](https://cylc.github.io/cylc-doc/stable/html/installation.html) |
-[Documentation](https://cylc.github.io/cylc-doc/stable/html/index.html)
-
 ### Copyright and Terms of Use
 
 [![License](https://img.shields.io/github/license/cylc/cylc-flow.svg?color=lightgrey)](https://github.com/cylc/cylc-flow/blob/master/COPYING)

diff --git a/conda-environment.yml b/conda-environment.yml
@@ -15,7 +15,6 @@ dependencies:
   - protobuf >=3.19.0,<3.20.0
   - psutil >=5.6.0
   - python
-  - pyuv >=1.4.0,<1.5.0
   - pyzmq >=22,<23
   - setuptools >=49
   - urwid >=2,<3

diff --git a/cylc/flow/async_util.py b/cylc/flow/async_util.py
@@ -16,11 +16,11 @@
 """Utilities for use with asynchronous code."""
 
 import asyncio
-from functools import partial
+import os
 from pathlib import Path
 from typing import List, Union
 
-import pyuv
+from aiofiles.os import wrap  # type: ignore[attr-defined]
 
 from cylc.flow import LOG
 
@@ -393,24 +393,15 @@ def _pipe(func):
         return _pipe
 
 
-def _scandir(future, path, request):
-    """Callback helper for scandir()."""
-    future.set_result([
-        Path(path, directory.name)
-        # request.result is None for empty dirs
-        for directory in request.result or []
-    ])
+async_listdir = wrap(os.listdir)
 
 
 async def scandir(path: Union[Path, str]) -> List[Path]:
-    """Asynchronous directory listing using pyuv."""
-    ret: asyncio.Future[List[Path]] = asyncio.Future()
-
-    loop = pyuv.Loop.default_loop()
-    pyuv.fs.scandir(loop, str(path), callback=partial(_scandir, ret, path))
-    loop.run()
-
-    return await ret
+    """Asynchronous directory listing (performs os.listdir in an executor)."""
+    return [
+        Path(path, sub_path)
+        for sub_path in await async_listdir(path)
+    ]
 
 
 async def asyncqgen(queue):

diff --git a/cylc/flow/cfgspec/workflow.py b/cylc/flow/cfgspec/workflow.py
@@ -691,7 +691,7 @@ def get_script_common_text(this: str, example: Optional[str] = None):
             with Conf('default', meta=Queue, desc='''
                 The default queue for all tasks not assigned to other queues.
             '''):
-                Conf('limit', VDR.V_INTEGER, 0, desc='''
+                Conf('limit', VDR.V_INTEGER, 100, desc='''
                     Controls the total number of active tasks in the default
                     queue.
 

diff --git a/cylc/flow/config.py b/cylc/flow/config.py
@@ -2096,36 +2096,38 @@ def set_required_outputs(
                     continue
                 taskdef.set_required_output(output, not optional)
 
-    def find_taskdefs(self, name: str) -> List[TaskDef]:
+    def find_taskdefs(self, name: str) -> Set[TaskDef]:
         """Find TaskDef objects in family "name" or matching "name".
 
         Return a list of TaskDef objects which:
         * have names that glob matches "name".
         * are in a family that glob matches "name".
         """
-        ret: List[TaskDef] = []
         if name in self.taskdefs:
             # Match a task name
-            ret.append(self.taskdefs[name])
-        else:
-            fams = self.runtime['first-parent descendants']
+            return {self.taskdefs[name]}
+
+        fams = self.runtime['first-parent descendants']
+        if name in fams:
             # Match a family name
-            if name in fams:
-                for member in fams[name]:
-                    if member in self.taskdefs:
-                        ret.append(self.taskdefs[member])
-            else:
-                # Glob match task names
-                for key, taskdef in self.taskdefs.items():
-                    if fnmatchcase(key, name):
-                        ret.append(taskdef)
-                # Glob match family names
-                for key, members in fams.items():
-                    if fnmatchcase(key, name):
-                        for member in members:
-                            if member in self.taskdefs:
-                                ret.append(self.taskdefs[member])
-        return ret
+            return {
+                self.taskdefs[member] for member in fams[name]
+                if member in self.taskdefs
+            }
+
+        # Glob match
+        from_task_names = {
+            taskdef for key, taskdef in self.taskdefs.items()
+            if fnmatchcase(key, name)
+        }
+        from_family_names = {
+            self.taskdefs[member]
+            for key, members in fams.items()
+            if fnmatchcase(key, name)
+            for member in members
+            if member in self.taskdefs
+        }
+        return from_task_names.union(from_family_names)
 
     def get_taskdef(
         self, name: str, orig_expr: Optional[str] = None

diff --git a/cylc/flow/pathutil.py b/cylc/flow/pathutil.py
@@ -42,6 +42,8 @@
 )
 """Matches relative paths that are explicit (starts with ./)"""
 
+SHELL_ENV_VARS = re.compile(r'\$[^$/]*')
+
 
 def expand_path(*args: Union[Path, str]) -> str:
     """Expand both vars and user in path and normalise it, joining any
@@ -174,11 +176,13 @@ def make_localhost_symlinks(
         else:
             symlink_path = os.path.join(rund, key)
         target = expand_path(value)
-        if '$' in target:
+        env_vars = SHELL_ENV_VARS.findall(target)
+        if env_vars:
             raise WorkflowFilesError(
-                f'Unable to create symlink to {target}.'
-                f" '{value}' contains an invalid environment variable."
-                ' Please check configuration.')
+                f"Can't symlink to {target}\n"
+                "Undefined variables, check "
+                f"global config: {', '.join(env_vars)}")
+
         symlink_success = make_symlink_dir(symlink_path, target)
         # Symlink info returned for logging purposes. Symlinks should be
         # created before logs as the log dir may be a symlink.

diff --git a/cylc/flow/task_pool.py b/cylc/flow/task_pool.py
@@ -77,6 +77,7 @@ class TaskPool:
     """Task pool of a workflow."""
 
     ERR_TMPL_NO_TASKID_MATCH = "No matching tasks found: {0}"
+    ERR_PREFIX_TASK_NOT_ON_SEQUENCE = "Invalid cycle point for task: {0}, {1}"
     SUICIDE_MSG = "suicide"
 
     def __init__(
@@ -1429,6 +1430,11 @@ def spawn_task(
         # Spawn if on-sequence and within recurrence bounds.
         taskdef = self.config.get_taskdef(name)
         if not taskdef.is_valid_point(point):
+            LOG.warning(
+                self.ERR_PREFIX_TASK_NOT_ON_SEQUENCE.format(
+                    taskdef.name, point
+                )
+            )
             return None
 
         itask = TaskProxy(
@@ -1498,8 +1504,7 @@ def force_spawn_children(
 
         Args:
             items: Identifiers for matching task definitions, each with the
-                form "name[.point][:state]" or "[point/]name[:state]".
-                Glob-like patterns will give a warning and be skipped.
+                form "point/name".
             outputs: List of outputs to spawn on
             flow_num: Flow number to attribute the outputs
 
@@ -1766,7 +1771,6 @@ def filter_task_proxies(
             [self.main_pool, self.hidden_pool],
             ids,
             warn=warn,
-
         )
         future_matched: 'Set[Tuple[str, PointBase]]' = set()
         if future and unmatched:
@@ -1844,7 +1848,11 @@ def match_future_tasks(
             if taskdef.is_valid_point(point):
                 matched_tasks.add((taskdef.name, point))
             else:
-                # is_valid_point() already logged warning
+                LOG.warning(
+                    self.ERR_PREFIX_TASK_NOT_ON_SEQUENCE.format(
+                        taskdef.name, point
+                    )
+                )
                 unmatched_tasks.append(id_)
                 continue
         return matched_tasks, unmatched_tasks
@@ -1857,8 +1865,10 @@ def match_taskdefs(
         Args:
             items:
                 Identifiers for matching task definitions, each with the
-                form "name[.point][:state]" or "[point/]name[:state]".
-                Glob-like patterns will give a warning and be skipped.
+                form "point/name".
+                Cycle point globs will give a warning and be skipped,
+                but task name globs will be matched.
+                Task states are ignored.
 
         """
         n_warnings = 0
@@ -1884,7 +1894,7 @@ def match_taskdefs(
                 )
                 n_warnings += 1
                 continue
-            taskdefs: List['TaskDef'] = self.config.find_taskdefs(name_str)
+            taskdefs = self.config.find_taskdefs(name_str)
             if not taskdefs:
                 LOG.warning(
                     self.ERR_TMPL_NO_TASKID_MATCH.format(
@@ -1898,8 +1908,13 @@ def match_taskdefs(
                 if taskdef.is_valid_point(point):
                     task_items[(taskdef.name, point)] = taskdef
                 else:
-                    # is_valid_point() already logged warning
-                    n_warnings += 1
+                    if not contains_fnmatch(name_str):
+                        LOG.warning(
+                            self.ERR_PREFIX_TASK_NOT_ON_SEQUENCE.format(
+                                taskdef.name, point
+                            )
+                        )
+                        n_warnings += 1
                     continue
         return n_warnings, task_items