Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit objects cached by DevWorkspace controller to reduce memory usage #652

Merged
merged 14 commits into from
Nov 5, 2021

Commits on Oct 20, 2021

  1. Restrict which k8s objects are cached in runtime

    Modify cache used in controller in order to restrict cache to items with
    the devworkspace_id label. This has the downside of making all objects
    _without_ that label invisible to the controller, but has the benefit of
    reduced memory usage on large clusters.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    3d3732d View commit details
    Browse the repository at this point in the history
  2. Restrict secrets and configmaps in the cache via new labels

    Add labels
    
      - controller.devfile.io/watch-configmap
      - controller.devfile.io/watch-secret
    
    which must be set to "true" in order for the DevWorkspace Operator to
    see the  corresponding secret/configmap.
    
    This is required (compare to the previous commit) because the controller
    is not only interested in secrets and configmaps it creates, but also
    any configmap/secret on the cluster with e.g. the automount label
    attached.
    
    Since each type in the controller gets a single informer, we can only
    specify a single label selector for the objects we are interested in.
    This means we cannot have e.g. "has devworkspace_id label OR has
    mount-to-devworkspace label".
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    d6a12b1 View commit details
    Browse the repository at this point in the history
  3. Update how workspace metadata configmap is handled

    Restricting the cache to only configmaps with the new label results in
    existing workspaces failing to reconcile. This occurs because attempting
    to Get() the configmap from the cluster returns a IsNotFound error,
    whereas attempting to Create() the configmap returns an AlreadyExists
    error (Create interacts with the cluster, Get interacts with the cache).
    
    To avoid this, if we encounter an AlreadyExists error when attempting to
    create an object, we optimistically try to update the object (thus
    adding the required label). This resolves the issue above, as if the
    obejct is updated, the subsequent Get() call will return the object as
    expected.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    1b2d3b7 View commit details
    Browse the repository at this point in the history
  4. Collect sync code into one interface to simplify syncing with cluster

    Restricting the controller-runtime cache to specific objects means that
    once-tracked objects can disappear from the controller's knowledge if
    the required label is removed. To work around this, it is necessary to
    update how we sync objects to specifically handle the case where:
    
      * client.Get(object) returns IsNotFound
      * client.Create(object) returns AlreadyExists
    
    This occurs because we can't read objects that aren't in the cache, but
    attempting to create objects collides with the actual object on the
    cluster.
    
    Since the basic flow of Get -> Create/Update is repeated for each type
    we handle, this commit collects that repeated logic into one package
    (pkg/provision/sync), allowing object handling to be done in one place.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    4e4e5b3 View commit details
    Browse the repository at this point in the history
  5. Adapt more packages to use sync package for managing objects

    Adapt the metadata and storage cleanup tasks to use the new sync flow
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    1b4a7e0 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2021

  1. Update devworkspaceRouting controller to use sync package for objects

    Update sync methods in devworkspaceRouting controller to use updated
    sync package where appropriate. Note deleting out-of-date network
    objects must still be done in the controller, as it's not possible to
    iterate through a generic list of client.Objects.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    0008111 View commit details
    Browse the repository at this point in the history
  2. Add documentation and logging to pkg/provision/sync package

    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    2130ef1 View commit details
    Browse the repository at this point in the history
  3. Fix cache initialization on Kubernetes

    On Kubernetes, we can't restrict the cache for Routes since they are not
    a part of the included scheme. As a result we have to work around adding
    Routes to the cache only on OpenShift.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    43aae00 View commit details
    Browse the repository at this point in the history
  4. Use sync package for async storage deployment

    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    465405e View commit details
    Browse the repository at this point in the history
  5. Rework automount package to use ClusterAPI instead of client

    Pass around the full clusterAPI struct to methods in automount package,
    to allow for shared Context/Logging.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    8290933 View commit details
    Browse the repository at this point in the history
  6. Adapt automount git credentials to use new sync mechanism

    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    299186d View commit details
    Browse the repository at this point in the history
  7. Treat updating services specially since we have to copy ClusterIP

    For most objects, we can client.Update() using the spec object without
    issue. However, for Services, updates are rejected if they try to unset
    spec.ClusterIP. This means we need to copy the ClusterIP from the
    cluster service before updating.
    
    This commit adds an extensible mechanism for specifying type-specific
    update functions that are called whenever we attempt to update a cluster
    object.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    76d43d7 View commit details
    Browse the repository at this point in the history
  8. Improve diff logging when updating items

    Use diffOpts when printing spec vs cluster object diffs when updates are
    required.
    
    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    6c83e0a View commit details
    Browse the repository at this point in the history
  9. Restrict caches for Roles and Rolebindings by name

    Signed-off-by: Angel Misevski <amisevsk@redhat.com>
    amisevsk committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    8e07871 View commit details
    Browse the repository at this point in the history