Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(analytics): Analytics refactor #307

Merged
merged 2 commits into from
May 17, 2022
Merged

feat(analytics): Analytics refactor #307

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
65bf964
Use recommendationMission to control recommendation lifecycle
qmhu May 9, 2022
3449055
Apply suggestions from code review
qmhu May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 14 additions & 12 deletions cmd/craned/app/manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
autoscalingapi "github.com/gocrane/api/autoscaling/v1alpha1"
ensuranceapi "github.com/gocrane/api/ensurance/v1alpha1"
predictionapi "github.com/gocrane/api/prediction/v1alpha1"

"github.com/gocrane/crane/cmd/craned/app/options"
"github.com/gocrane/crane/pkg/controller/analytics"
"github.com/gocrane/crane/pkg/controller/cnp"
Expand Down Expand Up @@ -282,29 +283,30 @@ func initializationControllers(ctx context.Context, mgr ctrl.Manager, opts *opti
}

if utilfeature.DefaultMutableFeatureGate.Enabled(features.CraneAnalysis) {
if err := (&analytics.Controller{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
RestMapper: mgr.GetRESTMapper(),
Recorder: mgr.GetEventRecorderFor("analytics-controller"),
}).SetupWithManager(mgr); err != nil {
klog.Exit(err, "unable to create controller", "controller", "AnalyticsController")
}

configSet, err := recommend.LoadConfigSetFromFile(opts.RecommendationConfigFile)
if err != nil {
klog.Errorf("Failed to load recommendation config file: %v", err)
os.Exit(1)
}
if err := (&recommendation.Controller{

if err := (&analytics.Controller{
Client: mgr.GetClient(),
ConfigSet: configSet,
Scheme: mgr.GetScheme(),
RestMapper: mgr.GetRESTMapper(),
Recorder: mgr.GetEventRecorderFor("recommendation-controller"),
Recorder: mgr.GetEventRecorderFor("analytics-controller"),
ConfigSet: configSet,
ScaleClient: scaleClient,
PredictorMgr: predictorMgr,
Provider: historyDataSource,
}).SetupWithManager(mgr); err != nil {
klog.Exit(err, "unable to create controller", "controller", "AnalyticsController")
}

if err := (&recommendation.Controller{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
RestMapper: mgr.GetRESTMapper(),
Recorder: mgr.GetEventRecorderFor("recommendation-controller"),
}).SetupWithManager(mgr); err != nil {
klog.Exit(err, "unable to create controller", "controller", "RecommendationController")
}
Expand Down
Binary file added docs/images/analytics-arch.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 4 additions & 2 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,10 @@ Analytics model analyzes the workload and provide recommendations about resource

Two Recommendations are currently supported:

- **ResourceRecommend**: Recommend container requests & limit resources based on historic metrics.
- **Effective HPARecommend**: Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.
- [**ResourceRecommend**](tutorials/resource-recommendation.md): Recommend container requests & limit resources based on historic metrics.
- [**Effective HPARecommend**](tutorials/hpa-recommendation.md): Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.

Please see [this document](tutorials/analytics-and-recommendation.md) to learn more.

### QoS Ensurance
Kubernetes is capable of starting multiple pods on same node, and as a result, some of the user applications may be impacted when there are resources(e.g. cpu) consumption competition. To mitigate this, Crane allows users defining PrioirtyClass for the pods and QoSEnsurancePolicy, and then detects disruption and ensure the high priority pods not being impacted by resource competition.
Expand Down
6 changes: 4 additions & 2 deletions docs/index.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@ Analytics model analyzes the workload and provide recommendations about resource

Two Recommendations are currently supported:

- **ResourceRecommend**: Recommend container requests & limit resources based on historic metrics.
- **Effective HPARecommend**: Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.
- [**ResourceRecommend**](tutorials/resource-recommendation.md): Recommend container requests & limit resources based on historic metrics.
- [**Effective HPARecommend**](tutorials/hpa-recommendation.md): Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.

Please see [this document](tutorials/analytics-and-recommendation.md) to learn more.

### QoS Ensurance
Kubernetes is capable of starting multiple pods on same node, and as a result, some of the user applications may be impacted when there are resources(e.g. cpu) consumption competition. To mitigate this, Crane allows users defining PrioirtyClass for the pods and QoSEnsurancePolicy, and then detects disruption and ensure the high priority pods not being impacted by resource competition.
Expand Down
294 changes: 19 additions & 275 deletions docs/tutorials/analytics-and-recommendation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,289 +4,33 @@ Analytics and Recommendation provide capacity that analyzes the workload in k8s

Two Recommendations are currently supported:

- **ResourceRecommend**: Recommend container requests & limit resources based on historic metrics.
- **Effective HPARecommend**: Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.
- [**ResourceRecommend**](resource-recommendation.md): Recommend container requests & limit resources based on historic metrics.
- [**HPARecommend**](hpa-recommendation.md): Recommend which workloads are suitable for autoscaling and provide optimized configurations such as minReplicas, maxReplicas.

## Analytics and Recommend Pod Resources
## Architecture

Create an **Resource** `Analytics` to give recommendation for deployment: `craned` and `metric-adapter` as a sample.
![analytics-arch](../images/analytics-arch.png)

```bash
kubectl apply -f https://raw.githubusercontent.com/gocrane/crane/main/examples/analytics/analytics-resource.yaml
kubectl get analytics -n crane-system
```
## An analytical process

```yaml title="analytics-resource.yaml" hl_lines="7 24 11-14 28-31"
apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: craned-resource
namespace: crane-system
spec:
type: Resource # This can only be "Resource" or "HPA".
completionStrategy:
completionStrategyType: Periodical # This can only be "Once" or "Periodical".
periodSeconds: 86400 # analytics selected resources every 1 day
resourceSelectors: # defines all the resources to be select with
- kind: Deployment
apiVersion: apps/v1
name: craned
1. Users create `Analytics` object and config ResourceSelector to select resources to be analyzed. Multiple types of resource selection (based on Group,Kind, and Version) are supported.
2. Analyze each selected resource in parallel and try to execute analysis and give recommendation. Each analysis process is divided into two stages: inspecting and advising:
1. Inspecting: Filter resources that don't match the recommended conditions. For example, for hpa recommendation, the workload that has many not running pod is excluded
2. Advising: Analysis and calculation based on algorithm model then provide the recommendation result.
3. If you paas the above two stages, it will create `Recommendation` object and display the result in `recommendation.Status`
4. You can find the failure reasons from `analytics.status.recommendations`
5. Wait for the next analytics based on the interval

---
## Core concept

apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: metric-adapter-resource
namespace: crane-system
spec:
type: Resource # This can only be "Resource" or "HPA".
completionStrategy:
completionStrategyType: Periodical # This can only be "Once" or "Periodical".
periodSeconds: 3600 # analytics selected resources every 1 hour
resourceSelectors: # defines all the resources to be select with
- kind: Deployment
apiVersion: apps/v1
name: metric-adapter
```
### Analytics

The output is:
Analysis defines a scanning analysis task. Two task types are supported: resource recommendation and hpa recommendation. Crane regularly runs analysis tasks and produces recommended results.

```bash
NAME AGE
craned-resource 15m
metric-adapter-resource 15m
```
### Recommendation

You can get created recommendation from analytics status:
The recommendation shows the results of an `Analytics`. The recommended result is a YAML configuration that allows users to take appropriate optimization actions, such as adjusting the resource configuration of the application.

```bash
kubectl get analytics craned-resource -n crane-system -o yaml
```
### Configuration

The output is similar to:

```yaml hl_lines="18-21"
apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: craned-resource
namespace: crane-system
spec:
completionStrategy:
completionStrategyType: Periodical
periodSeconds: 86400
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
labelSelector: {}
name: craned
type: Resource
status:
lastSuccessfulTime: "2022-01-12T08:40:59Z"
recommendations:
- name: craned-resource-resource-j7shb
namespace: crane-system
uid: 8ce2eedc-7969-4b80-8aee-fd4a98d6a8b6
```

The recommendation name presents on `status.recommendations[0].name`. Then you can get recommendation detail by running:

```bash
kubectl get recommend -n crane-system craned-resource-resource-j7shb -o yaml
```

The output is similar to:

```yaml hl_lines="32-37"
apiVersion: analysis.crane.io/v1alpha1
kind: Recommendation
metadata:
name: craned-resource-resource-j7shb
namespace: crane-system
ownerReferences:
- apiVersion: analysis.crane.io/v1alpha1
blockOwnerDeletion: false
controller: false
kind: Analytics
name: craned-resource
uid: a9e6dc0d-ab26-4f2a-84bd-4fe9e0f3e105
spec:
completionStrategy:
completionStrategyType: Periodical
periodSeconds: 86400
targetRef:
apiVersion: apps/v1
kind: Deployment
name: craned
namespace: crane-system
type: Resource
status:
conditions:
- lastTransitionTime: "2022-01-12T08:40:59Z"
message: Recommendation is ready
reason: RecommendationReady
status: "True"
type: Ready
lastSuccessfulTime: "2022-01-12T08:40:59Z"
lastUpdateTime: "2022-01-12T08:40:59Z"
resourceRequest:
containers:
- containerName: craned
target:
cpu: 114m
memory: 120586239m
```

The `status.resourceRequest` is recommended by crane's recommendation engine.

Something you should know about Resource recommendation:

* Resource Recommendation use historic prometheus metrics to calculate and propose.
* We use **Percentile** algorithm to process metrics that also used by VPA.
* If the workload is running for a long term like several weeks, the result will be more accurate.

## Analytics and Recommend HPA

Create an **HPA** `Analytics` to give recommendations for deployment: `craned` and `metric-adapter` as a sample.

```bash
kubectl apply -f https://raw.githubusercontent.com/gocrane/crane/main/examples/analytics/analytics-hpa.yaml
kubectl get analytics -n crane-system
```

```yaml title="analytics-hpa.yaml" hl_lines="7 24 11-14 28-31"
apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: craned-hpa
namespace: crane-system
spec:
type: HPA # This can only be "Resource" or "HPA".
completionStrategy:
completionStrategyType: Periodical # This can only be "Once" or "Periodical".
periodSeconds: 600 # analytics selected resources every 10 minutes
resourceSelectors: # defines all the resources to be select with
- kind: Deployment
apiVersion: apps/v1
name: craned

---

apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: metric-adapter-hpa
namespace: crane-system
spec:
type: HPA # This can only be "Resource" or "HPA".
completionStrategy:
completionStrategyType: Periodical # This can only be "Once" or "Periodical".
periodSeconds: 3600 # analytics selected resources every 1 hour
resourceSelectors: # defines all the resources to be select with
- kind: Deployment
apiVersion: apps/v1
name: metric-adapter
```


The output is:

```bash
NAME AGE
craned-hpa 5m52s
craned-resource 18h
metric-adapter-hpa 5m52s
metric-adapter-resource 18h

```

You can get created recommendation from analytics status:

```bash
kubectl get analytics craned-hpa -n crane-system -o yaml
```

The output is similar to:

```yaml hl_lines="21"
apiVersion: analysis.crane.io/v1alpha1
kind: Analytics
metadata:
name: craned-hpa
namespace: crane-system
spec:
completionStrategy:
completionStrategyType: Periodical
periodSeconds: 86400
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
labelSelector: {}
name: craned
type: HPA
status:
lastSuccessfulTime: "2022-01-13T07:26:18Z"
recommendations:
- apiVersion: analysis.crane.io/v1alpha1
kind: Recommendation
name: craned-hpa-hpa-2f22w
namespace: crane-system
uid: 397733ee-986a-4630-af75-736d2b58bfac
```

The recommendation name presents on `status.recommendations[0].name`. Then you can get recommendation detail by running:

```bash
kubectl get recommend -n crane-system craned-resource-resource-2f22w -o yaml
```

The output is similar to:

```yaml hl_lines="26-29"
apiVersion: analysis.crane.io/v1alpha1
kind: Recommendation
metadata:
name: craned-hpa-hpa-2f22w
namespace: crane-system
ownerReferences:
- apiVersion: analysis.crane.io/v1alpha1
blockOwnerDeletion: false
controller: false
kind: Analytics
name: craned-hpa
uid: b216d9c3-c52e-4c9c-b9e9-9d5b45165b1d
spec:
completionStrategy:
completionStrategyType: Periodical
periodSeconds: 86400
targetRef:
apiVersion: apps/v1
kind: Deployment
name: craned
namespace: crane-system
type: HPA
status:
conditions:
- lastTransitionTime: "2022-01-13T07:51:18Z"
message: 'Failed to offer recommend, Recommendation crane-system/craned-hpa-hpa-2f22w
error EHPAAdvisor prediction metrics data is unexpected, List length is 0 '
reason: FailedOfferRecommend
status: "False"
type: Ready
lastUpdateTime: "2022-01-13T07:51:18Z"
```

The `status.resourceRequest` is recommended by crane's recommendation engine. The fail reason is demo workload don't have enough run time.

Something you should know about HPA recommendation:

* HPA Recommendation use historic prometheus metrics to calculate, forecast and propose.
* We use **DSP** algorithm to process metrics.
* We recommend using Effective HorizontalPodAutoscaler to execute autoscaling, you can see [this document](using-time-series-prediction.md) to learn more.
* The Workload need match following conditions:
* Existing at least one ready pod
* Ready pod ratio should larger that 50%
* Must provide cpu request for pod spec
* The workload should be running for at least **a week** to get enough metrics to forecast
* The workload's cpu load should be predictable, **too low** or **too unstable** workload often is unpredictable
Different analytics uses different computing models. Crane provides a default computing model and a corresponding configuration that users can modify to customize the recommended effect. You can modify the default configuration globally or modify the configuration of a single analytics task.
Loading