Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create 8.4.manage-running-logs.md #2002

Merged
merged 1 commit into from
Mar 13, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Manage cluster logs

Running logs of NebulaGraph cluster services (graphd, metad, storaged) are generated and stored in the `/usr/local/nebula/logs` directory of each service container by default.

## View logs

To view the running logs of a NebulaGraph cluster, you can use the `kubectl logs` command.

For example, to view the running logs of the Storage service:

```bash
// View the name of the Storage service Pod, nebula-storaged-0.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nebula-exporter-84b6974497-cr54d 1/1 Running 0 43h
nebula-graphd-0 1/1 Running 0 22h
nebula-metad-0 1/1 Running 0 45h
nebula-storaged-0 1/1 Running 0 45h
...

// Enter the container storaged of the Storage service.
$ kubectl exec -it nebula-storaged-0 -c storaged -- /bin/bash

// View the running logs of the Storage service.
$ cd /usr/local/nebula/logs
```

## Clean logs

Running logs generated by cluster services during runtime will occupy disk space. To avoid occupying too much disk space, the Operator uses a sidecar container to periodically clean and archive logs.

To facilitate log collection and management, each NebulaGraph service deploys a sidecar container responsible for collecting logs generated by the service container and sending them to the specified log disk. The sidecar container automatically cleans and archives logs using the [logrotate](https://linux.die.net/man/8/logrotate) tool.

In the YAML configuration file of the cluster instance, you can configure log rotation to automatically clean and archive logs through the `spec.logRotate` field. By default, the log rotation feature is turned off. Here is an example of enabling log rotation:

```yaml
...
spec:
graphd:
config:
# Whether to include a timestamp in the log file name. "true" means yes, "false" means no.
"timestamp_in_logfile_name": "false"
metad:
config:
"timestamp_in_logfile_name": "false"
storaged:
config:
"timestamp_in_logfile_name": "false"
logRotate: # Log rotation configuration
# The number of times a log file is rotated before being deleted. The default value is 5, and 0 means the log file will not be rotated before being deleted.
rotate: 5
# The log file is rotated only if it grows larger than the specified size. The default value is 200M.
size: "200M"
```

## Collect logs

If you don't want to mount additional log disks to back up log files, or if you want to collect logs and send them to a log center using services like [fluent-bit](https://fluentbit.io/), you can configure logs to be output to standard error. The Operator uses the [glog](https://github.com/google/glog) tool to log to standard error output.

!!! caution

Currently, NebulaGraph Operator only collects standard error logs.。

In the YAML configuration file of the cluster instance, you can configure logging to standard error output in the `config` and `env` fields of each service.


```yaml
...
spec:
graphd:
config:
# Whether to redirect standard error to a separate output file. The default value is false, which means it is not redirected.
redirect_stdout: "false"
# The severity level of log content: INFO, WARNING, ERROR, and FATAL. The corresponding values are 0, 1, 2, and 3.
stderrthreshold: "0"
env:
- name: GLOG_logtostderr # Write log to standard error output instead of a separate file.
value: "1" # 1 represents writing to standard error output, and 0 represents writing to a file.
image: vesoft/nebula-graphd
replicas: 1
resources:
requests:
cpu: 500m
memory: 500Mi
service:
externalTrafficPolicy: Local
type: NodePort
version: v{{nebula.release}}
metad:
config:
redirect_stdout: "false"
stderrthreshold: "0"
dataVolumeClaim:
resources:
requests:
storage: 1Gi
storageClassName: ebs-sc
env:
- name: GLOG_logtostderr
value: "1"
image: vesoft/nebula-metad
...
```