Use Postgresql as an alternative storage for job history #4164

hzy46 · 2020-01-23T03:24:31Z

How to use

The following service config should be set to enable internal storage, postgresql, and set rest-server to read job history from database.

cluster:
  common:
    job-history: "true"
    job-attempt-storage-engine: "db"

internal-storage:
  root-path: /paiInternal
  enable: true

postgresql:
  enable: true

What will happen if the database crashes

In such case, the health check of job attempt api will not pass. User won't be able to click it.

In backend side, fluentd will try to send chunk to postgresql forever. Once the postgresql db is restarted, the job attempt history will be recovered.

Test

Check if the job history works normally
- Submit a special job, which will fail after 60 seconds: sleep 60s; echo failed at $(date); exit 233, and set a retry numbers for it.
- Expect: all the job retry history will be recorded normally.
Check the behavior when the database is down
- Delete the database service by ./paictl.py service stop -m postgresql
- Submit a job with retries, now the user will not be able to see the retry history
- Start the database service by ./paictl.py service start -m postgresql
- Expect: all the job retry history will be recorded normally after a few minutes

…esql_job_history

yqwang-ms · 2020-03-12T02:52:06Z

src/fluentd/deploy/fluentd-config.yaml.template

-      <format>
-        @type json
-      </format>
-    </filter>


Why remote it? #Closed

Actually this log makes my debugging harder since fc-objectsnapshots is very huge, which makes other userful logs difficult to see. If it is necessary, I can revert this change. #Closed

If so let's just comment it out,

In reply to: 391375514 [](ancestors = 391375514)

If so let's just comment it out,

In reply to: 391375514 [](ancestors = 391375514)

OK, done. #Closed

yqwang-ms · 2020-03-12T02:52:51Z

src/fluentd/deploy/fluentd-config.yaml.template

@@ -143,4 +135,26 @@ data:
          overflow_action block
        </buffer>
      </store>
+{% if (cluster_cfg['internal-storage']['enable']) and (cluster_cfg['postgresql']['enable']) %}
+      <store>


Also and IF for elasticsearch? #Closed

Done #Closed

yqwang-ms · 2020-03-12T06:21:44Z

src/fluentd/build/fluentd.k8s.dockerfile

+ && sudo gem install rake bundler pg
+
+# Build fluent-plugin-pgjson from scratch
+# Original fluent-plugin-pgjson is from https://github.com/fluent-plugins-nursery/fluent-plugin-pgjson


Comment why we need build our own

yqwang-ms · 2020-03-12T06:23:19Z

src/fluentd/deploy/fluentd-config.yaml.template

@@ -143,4 +144,27 @@ data:
          overflow_action block
        </buffer>
      </store>
+{% endif %}
+{% if (cluster_cfg['internal-storage']['enable']) and (cluster_cfg['postgresql']['enable']) %}


seems you can use if else...

Better use something like:
job-attempt-storage-engine: "elasticsearch" # set it to "db" to use database backend

In reply to: 391421409 [](ancestors = 391421409)

use cluster_cfg['cluster']['common']['job-attempt-storage-engine'] instead

yqwang-ms

debuggy

approve with minor comment

src/rest-server/src/utils/postgresUtil.js

hzy46 added 10 commits January 22, 2020 14:14

fix table schema & fluentd

fad4831

fix fluentd

4560cc7

fix fluentd

b9bd862

fix fluentd

d43c441

fix rest-server

0b30dd4

lint

260f143

fix rest-server

7e4d297

fix rest-server

1791888

fix index setting

ce5a50a

Merge branch 'master' of github.com:microsoft/pai into zhiyuhe/postgr…

7ee4a26

…esql_job_history

hzy46 requested review from debuggy, sunqinzheng and yqwang-ms January 23, 2020 03:24

hzy46 added 7 commits February 3, 2020 10:27

lint

c712026

rm header

15bcfa0

add postgresql init container

8ab2466

fix

2e78d90

fix

f4e3317

add readiness probe

89aded7

add time period for probe

9fe698e

hzy46 mentioned this pull request Feb 3, 2020

P0.5 - elastic search alternative option (Azure search or DB search as switchable options) #4028

Closed

scarlett2018 mentioned this pull request Feb 3, 2020

Pure K8S Beta Release Plan - v0.17 #3872

Closed

54 tasks

hzy46 mentioned this pull request Feb 20, 2020

PAI 0.18.0 Release and Endgame Plan #4225

Closed

74 tasks

hzy46 added 5 commits February 25, 2020 10:54

Merge branch 'master' of github.com:microsoft/pai into zhiyuhe/postgr…

fe24c56

…esql_job_history

fix for new build

f71cef2

fix

0c2fb46

Merge branch 'master' of github.com:microsoft/pai into zhiyuhe/postgr…

fb3cec7

…esql_job_history

fix

56a646a

hzy46 force-pushed the zhiyuhe/postgresql_job_history branch from 30ef3a8 to 6bf6184 Compare March 11, 2020 10:29

fix bad connection problem in fluent-plugin-pgjson

7451a1c

hzy46 force-pushed the zhiyuhe/postgresql_job_history branch from 8323ae1 to 7451a1c Compare March 11, 2020 15:37

yqwang-ms reviewed Mar 12, 2020

View reviewed changes

hzy46 added 4 commits March 12, 2020 11:20

add if for elasticsearch

8468816

fix

4fff748

add NOTICE.txt

8bb09c1

comment out debug stdout

b5f5be4

yqwang-ms reviewed Mar 12, 2020

View reviewed changes

hzy46 requested a review from qfyin March 12, 2020 06:21

yqwang-ms reviewed Mar 12, 2020

View reviewed changes

add comment

9124d03

yqwang-ms approved these changes Mar 12, 2020

View reviewed changes

change config way

889c73a

debuggy approved these changes Mar 16, 2020

View reviewed changes

src/rest-server/src/utils/postgresUtil.js Outdated Show resolved Hide resolved

hzy46 added 2 commits March 16, 2020 18:07

fix

c28dbd0

copyright header

90b4894

hzy46 merged commit 2cb5a3c into master Mar 19, 2020

hzy46 mentioned this pull request Apr 7, 2020

[Error]rest-server error in depolyment #3304

Closed

hzy46 mentioned this pull request Apr 20, 2020

OpenPAI v1.0.0 (a.k.a 0.18.0) Endgame Plan #4315

Closed

36 tasks

debuggy mentioned this pull request Apr 23, 2020

test cases of bug bash microsoft/openpaimarketplace#20

Closed

11 tasks

hzy46 mentioned this pull request Apr 27, 2020

add release note #4452

Merged

hzy46 mentioned this pull request Jul 7, 2020

Feedback v0.17.0 and v1.0 deployment issues met #4675

Open

hzy46 deleted the zhiyuhe/postgresql_job_history branch September 3, 2020 07:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Postgresql as an alternative storage for job history #4164

Use Postgresql as an alternative storage for job history #4164

hzy46 commented Jan 23, 2020 •

edited

Loading

yqwang-ms Mar 12, 2020 •

edited

Loading

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

yqwang-ms Mar 12, 2020

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

yqwang-ms Mar 12, 2020 •

edited

Loading

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

yqwang-ms Mar 12, 2020

hzy46 Mar 12, 2020

yqwang-ms Mar 12, 2020

yqwang-ms Mar 12, 2020

hzy46 Mar 12, 2020 •

edited

Loading

yqwang-ms left a comment

debuggy left a comment

Use Postgresql as an alternative storage for job history #4164

Use Postgresql as an alternative storage for job history #4164

Conversation

hzy46 commented Jan 23, 2020 • edited Loading

What will happen if the database crashes

Test

yqwang-ms Mar 12, 2020 • edited Loading

Choose a reason for hiding this comment

hzy46 Mar 12, 2020 • edited by yqwang-ms Loading

Choose a reason for hiding this comment

yqwang-ms Mar 12, 2020

Choose a reason for hiding this comment

hzy46 Mar 12, 2020 • edited by yqwang-ms Loading

Choose a reason for hiding this comment

yqwang-ms Mar 12, 2020 • edited Loading

Choose a reason for hiding this comment

hzy46 Mar 12, 2020 • edited by yqwang-ms Loading

Choose a reason for hiding this comment

yqwang-ms Mar 12, 2020

Choose a reason for hiding this comment

hzy46 Mar 12, 2020

Choose a reason for hiding this comment

yqwang-ms Mar 12, 2020

Choose a reason for hiding this comment

yqwang-ms Mar 12, 2020

Choose a reason for hiding this comment

hzy46 Mar 12, 2020 • edited Loading

Choose a reason for hiding this comment

yqwang-ms left a comment

Choose a reason for hiding this comment

debuggy left a comment

Choose a reason for hiding this comment

hzy46 commented Jan 23, 2020 •

edited

Loading

yqwang-ms Mar 12, 2020 •

edited

Loading

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

yqwang-ms Mar 12, 2020 •

edited

Loading

hzy46 Mar 12, 2020 •

edited by yqwang-ms

Loading

hzy46 Mar 12, 2020 •

edited

Loading