[ML-DataFrame] Checkpoint information is wrong for stopped dataframes

[hendrikmuhs]

For stopped data frames the checkpoint information is taken from the stored stats document in the internal index rather than calculated. The stored information is always empty, therefore it always reports 0:

     "checkpointing" : {
        "operations_behind" : 0
      }

Solution:

• calculate checkpoint information the same ways as it's done for started data frames
• do not persist checkpoint information in the state-and-stats document in the internal index

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

I think this is a prerequisite for #43767, and hence elastic/kibana#40378.

[sophiec20]

A stopped continuous data frame now returns useful info. Can this be closed?

{
  "count" : 1,
  "transforms" : [
    {
      "id" : "simple-01",
      "state" : "stopped",
      "stats" : {
        "pages_processed" : 27003,
        "documents_processed" : 437684532,
        "documents_indexed" : 4291370,
        "trigger_count" : 4289,
        "index_time_in_ms" : 2274847,
        "index_total" : 8805,
        "index_failures" : 0,
        "search_time_in_ms" : 40555084,
        "search_total" : 27004,
        "search_failures" : 0,
        "exponential_avg_checkpoint_duration_ms" : 65092.07653134306,
        "exponential_avg_documents_indexed" : 7297.119039406083,
        "exponential_avg_documents_processed" : 746541.5988733966
      },
      "checkpointing" : {
        "last" : {
          "checkpoint" : 633,
          "timestamp" : "2019-08-22T12:05:36.604Z",
          "timestamp_millis" : 1566475536604,
          "time_upper_bound" : "2019-08-22T12:04:36.604Z",
          "time_upper_bound_millis" : 1566475476604
        },
        "next" : {
          "checkpoint" : 634,
          "position" : {
            "bucket_position" : {
              "clientip" : "10.11.122.11"
            }
          },
          "checkpoint_progress" : {
            "docs_indexed" : 2000,
            "docs_processed" : 210542
          },
          "timestamp" : "2019-08-22T12:06:54.717Z",
          "timestamp_millis" : 1566475614717,
          "time_upper_bound" : "2019-08-22T12:05:54.717Z",
          "time_upper_bound_millis" : 1566475554717
        },
        "operations_behind" : 15996
      }
    }
  ]
}

[benwtrent]

@sophiec20 definitely, looking at @droberts195 PR, there was logic that gathered checkpointing information for stopped transforms that did not have actively running tasks. Should be good to go now.