Skip to content

Latest commit

 

History

History
193 lines (132 loc) · 9.63 KB

log_frwd_log_anlys.md

File metadata and controls

193 lines (132 loc) · 9.63 KB
copyright lastupdated subcollection
years
2017, 2024
2024-01-31
AnalyticsEngine

{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:codeblock: .codeblock} {:screen: .screen} {:pre: .pre} {:external: target="_blank" .external}

Forwarding logs to IBM Cloud Logs

{: #platform-logs}

Platform logs are logs that are exposed to users by various services on the {{site.data.keyword.cloud_notm}} platform. {{site.data.keyword.iae_full_notm}} supports forwarding logs from the {{site.data.keyword.iae_full_notm}} service to an IBM Cloud Logs instance that you have enabled to receive platform logs.

  • Platform logs are regional.

    You can monitor logs from {{site.data.keyword.iae_full_notm}} on {{site.data.keyword.cloud_notm}} in the region where the service is available.

  • You can configure only one instance of IBM Cloud Logs per region and per {{site.data.keyword.cloud_notm}} account to collect platform logs in that location.

    You can have multiple IBM Cloud Logs instances in a location. However, only one instance in a location (region) can be configured to receive logs from {{site.data.keyword.cloud_notm}} services such as {{site.data.keyword.iae_full_notm}} in that {{site.data.keyword.cloud_notm}} location.

    Also, the logs that are generated by an {{site.data.keyword.iae_full_notm}} instance belonging to an account are forwarded to an IBM Cloud Logs instance under the same {{site.data.keyword.cloud_notm}} account.

To check for the IBM Cloud Logs service that is configured to receive platform logs, see the Observability dashboard.

Prerequisite to enable platform logging

{: #enable-logging-from-dashboard}

To view {{site.data.keyword.iae_full_notm}} platform logs, you must use the Observability dashboard in {{site.data.keyword.cloud_notm}} to configure platform logging. See Configuring platform logs through the Observability dashboard for the steps that you need to follow to enable logging through the Observability dashboard.

Logging APIs

{: #logging-apis}

After you have enabled platform logging through the Observability dashboard, proceed with the steps in the following sections to learn how to use the API endpoints to enable logging, disable logging, and view the logging configuration.

The logging API calls require the GUID of the service instance. If you didn't make a note of the GUID, see Retrieving the GUID of a serverless instance.

Now, two logging API endpoints are exposed:

  • log_forwarding_config
  • logging. This API endpoint is deprecated and will be discontinued soon. Although you may use this endpoint, you should start using the log_forwarding_config endpoint.

Enabling log forwarding from {{site.data.keyword.iae_full_notm}}

{: #enable-logging-from-iae}

To enable forwarding {{site.data.keyword.iae_full_notm}} platform logs to IBM Cloud Logs, invoke the following API against an existing {{site.data.keyword.iae_short}} instance.

There are two methods that you can use to enable forwarding logs:

  • Default: In this case you only put a request for enabling log forwarding, and the logs related to the default log source or sources (spark-driver), will start getting forwarded. You can do this in one of the following ways:

    • By using the log_forwarding_config API endpoint:

      curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":true}'

      {: codeblock}

    • By using the logging API endpoint (deprecated):

      curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":true}

      {: codeblock}

  • Customized: With this option, you get more control over the log forwarding configuration. For example, you can specify the sources of the logs you want to get forwarded. You can also choose to associate some tags with the logs, which can later be used in IBM Cloud Logs as search keywords to narrow down the search on the applicable logs. You can do this in one of the following ways:

    • By using the log_forwarding_config API endpoint:

      curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enabled":true,"sources": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '

      {: codeblock}

    • By using the logging API endpoint (deprecated):

      curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enable":true,"components": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '

      {: codeblock}

Disabling platform logging from {{site.data.keyword.iae_full_notm}}

{: #disable-loggingfrom-iae}

Disable forwarding {{site.data.keyword.iae_full_notm}} platform logs to IBM Cloud Logs, in one of the following ways.

  • By using the log_forwarding_config API endpoint:

    curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":false}'

    {: codeblock}

  • By using the logging API endpoint (deprecated):

    curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":false}'

    {: codeblock}

Viewing the log forwarding configuration of an {{site.data.keyword.iae_full_notm}} instance

{: #log-config}

View the current logging configuration of an {{site.data.keyword.iae_full_notm}} instance, in one of the following ways:

  • By using the log_forwarding_config API endpoint:

    curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN"

    {: codeblock}

  • By using the logging API endpoint (deprecated):

    curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN"

    {: codeblock}

Examples of {{site.data.keyword.iae_full_notm}} log searches in the IBM Cloud Logs dashboard

{: #analyzing-logs}

You can narrow down the search for the platform logs related to {{site.data.keyword.iae_full_notm}} as follows:

  1. Click Sources on IBM Cloud Logs dashboard and choose ibmanalyticsengine to get the logs from {{site.data.keyword.iae_full_notm}}.
  2. To further narrow down your search, you can enter and search for any keywords related to your Spark application in the search window.

Examples of searches

{: #analyzing-logs-1}

This section lists examples of how you can search for logs. The examples all use the following sample values:

  • IBM Analytics Engine instance ID: 4308f7a6-2cfd-4d7e-b7a7-678e71a27957
  • Spark application ID: 5438585f-6413-4bb2-9251-8162d34f3dc3
  • Sources for which log forwarding is enabled: spark-driver and spark-executor
  • Associated tags: prod and us-south

The following image shows you an example of the returned search results in the IBM Cloud Logs dashboard. At the bottom of the image you can see the search entry field where you enter your search queries. In the example, the application ID 5438585f-6413-4bb2-9251-8162d34f3dc3 and the source spark-driver were entered.

Examples of searches:

  • To retrieve the logs for all Spark applications submitted in a specific {{site.data.keyword.iae_full_notm}} instance, pass the instance ID in the search window:

    4308f7a6-2cfd-4d7e-b7a7-678e71a27957

  • To fetch the logs for a specific Spark application, you can pass:

    • A combination of the {{site.data.keyword.iae_full_notm}} instance ID and the Spark application ID: 
      4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3
    • Only the application ID: 
      5438585f-6413-4bb2-9251-8162d34f3dc3

  • To search the Spark master logs of an application, you can pass:

    • A combination of the {{site.data.keyword.iae_full_notm}} instance ID, the Spark application ID and the keyword spark-driver: 
      4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver
    • A combination of the application ID and the keyword spark-driver: 
      5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver

  • To search by multiple log source names, you can pass:

    • A combination of the {{site.data.keyword.iae_full_notm}} instance ID, the Spark application ID and the source names: 
      4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)
    • A combination of the application ID and the log source names: 
      5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)

  • To get the logs related to all the applications running in a particular instance within the us-south region and the prod environment, you can enter the following in the search field:

    4308f7a6-2cfd-4d7e-b7a7-678e71a27957 "prod us-south"