-
Notifications
You must be signed in to change notification settings - Fork 2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Monitor Query] Add samples/docs for running big queries and overcomi…
…ng API limits for size and record count (#35704) * Documentation updates for large queries * Large Query Documentation * Large Query Documentation updates * Large Log Query Documentation * Large Log Query Documentation * Remove additional test * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/LargeQuerySample.java Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> * Update sdk/monitor/azure-monitor-query/LargeLogQuery.md Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> * Updating Large Log Query Documentation * Updating Large Query Sample * Updating Large Query Sample PR * Updating Large Query Sample PR * Updating Large Query Sample PR * Updating Large Query Sample PR * Update sdk/monitor/azure-monitor-query/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/OvercomingLargeQueryLimitationsSample.java Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Updating Samples README.md * Updating Samples README.md * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/OvercomingLargeQueryLimitationsSample.java Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/OvercomingLargeQueryLimitationsSample.java Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/README.md Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/OvercomingLargeQueryLimitationsSample.java Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/OvercomingLargeQueryLimitationsSample.java Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com> * Updating the large query documentation PR * Updating the large query documentation PR * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/SplitQueryByByteSample.java Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Update sdk/monitor/azure-monitor-query/src/samples/java/com/azure/monitor/query/SplitQueryByRowSample.java Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> * Updating the large query documentation PR and adding more documentation to QueryTimeInterval * Updating the large query documentation PR --------- Co-authored-by: Liudmila Molkova <limolkova@microsoft.com> Co-authored-by: Srikanta <51379715+srnagar@users.noreply.github.com> Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com>
- Loading branch information
4 people
authored
Sep 7, 2023
1 parent
f84db5b
commit 8b00302
Showing
5 changed files
with
297 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
142 changes: 142 additions & 0 deletions
142
...re-monitor-query/src/samples/java/com/azure/monitor/query/SplitQueryByByteSizeSample.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT License. | ||
|
|
||
| package com.azure.monitor.query; | ||
|
|
||
| import java.time.OffsetDateTime; | ||
| import java.util.ArrayList; | ||
| import java.util.Collections; | ||
| import java.util.List; | ||
|
|
||
| import com.azure.core.util.Configuration; | ||
| import com.azure.identity.DefaultAzureCredentialBuilder; | ||
| import com.azure.monitor.query.models.LogsBatchQuery; | ||
| import com.azure.monitor.query.models.LogsBatchQueryResult; | ||
| import com.azure.monitor.query.models.LogsQueryResult; | ||
| import com.azure.monitor.query.models.LogsTable; | ||
| import com.azure.monitor.query.models.LogsTableCell; | ||
| import com.azure.monitor.query.models.LogsTableColumn; | ||
| import com.azure.monitor.query.models.LogsTableRow; | ||
| import com.azure.monitor.query.models.QueryTimeInterval; | ||
|
|
||
| /** | ||
| * A sample to demonstrate using a batch logs query to overcome the service limits for a large query. | ||
| */ | ||
| public class SplitQueryByByteSizeSample { | ||
|
|
||
|
|
||
| static String workspaceId; | ||
| static LogsQueryClient client; | ||
|
|
||
| /** | ||
| * The main method to execute the sample. | ||
| * @param args Ignored args. | ||
| */ | ||
| public static void main(String[] args) { | ||
|
|
||
| String queryString = "AppRequests"; | ||
| workspaceId = Configuration.getGlobalConfiguration().get("AZURE_MONITOR_LOGS_WORKSPACE_ID"); | ||
|
|
||
| int maxByteSizePerBatch = 1024 * 1024 * 10; // 10 MB | ||
|
|
||
| client = new LogsQueryClientBuilder() | ||
| .credential(new DefaultAzureCredentialBuilder().build()) | ||
| .buildClient(); | ||
|
|
||
|
|
||
| // Running a log batch query with a byte size limit on each batch | ||
| LogsBatchQuery byteBasedBatchQuery = createBatchQueryFromTimeRanges(queryString, | ||
| createQueryTimeIntervalsForBatchQueryByByteSize(queryString, maxByteSizePerBatch)); | ||
|
|
||
| // The result of the byte split query | ||
| List<LogsBatchQueryResult> byteLimitedResults = client.queryBatch(byteBasedBatchQuery).getBatchResults(); | ||
|
|
||
| // consolidate the results from the batch query | ||
| LogsQueryResult combineByteBasedQuery = combineResults(byteLimitedResults); | ||
| } | ||
|
|
||
| /** | ||
| * Helper method to create a batch query from a query string and a list of time intervals. | ||
| * | ||
| * @param originalQuery The original query string. | ||
| * @param queryTimeIntervals The list of time intervals. | ||
| * @return A {@link LogsBatchQuery} object equivalent to the original query. | ||
| */ | ||
| static LogsBatchQuery createBatchQueryFromTimeRanges(String originalQuery, | ||
| List<QueryTimeInterval> queryTimeIntervals) { | ||
| LogsBatchQuery batchQuery = new LogsBatchQuery(); | ||
|
|
||
| for (QueryTimeInterval timeInterval : queryTimeIntervals) { | ||
| batchQuery.addWorkspaceQuery(workspaceId, originalQuery, timeInterval); | ||
| } | ||
|
|
||
| return batchQuery; | ||
| } | ||
|
|
||
|
|
||
| /** | ||
| * Helper method to create a list of time intervals for a batch query based on the byte size limit. | ||
| * | ||
| * @param originalQuery The original query string. | ||
| * @param maxByteSizePerBatch The maximum byte size per batch. If multiple log entries returned in the original | ||
| * query have the exact same timestamp, the byte size per batch may exceed this limit. | ||
| * @return A list of {@link QueryTimeInterval} objects. | ||
| */ | ||
| static List<QueryTimeInterval> createQueryTimeIntervalsForBatchQueryByByteSize(String originalQuery, | ||
| int maxByteSizePerBatch) { | ||
| /* | ||
| * This query finds the start time of each batch extending the query with a batch_num column that determined | ||
| * using the estimate_data_size() function. The estimate_data_size() function returns the estimated byte size of | ||
| * each row. The batch_num is calculated by dividing the cumulative sum of the byte size of each row by the max | ||
| * amount of bytes per row. The batchStart is then calculated by finding the minimum time for each batch_num. | ||
| * The batchStart times are then sorted and projected as the result of the query. | ||
| */ | ||
| String findBatchEndpointsQuery = String.format( | ||
| "%1$s | sort by TimeGenerated desc | extend batch_num = row_cumsum(estimate_data_size(*)) / %2$s | summarize batchStart=min(TimeGenerated) by batch_num | sort by batch_num desc | project batchStart", | ||
| originalQuery, | ||
| maxByteSizePerBatch); | ||
|
|
||
| LogsQueryResult result = client.queryWorkspace(workspaceId, findBatchEndpointsQuery, QueryTimeInterval.ALL); | ||
| List<LogsTableRow> rows = result.getTable().getRows(); | ||
| List<OffsetDateTime> offsetDateTimes = new ArrayList<>(); | ||
| List<QueryTimeInterval> queryTimeIntervals = new ArrayList<>(); | ||
|
|
||
| for (LogsTableRow row : rows) { | ||
| row.getColumnValue("batchStart").ifPresent(rowValue -> { | ||
| offsetDateTimes.add(rowValue.getValueAsDateTime()); | ||
| }); | ||
| } | ||
|
|
||
|
|
||
| for (int i = 0; i < offsetDateTimes.size(); i++) { | ||
| OffsetDateTime startTime = offsetDateTimes.get(i); | ||
| OffsetDateTime endTime = i == offsetDateTimes.size() - 1 ? OffsetDateTime.now() : offsetDateTimes.get(i + 1); | ||
| QueryTimeInterval timeInterval = new QueryTimeInterval(startTime, endTime); | ||
| queryTimeIntervals.add(timeInterval); | ||
| } | ||
|
|
||
| return queryTimeIntervals; | ||
| } | ||
|
|
||
|
|
||
| /** | ||
| * This method simulates the result of a single query from the results of a batch query by combining lists of | ||
| * log tables from each batch query result. It is intended only for batch queries resulting from a split single | ||
| * query. It is not intended to be used on queries containing statistics, visualization data, or errors. | ||
| * @param batchQueryResults The results (lists of log tables) from the split single query. | ||
| * @return The result (log tables) in the form of a single query. | ||
| */ | ||
| static LogsQueryResult combineResults(List<LogsBatchQueryResult> batchQueryResults) { | ||
| List<LogsTableCell> logsTablesCells = new ArrayList<>(); | ||
| List<LogsTableRow> logsTablesRows = new ArrayList<>(); | ||
| List<LogsTableColumn> logsTablesColumns = new ArrayList<>(); | ||
| for (LogsBatchQueryResult batchQueryResult : batchQueryResults) { | ||
| for(LogsTable logsTable: batchQueryResult.getAllTables()) { | ||
| logsTablesCells.addAll(logsTable.getAllTableCells()); | ||
| logsTablesRows.addAll(logsTable.getRows()); | ||
| logsTablesColumns.addAll(logsTable.getColumns()); | ||
| } | ||
| } | ||
| return new LogsQueryResult(Collections.singletonList(new LogsTable(logsTablesCells, logsTablesRows, logsTablesColumns)), null, null, null); | ||
| } | ||
| } |
127 changes: 127 additions & 0 deletions
127
...re-monitor-query/src/samples/java/com/azure/monitor/query/SplitQueryByRowCountSample.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT License. | ||
|
|
||
| package com.azure.monitor.query; | ||
|
|
||
| import com.azure.core.util.Configuration; | ||
| import com.azure.identity.DefaultAzureCredentialBuilder; | ||
| import com.azure.monitor.query.models.LogsBatchQuery; | ||
| import com.azure.monitor.query.models.LogsBatchQueryResult; | ||
| import com.azure.monitor.query.models.LogsQueryResult; | ||
| import com.azure.monitor.query.models.LogsTable; | ||
| import com.azure.monitor.query.models.LogsTableCell; | ||
| import com.azure.monitor.query.models.LogsTableColumn; | ||
| import com.azure.monitor.query.models.LogsTableRow; | ||
| import com.azure.monitor.query.models.QueryTimeInterval; | ||
|
|
||
| import java.time.OffsetDateTime; | ||
| import java.util.ArrayList; | ||
| import java.util.Collections; | ||
| import java.util.List; | ||
|
|
||
| public class SplitQueryByRowCountSample { | ||
|
|
||
| static String workspaceId; | ||
| static LogsQueryClient client; | ||
|
|
||
| /** | ||
| * The main method to execute the sample. | ||
| * @param args Ignored args. | ||
| */ | ||
| public static void main(String[] args) { | ||
| String queryString = "AppRequests"; | ||
| workspaceId = Configuration.getGlobalConfiguration().get("AZURE_MONITOR_LOGS_WORKSPACE_ID"); | ||
|
|
||
| // These values are for demonstration purposes only. Please set to the appropriate values for your use case. | ||
| int maxRowsPerBatch = 100; // 100 rows. The service maximum is 500,000 rows per query | ||
|
|
||
| client = new LogsQueryClientBuilder() | ||
| .credential(new DefaultAzureCredentialBuilder().build()) | ||
| .buildClient(); | ||
|
|
||
| // Running a log batch query with a row count limit on each batch | ||
| LogsBatchQuery rowBasedBatchQuery = createBatchQueryFromTimeRanges(queryString, | ||
| createQueryTimeIntervalsForBatchQueryByRowCount(queryString, maxRowsPerBatch)); | ||
|
|
||
| // The result of the row split query | ||
| List<LogsBatchQueryResult> rowLimitedResults = client.queryBatch(rowBasedBatchQuery).getBatchResults(); | ||
|
|
||
| // consolidate the results from the batch query | ||
| LogsQueryResult combineRowBasedQuery = combineResults(rowLimitedResults); | ||
| } | ||
|
|
||
| /** | ||
| * Helper method to create a batch query from a query string and a list of time intervals. | ||
| * | ||
| * @param originalQuery The original query string. | ||
| * @param queryTimeIntervals The list of time intervals. | ||
| * @return A {@link LogsBatchQuery} object equivalent to the original query. | ||
| */ | ||
| static LogsBatchQuery createBatchQueryFromTimeRanges(String originalQuery, | ||
| List<QueryTimeInterval> queryTimeIntervals) { | ||
| LogsBatchQuery batchQuery = new LogsBatchQuery(); | ||
|
|
||
| for (QueryTimeInterval timeInterval : queryTimeIntervals) { | ||
| batchQuery.addWorkspaceQuery(workspaceId, originalQuery, timeInterval); | ||
| } | ||
|
|
||
| return batchQuery; | ||
| } | ||
|
|
||
| /** | ||
| * Helper method to create a list of time intervals for a batch query based on the row count limit. | ||
| * | ||
| * @param originalQuery The original query string. | ||
| * @param maxRowPerBatch The maximum row count per batch. If multiple log entries returned in the original query | ||
| * have the exact same timestamp, the row count per batch may exceed this limit. | ||
| * @return A list of {@link QueryTimeInterval} objects. | ||
| */ | ||
| static List<QueryTimeInterval> createQueryTimeIntervalsForBatchQueryByRowCount(String originalQuery, | ||
| int maxRowPerBatch) { | ||
|
|
||
| /* | ||
| * This query finds the start time of each batch. The batch number is calculated by dividing the cumulative row | ||
| * count at each row by the max row count per batch. The batch start time is the minimum time generated for each | ||
| * batch number. The batch numbers are then sorted and projected as the result of the query. | ||
| */ | ||
| String findBatchEndpointsQuery = String.format( | ||
| "%1$s | sort by TimeGenerated desc | extend batch_num = row_cumsum(1) / %2$s | summarize batchStart=min(TimeGenerated) by batch_num | sort by batch_num desc | project batchStart", | ||
| originalQuery, | ||
| maxRowPerBatch); | ||
|
|
||
| LogsQueryResult result = client.queryWorkspace(workspaceId, findBatchEndpointsQuery, QueryTimeInterval.ALL); | ||
| List<LogsTableRow> rows = result.getTable().getRows(); | ||
| List<OffsetDateTime> offsetDateTimes = new ArrayList<>(); | ||
| List<QueryTimeInterval> queryTimeIntervals = new ArrayList<>(); | ||
|
|
||
| for (LogsTableRow row : rows) { | ||
| row.getColumnValue("batchStart").ifPresent(rowValue -> { | ||
| offsetDateTimes.add(rowValue.getValueAsDateTime()); | ||
| }); | ||
| } | ||
|
|
||
|
|
||
| for (int i = 0; i < offsetDateTimes.size(); i++) { | ||
| OffsetDateTime startTime = offsetDateTimes.get(i); | ||
| OffsetDateTime endTime = i == offsetDateTimes.size() - 1 ? OffsetDateTime.now() : offsetDateTimes.get(i + 1); | ||
| QueryTimeInterval timeInterval = new QueryTimeInterval(startTime, endTime); | ||
| queryTimeIntervals.add(timeInterval); | ||
| } | ||
|
|
||
| return queryTimeIntervals; | ||
| } | ||
|
|
||
| static LogsQueryResult combineResults(List<LogsBatchQueryResult> batchQueryResults) { | ||
| List<LogsTableCell> logsTablesCells = new ArrayList<>(); | ||
| List<LogsTableRow> logsTablesRows = new ArrayList<>(); | ||
| List<LogsTableColumn> logsTablesColumns = new ArrayList<>(); | ||
| for (LogsBatchQueryResult batchQueryResult : batchQueryResults) { | ||
| for(LogsTable logsTable: batchQueryResult.getAllTables()) { | ||
| logsTablesCells.addAll(logsTable.getAllTableCells()); | ||
| logsTablesRows.addAll(logsTable.getRows()); | ||
| logsTablesColumns.addAll(logsTable.getColumns()); | ||
| } | ||
| } | ||
| return new LogsQueryResult(Collections.singletonList(new LogsTable(logsTablesCells, logsTablesRows, logsTablesColumns)), null, null, null); | ||
| } | ||
| } |