Skip to content

Commit

Permalink
Final consistency review of Monitor Query README (#20667)
Browse files Browse the repository at this point in the history 
* Final consistency review of Monitor Query README

* Add rate limits section
  • Loading branch information
@scottaddie
scottaddie committed Sep 13, 2021
1 parent c4d64b2 commit ce307fe
Showing 1 changed file with 113 additions and 123 deletions.
236 changes: 113 additions & 123 deletions sdk/monitor/azure-monitor-query/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,38 +1,43 @@
# Azure Monitor Query client library for Python

Azure Monitor helps you maximize the availability and performance of your apps. It delivers a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
The Azure Monitor Query client library is used to execute read-only queries against [Azure Monitor][azure_monitor_overview]'s two data platforms:

All data collected by Azure Monitor fits into one of two fundamental types:
- [Logs](https://docs.microsoft.com/azure/azure-monitor/logs/data-platform-logs) - Collects and organizes log and performance data from monitored resources. Data from different sources such as platform logs from Azure services, log and performance data from virtual machines agents, and usage and performance data from apps can be consolidated into a single [Azure Log Analytics workspace](https://docs.microsoft.com/azure/azure-monitor/logs/data-platform-logs#log-analytics-workspaces). The various data types can be analyzed together using the [Kusto Query Language][kusto_query_language].
- [Metrics](https://docs.microsoft.com/azure/azure-monitor/essentials/data-platform-metrics) - Collects numeric data from monitored resources into a time series database. Metrics are numerical values that are collected at regular intervals and describe some aspect of a system at a particular time. Metrics are lightweight and capable of supporting near real-time scenarios, making them particularly useful for alerting and fast detection of issues.

- **Metrics** - Numerical values that describe some aspect of a system at a particular time. They're lightweight and can support near real-time scenarios.
- **Logs** - Disparate types of data organized into records with different sets of properties for each type. Performance data and telemetry such as events, exceptions, and traces are stored as logs.
**Resources:**

To programmatically analyze these data sources, the Azure Monitor Query client library can be used.

[Source code][python-query-src] | [Package (PyPI)][python-query-pypi] | [API reference documentation][python-query-ref-docs] | [Product documentation][python-query-product-docs] | [Samples][python-query-samples] | [Changelog][python-query-changelog]
- [Source code][source]
- [Package (PyPI)][package]
- [API reference documentation][python-query-ref-docs]
- [Service documentation][azure_monitor_overview]
- [Samples][samples]
- [Change log][changelog]

## Getting started

### Prerequisites

- Python 2.7, or 3.6 or later.
- An [Azure subscription][azure_subscription].
- Python 2.7, or 3.6 or later
- An [Azure subscription][azure_subscription]
- To query Logs, you need an [Azure Log Analytics workspace][azure_monitor_create_using_portal].
- To query Metrics, you need an Azure resource of any kind (Storage Account, Key Vault, Cosmos DB, etc.).

### Install the package

Install the Azure Monitor Query client library for Python with [pip][pip]:

```bash
pip install azure-monitor-query --pre
pip install azure-monitor-query
```

### Create the client

To interact with the Azure Monitor service, create an instance of a token credential. Use that instance when creating a `LogsQueryClient` or `MetricsQueryClient`.
An authenticated client is required to query Logs or Metrics. The library includes both synchronous and asynchronous forms of the clients. To authenticate, create an instance of a token credential. Use that instance when creating a `LogsQueryClient` or `MetricsQueryClient`. The following examples use `DefaultAzureCredential` from the [azure-identity](https://pypi.org/project/azure-identity/) package.

#### Synchronous clients

Consider the following example, which creates synchronous clients for both logs and metrics querying:
Consider the following example, which creates synchronous clients for both Logs and Metrics querying:

```python
from azure.identity import DefaultAzureCredential
Expand All @@ -56,37 +61,23 @@ async_logs_client = LogsQueryClient(credential)
async_metrics_client = MetricsQueryClient(credential)
```

## Key concepts

### Logs

Azure Monitor Logs collects and organizes log and performance data from monitored resources. Data from different sources can be consolidated into a single workspace. Examples of data sources include:

- Platform logs from Azure services.
- Log and performance data from virtual machine agents.
- Usage and performance data from apps.
### Execute the query

#### Azure Log Analytics workspaces
For examples of Logs and Metrics queries, see the [Examples](#examples) section.

Data collected by Azure Monitor Logs is stored in one or more [Log Analytics workspaces](https://docs.microsoft.com/azure/azure-monitor/logs/data-platform-logs#log-analytics-workspaces). The workspace defines the:

- Geographic location of the data.
- Access rights defining which users can access data.
- Configuration settings, such as the pricing tier and data retention.

#### Log queries
## Key concepts

Data from the disparate sources can be analyzed together using [Kusto Query Language (KQL)](https://docs.microsoft.com/azure/data-explorer/kusto/query/)—the same query language used by [Azure Data Explorer](https://docs.microsoft.com/azure/data-explorer/data-explorer-overview). Data is retrieved from a Log Analytics workspace using a KQL query—a read-only request to process data and return results. For more information, see [Log queries in Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/logs/log-query-overview).
### Logs query rate limits and throttling

### Metrics
Each Azure Active Directory user is able to make up to 200 requests per 30 seconds, with no cap on the total calls per day. If requests are made at a rate higher than this, these requests will receive HTTP status code 429 (Too Many Requests) along with the `Retry-After: <delta-seconds>` header. The header indicates the number of seconds until requests to this app are likely to be accepted.

Azure Monitor Metrics collects numeric data from monitored resources into a time series database. Metrics are collected at regular intervals and describe some aspect of a system at a particular time. Metrics in Azure Monitor are lightweight and can support near real-time scenarios. They're useful for alerting and fast detection of issues. Metrics can be:
In addition to call rate limits and daily quota caps, there are limits on queries themselves. Queries cannot:

- Analyzed interactively with [Metrics Explorer](https://docs.microsoft.com/azure/azure-monitor/essentials/metrics-getting-started).
- Used to receive notifications with an alert when a value crosses a threshold.
- Visualized in a workbook or dashboard.
- Return more than 500,000 rows.
- Return more than 64,000,000 bytes (~61 MiB total data).
- Run longer than 10 minutes by default. See this for details.

#### Metrics data structure
### Metrics data structure

Each set of metric values is a time series with the following characteristics:

Expand All @@ -99,19 +90,20 @@ Each set of metric values is a time series with the following characteristics:

## Examples

- [Single logs query](#single-logs-query)
- [Logs query](#logs-query)
- [Specify timespan](#specify-timespan)
- [Set logs query timeout](#set-logs-query-timeout)
- [Handle logs query response](#handle-logs-query-response)
- [Batch logs query](#batch-logs-query)
- [Query metrics](#query-metrics)
- [Handle metrics response](#handle-metrics-response)
- [Example of handling response](#example-of-handling-response)
- [Advanced scenarios](#advanced-scenarios)
- [Advanced logs query scenarios](#advanced-logs-query-scenarios)
- [Set logs query timeout](#set-logs-query-timeout)
- [Query multiple workspaces](#query-multiple-workspaces)
- [Metrics query](#metrics-query)
- [Handle metrics query response](#handle-metrics-query-response)
- [Example of handling response](#example-of-handling-response)

### Single logs query
### Logs query

This example shows getting a log query. To handle the response and view it in a tabular form, the [pandas](https://pypi.org/project/pandas/) library is used. See the [samples][python-query-samples] if you choose not to use pandas.
This example shows getting a logs query. To handle the response and view it in a tabular form, the [pandas](https://pypi.org/project/pandas/) library is used. See the [samples][samples] if you choose not to use pandas.

#### Specify timespan

Expand Down Expand Up @@ -150,29 +142,45 @@ for table in response.tables:
print(df)
```

#### Set logs query timeout
#### Handle logs query response

The following example shows setting a server timeout in seconds. A gateway timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set up to 10 minutes (600 seconds).
The `query` API returns the `LogsQueryResult` while the `batch_query` API returns list of `LogsQueryResult`. Here's a hierarchy of the response:

```
LogsQueryResult
|---statistics
|---visualization
|---error
|---tables (list of `LogsTable` objects)
|---name
|---rows
|---columns (list of `LogsTableColumn` objects)
|---name
|---type
```

For example, to handle a logs query response with tables and display it using pandas:

```python
import os
import pandas as pd
from azure.monitor.query import LogsQueryClient
from azure.identity import DefaultAzureCredential
table = response.tables[0]
df = pd.DataFrame(table.rows, columns=[col.name for col in table.columns])
```

credential = DefaultAzureCredential()
client = LogsQueryClient(credential)
A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_log_query_client.py).

response = client.query(
os.environ['LOG_WORKSPACE_ID'],
"range x from 1 to 10000000000 step 1 | count",
server_timeout=1,
)
In a similar fashion, to handle a batch logs query response:

```python
for result in response:
table = result.tables[0]
df = pd.DataFrame(table.rows, columns=[col.name for col in table.columns])
```

A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_batch_query.py).

### Batch logs query

The following example demonstrates sending multiple queries at the same time using batch query API. The queries can either be represented as a list of `LogQueryRequest` objects or a dictionary. This example uses the former approach.
The following example demonstrates sending multiple queries at the same time using batch query API. The queries can either be represented as a list of `LogsBatchQuery` objects or a dictionary. This example uses the former approach.

```python
import os
Expand All @@ -196,7 +204,7 @@ requests = [
timespan=(datetime(2021, 6, 2), timedelta(hours=1)),
workspace_id=os.environ['LOG_WORKSPACE_ID']
),
LogsBatchQueryRequest(
LogsBatchQuery(
query= "AppRequests | take 2",
workspace_id=os.environ['LOG_WORKSPACE_ID']
),
Expand All @@ -213,43 +221,51 @@ for rsp in response:
print(df)
```

#### Handling the response for Logs Query
### Advanced logs query scenarios

The `query` API returns the `LogsQueryResult` while the `batch_query` API returns list of `LogsQueryResult`.
#### Set logs query timeout

Here is a heirarchy of the response:
The following example shows setting a server timeout in seconds. A gateway timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set up to 10 minutes (600 seconds).

```
LogsQueryResult
|---statistics
|---visualization
|---error
|---tables (list of `LogsTable` objects)
|---name
|---rows
|---columns (list of `LogsTableColumn` objects)
|---name
|---type
```
```python
import os
import pandas as pd
from azure.monitor.query import LogsQueryClient
from azure.identity import DefaultAzureCredential

So, to handle a response with tables and display it using pandas,
credential = DefaultAzureCredential()
client = LogsQueryClient(credential)

```python
table = response.tables[0]
df = pd.DataFrame(table.rows, columns=[col.name for col in table.columns])
response = client.query(
os.environ['LOG_WORKSPACE_ID'],
"range x from 1 to 10000000000 step 1 | count",
server_timeout=1,
)
```
A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_log_query_client.py).

In a very similar fashion, to handle a batch response,
#### Query multiple workspaces

The same logs query can be executed across multiple Log Analytics workspaces. In addition to the Kusto query, the following parameters are required:

- `workspace_id` - The first (primary) workspace ID.
- `additional_workspaces` - A list of workspaces, excluding the workspace provided in the `workspace_id` parameter. The parameter's list items may consist of the following identifier formats:
- Qualified workspace names
- Workspace IDs
- Azure resource IDs

For example, the following query executes in three workspaces:

```python
for result in response:
table = result.tables[0]
df = pd.DataFrame(table.rows, columns=[col.name for col in table.columns])
client.query(
<workspace_id>,
query,
additional_workspaces=['<workspace 2>', '<workspace 3>']
)
```
A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_batch_query.py).

### Query metrics
A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_log_query_multiple_workspaces.py).

### Metrics query

The following example gets metrics for an Event Grid subscription. The resource URI is that of an event grid topic.

Expand Down Expand Up @@ -285,7 +301,7 @@ for metric in response.metrics:
print(metric_value.time_stamp)
```

### Handle metrics response
#### Handle metrics query response

The metrics query API returns a `MetricsResult` object. The `MetricsResult` object contains properties such as a list of `Metric`-typed objects, `granularity`, `namespace`, and `timespan`. The `Metric` objects list can be accessed using the `metrics` param. Each `Metric` object in this list contains a list of `TimeSeriesElement` objects. Each `TimeSeriesElement` contains `data` and `metadata_values` properties. In visual form, the object hierarchy of the response resembles the following structure:

Expand Down Expand Up @@ -337,30 +353,6 @@ for metric in response.metrics:
)
```

### Advanced scenarios

#### Query multiple workspaces

The same log query can be executed across multiple Log Analytics workspaces. In addition to the KQL query, the following parameters are required:

- `workspace_id` - The first (primary) workspace ID.
- `additional_workspaces` - A list of workspaces, excluding the workspace provided in the `workspace_id` parameter. The parameter's list items may consist of the following identifier formats:
- Qualified workspace names
- Workspace IDs
- Azure resource IDs

For example, the following query executes in three workspaces:

```python
client.query(
<workspace_id>,
query,
additional_workspaces=['<workspace 2>', '<workspace 3>']
)
```

A full sample can be found [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/samples/sample_log_query_multiple_workspaces.py).

## Troubleshooting

Enable the `azure.monitor.query` logger to collect traces from the library.
Expand All @@ -379,9 +371,7 @@ Optional keyword arguments can be passed in at the client and per-operation leve

## Next steps

### Additional documentation

For more extensive documentation, see the [Azure Monitor Query documentation][python-query-product-docs].
To learn more about Azure Monitor, see the [Azure Monitor service documentation][azure_monitor_overview].

## Contributing

Expand All @@ -393,19 +383,19 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con

<!-- LINKS -->

[azure_cli_link]: https://pypi.org/project/azure-cli/
[python-query-src]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/
[python-query-pypi]: https://aka.ms/azsdk-python-monitor-query-pypi
[python-query-product-docs]: https://docs.microsoft.com/azure/azure-monitor/
[python-query-ref-docs]: https://docs.microsoft.com/python/api/overview/azure/monitor-query-readme?view=azure-python-preview
[python-query-samples]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-query/samples
[python-query-changelog]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-query/CHANGELOG.md
[pip]: https://pypi.org/project/pip/

[azure_core_exceptions]: https://aka.ms/azsdk/python/core/docs#module-azure.core.exceptions
[python_logging]: https://docs.python.org/3/library/logging.html
[azure_core_ref_docs]: https://aka.ms/azsdk/python/core/docs
[azure_monitor_create_using_portal]: https://docs.microsoft.com/azure/azure-monitor/logs/quick-create-workspace
[azure_monitor_overview]: https://docs.microsoft.com/azure/azure-monitor/
[azure_subscription]: https://azure.microsoft.com/free/python/
[changelog]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-query/CHANGELOG.md
[kusto_query_language]: https://docs.microsoft.com/azure/data-explorer/kusto/query/
[package]: https://aka.ms/azsdk-python-monitor-query-pypi
[pip]: https://pypi.org/project/pip/
[python_logging]: https://docs.python.org/3/library/logging.html
[python-query-ref-docs]: https://docs.microsoft.com/python/api/overview/azure/monitor-query-readme?view=azure-python-preview
[samples]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-query/samples
[source]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-query/

[cla]: https://cla.microsoft.com
[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
Expand Down

0 comments on commit ce307fe

Please sign in to comment.