Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Benchmark section to our documentation page #3125

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/download_svgs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Download SVGs

on:
workflow_dispatch: # You can trigger the workflow manually, or set it on a schedule.

jobs:
download-svgs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.x' # Use Python 3.x

- name: Install dependencies
run: |
pip install requests

- name: Run download_svgs.py
env:
MUNIN_USERNAME: ${{ secrets.MUNIN_USERNAME }}
MUNIN_PASSWORD: ${{ secrets.MUNIN_PASSWORD }}
run: |
python3 /scripts/download_svgs.py
101 changes: 101 additions & 0 deletions docs/src/benchmarks/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
title: Benchmarks
sidebar_position: 10
---

# Benchmarks
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key point is missing: what is the workload used for these benchmarks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree fully, I put it in the ToDo section as decision needed what we want to be the workload


This section contains performance benchmarks for ThinEdge.io, including detailed CPU and memory usage for key processes, measured on a device running in the OSADL QA Farm.

## Hardware Information

The following hardware and software specifications describe the environment in which these benchmarks are measured. The device runs ThinEdge.io and its associated plugins for data collection.

| Component | Specification |
|----------------|---------------------------------------------------------------|
| **Processor** | Quad-core ARM Cortex-A72 (ARM v8) |
| **Memory** | 1.8 GB RAM |
| **Operating System** | Debian GNU/Linux 12 (bookworm) |
| **MQTT Broker**| Mosquitto (v2.0.18) |
| **Firmware** | ThinEdge.io (v1.1.2) |
| **Plugins Used** | `tedgecpuprocent`, `memory`, `tedge-agent`, `tedge-mapper` |

> The device is a Raspberry Pi 4 Model B Rev 1.5, a member of the OSADL QA Farm, which continuously collects data on performance and resource consumption.
## CPU Run-time Consumption of Key ThinEdge.io Tasks

In this section, we monitor the CPU usage of the most critical ThinEdge.io tasks: `mosquitto`, `tedge-agent`, and `tedge-mapper`. These tasks are essential for device-to-cloud communication, and their performance is crucial for the overall efficiency of the system.

### Graph of CPU Usage

Below is a visual representation of the CPU consumption by the main ThinEdge.io processes over the past month.

![CPU Usage](./tedgecpuprocent-month.svg)

For each process, the graph shows:
- **Command Name (comm)**: The name of the task running on the device.
- **CPU Usage (cuc)**: The percentage of CPU time utilized by each process.


The graph allows you to visually track CPU performance trends and identify potential bottlenecks in the system.

### Detailed Metrics

The following table provides a detailed description of the CPU consumption of measured tasks:

| Task | Description |
|----------------------|----------------------------------------------------------------|
| `mosquitto` | Handles MQTT communications between devices and cloud |
| `tedge-agent` | Coordinates device management, software updates, and telemetry |
| `tedge-mapper_c8y` | Converts telemetry data into the format required by cloud |
| `tedge-mapper-colle` | Converts telemetry data into the format required |

> Note: The values in this table are updated dynamically once per month.

## Memory Usage

In this section, we monitor the memory consumption of key system components to ensure efficient performance. The memory plugin tracks several metrics to provide a detailed view of how the system memory is being utilized.

### Graph of Memory Usage

Below is a visual representation of the memory consumption on the device over the past month.

![Memory Usage](./memory-month.svg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The memory script should only show the thin-edge.io applications and not the whole OS memory usage etc.


### Monitored Metrics:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see these metrics as useful for giving an idea of the memory footprint of thin-edge, since there are not directly related.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there are other metrics which just look at the memory for each of the thin-edge.io related memory usage, e.g. https://munin.osadl.org/munin/osadl.org/rackfslot1.osadl.org/tedgemem.html

- **Total Memory**: The total amount of memory available on the system.
- **Free Memory**: Memory that is available for new processes.
- **Buffers**: Memory used by the system for temporary storage, primarily for block devices (like hard disks).
- **Cached Memory**: Memory that is cached for quicker access to frequently used data.
- **Swap Memory**: Disk space used as virtual memory when physical memory is full.
- **Slab Memory**: Memory used by the kernel for various internal data structures.
- **Shared Memory (shmem)**: Memory used for shared memory segments and tmpfs (temporary file system).
- **Swap Cache**: Memory that keeps track of pages that have been fetched from swap but not yet modified.
- **Page Tables**: Memory used to map virtual memory addresses to physical addresses.

For each metric, the graph shows the current and historical values, allowing you to monitor how memory is utilized by the system.

### Detailed Metrics

The following table provides a description of memory usage metrics:

| Metric | Description |
|------------------|-----------------------------------------------------------------------------|
| **Total Memory** | The total amount of memory available on the system. |
| **Free Memory** | Memory that is available for new processes. |
| **Buffers** | Memory used for temporary storage for block devices (e.g., hard disks). |
| **Cached Memory** | Memory cached for quicker access to frequently used data. |
| **Swap Memory** | Disk space used as virtual memory. |
| **Slab Memory** | Memory used by the kernel for internal data structures. |
| **Shared Memory** | Memory used for shared memory segments and tmpfs. |
| **Swap Cache** | Memory keeping track of pages fetched from swap but not yet modified. |
| **Page Tables** | Memory used to map virtual memory addresses to physical memory addresses. |

> The values in this table are updated dynamically once per month.
By monitoring these metrics, we can ensure that the system is using memory efficiently, and we can detect potential memory leaks or bottlenecks in the system's operation.

---

This section now contains all relevant CPU and memory metrics, helping you track and analyse how system resources are allocated and used over time. These benchmarks are crucial for maintaining overall system performance.
Loading