linux-metrics-exporter

= Linux Metrics Exporter for OpenShift Nodes =

== Components ==

. Container Image for SAR
. Container Image for PSACCT
. Container Image for Exporter

== How It All Works ==

Very simple: two sidecar containers, `collector-sysstat` and
`collector-psacct`, produce data on a shared ephemeral volume, and the third
container, `metrics-exporter`, consumes the data and exposes it on the
`/q/metrics` endpoint where Prometheus can pick them up.

The specific thing about how the entire composition works is that care has been
taken, especially with `psacct` (which can grow excessively during periods of
high activity), that accounting files are regularly truncated or moved out of
the way in order to keep the disk space utilisation as low as possible.

== OpenShift Deployment ==

TBD

== Standalone ==

Start the composition.

// TODO: podman pod

[subs=+quotes]
------
$ *podman volume create metrics*
metrics

$ *podman run -d --rm -v metrics:/var/account --cap-add SYS_PACCT --pid=host collector-psacct:latest*
dd9f4825d23614df2acefdcd70ec1e6c3ea18a58b86c9d17ddc4f91038487919

$ *podman run -d --rm -v metrics:/var/log/sa collector-sysstat*
ec3d0957525cc907023956a185b15123c20947460a48d37196d511ae42de2e27

$ *podman run --name exporter -d --rm -v metrics:/metrics -p 8080:8080 metrics-exporter*
d4840ad57bfffd4b069e7c2357721ff7aaa6b6ee77f90ad4866a76a1ceb6adb7

------

Add prometheus and grafana.

[subs=+quotes]
------
$ *podman run --name prometheus \*
*-d --rm \*
*-v ./test/prometheus.yml:/etc/prometheus/prometheus.yml*
*-v prometheus:/prometheus \*
*-p 9090:9090 \*
*registry.redhat.io/openshift4/ose-prometheus:v4.11*
6eae04677fcded65bbe1cb7f66aa887d94587977a0616f7ec838f9453702474c

$ *podman run --name grafana -d --rm -p 3000:3000 \*
*-v ./test/grafana.ini:/etc/grafana/grafana.ini \*
*registry.redhat.io/openshift4/ose-grafana:v4.11*
78d5bfa7977923b828c1818bb877fa87bdd96086cc8c875fbc46073489f6760e
------

== Images ==

This set of images requires a valid entitlement for RHEL (and consequently
either a RHEL system to build on or a RHEL system to create an entitlement
secret from).

IMPORTANT: You do not have to build the images, they are already provided by the `is-readymade.yml` resource.

=== SAR ===

The _system activity reporting_ image is based on `ubi-minimal` and includes just the `sysstat` package.

It expects a volume to be attached at `/var/log/sa`.

Entrypoint takes care of initialising the `saXX` files.

// TODO: and rotating any old files out of the way.

It *requires* to be executed under `root` UID (can be rootless, but that may affect your data depending on host and container configuration).

It also *requires* access to host's network namespace if you want to measure global network statistics.

// NOTE: When running in a pod, the below is irrelevant as the exporter sets
// the hostname, and you can override it there. It does however obtain
// the default hostname from data sources.
//
// If aggregating from several hosts, you also need to use the hosts' UTS
// namespace to report correct hostnames, or set the corresponding environment
// variables to override detection.

==== Parameters ====

TBD

=== PSACCT ===

The _process accounting_ image is based on `ubi-minimal` and includes just the `psacct` package.

It expects a volume to be attached at `/var/account`.

Entrypoint takes care of rotating any old `pacct` files out of the way.

In addition to *requiring* execution under a *real* `root` UID (i.e. *NOT* a rootless container), it also *requires* the `CAP_SYS_PACCT` capability (`--cap-add=SYS_PACCT`) and access to host's PID namespace (`--pid=host`).

// NOTE: When running in a pod, the below is irrelevant as the exporter sets
// the hostname, and you can override it there. It does however obtain
// the default hostname from data sources.
//
// If aggregating from several hosts, you also need to use the hosts' UTS
// namespace to report correct hostnames, or set the corresponding environment
// variables to override detection.

==== Parameters ====

TBD

=== Exporter ===

TBD

// TODO: Add support for hostname overrides in app.

// run a maven registry.access.redhat.com/ubi9/openjdk-17 container:
//
// podman volume create maven
//
// podman run -it \
// --name exporter \
// -v maven:/home/default/.m2/repository \
// -v metrics:/metrics \
// -v /Users/johndoe/Documents/workspaces/projects/p0f/linux-metrics-exporter/exporter:/exporter \
// -p 8080:8080 \
// registry.access.redhat.com/ubi9/openjdk-17 bash
//
// $ cd /exporter
// $ mvn quarkus:dev

==== Parameters ====

TBD

=== Building with Podman ===

If building the images using `podman` on an entitled host, no extra steps need
to be performed as host entitlements will automatically be imported into the
build container.

NOTE: When building for an architecture without the `ubi-minimal` image or on a
host that can not be entitled (f.e. Fedora CoreOS), you can choose a different
base image by using the `--from` option in `podman build`.
[subs=+quotes]
-------------------------------
$ *podman build --from=registry.fedoraproject.org/fedora-minimal:36 -f ./images/Containerfile-sysstat -t collector-sysstat:latest*
-------------------------------

=== Building in OpenShift ===

If building the images in OpenShift Container Platform, you must make sure an
entitlement secret and corresponding RHSM certificate secret are mounted inside
the build pod in order for packages to be found and installed.

NOTE: The entitled system architecture needs to match the container host!

The process is as follows.

.Verify access to host entitlement data.
[subs=+quotes]
-------------------------------
$ **ls -l /etc/pki/entitlement/*.pem /etc/rhsm/ca/*.pem**
-rw-r--r--. 1 root root 3272 Oct 31 06:09 /etc/pki/entitlement/_6028779042203586857_-key.pem
-rw-r--r--. 1 root root 149007 Oct 31 06:09 /etc/pki/entitlement/_6028779042203586857_.pem
-rw-r--r--. 1 root root 2305 Sep 2 2021 /etc/rhsm/ca/redhat-entitlement-authority.pem
-rw-r--r--. 1 root root 7411 Sep 2 2021 /etc/rhsm/ca/redhat-uep.pem
-------------------------------

.Create corresponding secrets.
[subs=+quotes]
-------------------------------
$ *oc create secret generic etc-pki-entitlement \*
*--from-file=/etc/pki/entitlement/_6028779042203586857_-key.pem \*
*--from-file=/etc/pki/entitlement/_6028779042203586857_.pem*
secret/etc-pki-entitlement created

$ *oc create secret generic rhsm-ca \*
*--from-file=/etc/rhsm/ca/redhat-entitlement-authority.pem \*
*--from-file=/etc/rhsm/ca/redhat-uep.pem*
secret/rhsm-ca created
-------------------------------

.Make sure the BuildConfig mounts those secrets.
[subs=+quotes]
-------------------------------
apiVersion: build.openshift.io/v1
kind: BuildConfig
...
strategy:
type: Docker
dockerStrategy:
dockerfilePath: Containerfile-psacct
from:
kind: ImageStreamTag
name: ubi-minimal:latest
**volumes:
- source:
type: Secret
secret:
secretName: etc-pki-entitlement
name: etc-pki-entitlement
mounts:
- destinationPath: /etc/pki/entitlement
- source:
type: Secret
secret:
secretName: rhsm-ca
name: rhsm-ca
mounts:
- destinationPath: /etc/rhsm/ca**
-------------------------------

`Containerfile` instructions are written such that they should work without
modification regardless of whether the build is running in `podman` on an
entitled host or inside a correctly configured OpenShift builder pod.

NOTE: Key thing in `Containerfile` steps is to remove `/etc/rhsm-host` at some
point unless `/etc/pki/entitlement-host` contains something (such as for
example, valid entitlemets). Both are symlinks to `/run/secrets`.