= Linux Metrics Exporter for OpenShift Nodes = Grega Bremec :revnumber: 1.0 :revdate: 6th November 2022 ifdef::env-github[] :status: :toc: macro :toclevels: 3 :tip-caption: :bulb: :note-caption: :information_source: :important-caption: :heavy_exclamation_mark: :caution-caption: :fire: :warning-caption: :warning: endif::[] A composite pod that can be run in a `DaemonSet` and exposes standard Linux system activity data collector (`sadc`) and process accounting (`psacct`) data as scrapable Prometheus metrics. toc::[] == Components == . Container Image for SAR . Container Image for PSACCT . Container Image for Exporter == How It All Works == Very simple: two sidecar containers, `collector-sysstat` and `collector-psacct`, produce data on a shared ephemeral volume, and the third container, `metrics-exporter`, consumes the data and exposes it on the `/q/metrics` endpoint where Prometheus can pick them up. The specific thing about how the entire composition works is that care has been taken, especially with `psacct` (which can grow excessively during periods of high activity), that accounting files are regularly truncated or moved out of the way in order to keep the disk space utilisation as low as possible. == OpenShift Deployment == === Defaults === The easiest? Just use Kustomize to deploy existing resource definitions from the `exporter` manifest in `deployment/`: [subs=+quotes] ------ $ *oc apply -k ./deployment/exporter/base/* namespace/exporter created serviceaccount/exporter created clusterrolebinding.rbac.authorization.k8s.io/metrics-exporter-serviceaccount created daemonset.apps/metrics created ------ The above will create everything in the `exporter` project. See below for kustomizations. To integrate the application with Prometheus in the `prometheus` project, just use the `integrate` manifest as it is: [subs=+quotes] ------ $ *oc apply -k ./deployment/integrate/base/* ------ === Kustomizations === If you need to change the names of namespaces or some other settings, feel free to have a look at the `custom` kustomization next to `base`, then apply it instead of the base set of resources. [subs=+quotes] ------ $ *cat deployment/exporter/custom/use-custom-namespace.yml* apiVersion: builtin kind: NamespaceTransformer metadata: namespace: *my-very-own-namespace* setRoleBindingSubjects: allServiceAccounts fieldSpecs: - path: metadata/name kind: Namespace $ *oc apply -k ./deployment/exporter/custom/* ------ You will probably already have Prometheus deployed somewhere prior to deploying the exporter, so you might also want to have a look at the kustomizations for the `integrate` manifest in order to target the right places. That one is a bit more complicated as a single manifest targets multiple namespaces. So a `NamespaceTransformer` is used for the `Role` and `RoleBinding` in the exporter project, and then two patches are used for the the `ServiceAccount` namespace in the `RoleBinding` (giving the right `prometheus-k8s` service account access to your project), and the `PodMonitor` namespace (which will be affected by the initial namespace transformation). [subs=+quotes] ------ $ ls -1 deployment/integrate/custom/ fix-podmonitor-namespace.json <1> fix-podmonitor-namespace.yml fix-prometheus-namespace.json <2> fix-prometheus-namespace.yml kustomization.yml use-custom-namespace.yml <3> ------ <1> This is the namespace the `PodMonitor` will be created in. <2> This is where the `RoleBinding` target `ServiceAccount` is fixed. <3> This is where every other resource will be created. === No Prometheus? No Problem. === If you still need to deploy Prometheus, there is a sample manifest in there as well. Two, actually. One to deploy the Prometheus and Grafana operators (you won't believe it, it's called `operators`), and once those are running, you can use the other one (called very innovatively `prometheus`) to deploy their actual instances. That will also target the `prometheus` OpenShift project, so kustomize away if that's not what you want. == Standalone Containers == Start the composition. // TODO: podman pod [subs=+quotes] ------ $ *podman volume create metrics* metrics $ *podman run -d --rm -v metrics:/var/account --cap-add SYS_PACCT --pid=host collector-psacct:latest* dd9f4825d23614df2acefdcd70ec1e6c3ea18a58b86c9d17ddc4f91038487919 $ *podman run -d --rm -v metrics:/var/log/sa collector-sysstat* ec3d0957525cc907023956a185b15123c20947460a48d37196d511ae42de2e27 $ *podman run --name exporter -d --rm -v metrics:/metrics -p 8080:8080 metrics-exporter* d4840ad57bfffd4b069e7c2357721ff7aaa6b6ee77f90ad4866a76a1ceb6adb7 ------ Configure Prometheus with a data source from the `exporter` container. [subs=+quotes] ------ $ *podman inspect -f '{{.NetworkSettings.IPAddress}}' exporter* 10.88.0.8 $ *tail -n15 tmp-test/prometheus.yml* scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] **- job_name: "exporter" metrics_path: "/q/metrics" scheme: "http" static_configs: - targets: ["10.88.0.8:8080"] scrape_interval: 10s scrape_timeout: 5s** ------ Add Prometheus and Grafana. [subs=+quotes] ------ $ *podman run --name prometheus \* *-d --rm \* *-v ./test/prometheus.yml:/etc/prometheus/prometheus.yml* *-v prometheus:/prometheus \* *-p 9090:9090 \* *registry.redhat.io/openshift4/ose-prometheus:v4.11* 6eae04677fcded65bbe1cb7f66aa887d94587977a0616f7ec838f9453702474c $ *podman run --name grafana -d --rm -p 3000:3000 \* *-v ./test/grafana.ini:/etc/grafana/grafana.ini \* *registry.redhat.io/openshift4/ose-grafana:v4.11* 78d5bfa7977923b828c1818bb877fa87bdd96086cc8c875fbc46073489f6760e ------ Configure Grafana with Prometheus as the datasource and dashboard away! .Process Accounting Graphs from a Single Host image::pics/psacct-sample.png[scaledwidth="95%" width="95%"] .Sysstat Scheduler Information, Single Host image::pics/sysstat-sample-sched.png[scaledwidth="95%" width="95%"] .Sysstat I/O Information, Single Host image::pics/sysstat-sample-io.png[scaledwidth="95%" width="95%"] == Container Images == This set of images requires a valid entitlement for RHEL (and consequently either a RHEL system to build on or a RHEL system to create an entitlement secret from). IMPORTANT: You do not have to build the images, I have already built them (for `x86_64` architecture only) and made them available on `quay.io/benko/`. === SAR === The _system activity reporting_ image is based on `ubi-minimal` and includes just the `sysstat` package. It expects a volume to be attached at `/var/log/sa`. Entrypoint takes care of initialising the `saXX` files. // TODO: and rotating any old files out of the way. It *requires* to be executed under `root` UID (can be rootless, but that may affect your data depending on host and container configuration). It also *requires* access to host's network namespace if you want to measure global network statistics. ==== Parameters ==== `PERIOD`:: Sampling period in seconds. Defaults to `10`. Increase this to something like `30` (or more) for hosts with many network interfaces, block devices, and/or CPUs. `STARTUP_SCRATCH`:: Whether to scratch existing `sa1` data at startup. Defaults to `0`, but could be anything except `1`, `yes`, or `true`, which activates it. `STARTUP_ROTATE`:: Whether to mark data as rotated at startup. Basically just writes a marker in the previous `sadc` data file. Defaults to `0`, but could be anything except `1`, `yes`, or `true`, which activates it. === PSACCT === The _process accounting_ image is based on `ubi-minimal` and includes just the `psacct` package. It expects a volume to be attached at `/var/account`. Entrypoint takes care of rotating any old `pacct` files out of the way. In addition to *requiring* execution under a *real* `root` UID (i.e. *NOT* a rootless container), it also *requires* the `CAP_SYS_PACCT` capability (`--cap-add=SYS_PACCT`) and access to host's PID namespace (`--pid=host`). ==== Parameters ==== `PERIOD`:: Sampling period in seconds. Defaults to `10`. Increase this to something like `30` (or more) for hosts with many thousands of processes. `CUMULATIVE`:: Tells the collection process to never reset the `pacct` file and just keep it growing, thus reporting cumulative stats since container start. Beware that the `pacct` file will grow correspondinly large as time goes by. `STARTUP_SCRATCH`:: Whether to scratch existing `pacct` data at startup. Defaults to `0`, but could be anything except `1`, `yes`, or `true`, which activates it. === Exporter === The brain of the group. // TODO: Add support for hostname overrides in app. // run a maven registry.access.redhat.com/ubi9/openjdk-17 container: // // podman volume create maven // // podman run -it \ // --name exporter \ // -v maven:/home/default/.m2/repository \ // -v metrics:/metrics \ // -v /Users/johndoe/Documents/workspaces/projects/p0f/linux-metrics-exporter/exporter:/exporter \ // -p 8080:8080 \ // registry.access.redhat.com/ubi9/openjdk-17 bash // // $ cd /exporter // $ mvn quarkus:dev ==== Parameters ==== In `application.properties` or as Java system properties: `exporter.data.path`:: Override the location where the metrics files are expected to show up. Defaults to `/metrics` but obviously can't be that for testing outside of a container. You can set the same settings https://quarkus.io/guides/config-reference[from environment variables]. ==== Debugging ==== There are a couple of logger categories that might help you see what's going on. By default, the routes are fairly noisy, as apparently `TRACE` level logging doesn't work for some reason, so I had to bump everything up a level, so at `INFO` you already see a note about every record that's been processed - you will see their unmarshaled bodies (completely shameless, I know). These can be bumped up to `DEBUG` if you need more info: `psacct-reader`:: The route reading process accounting files from `psacct-dump-all` file. Pretty much all the logic is here, but since there can be a large number of process records in the file it is split and each record is processed asynchronously by the dispatch route. `psacct-dispatch`:: The route dispatching the records to the registration service. `psacct-reset`:: To be able to work with instantaneous data, rather than cumulative, all previously registered records are synchronously reset to zero upon the arrival of a new snapshot. This prevents metrics for previously registered processes from disappearing. `sysstat-reader`:: The route that reads `sysstat-dump.json` file. All the logic is here. `net.p0f.openshift.metrics`:: Non-camel stuff is all logged in this category. `net.p0f.openshift.metrics.exporter`:: Metric registration and a silly REST endpoint that reports the version. `net.p0f.openshift.metrics.model`:: `ProcessAccountingRecord` and `SysstatMeasurement` live here. `net.p0f.openshift.metrics.processor`:: Just a simple processor that transforms a `psacct` record into CSV. `net.p0f.openshift.metrics.routes`:: Camel routes. See the first four categories for this. === Building with Podman === If building the images using `podman` on an entitled host, no extra steps need to be performed as host entitlements will automatically be imported into the build container. [NOTE] ======== When building for an architecture without the `ubi-minimal` image or on a host that can not be entitled (f.e. Fedora CoreOS), you can choose a different base image by using the `--from` option in `podman build`. [subs=+quotes] ------------------------------- $ *podman build --from=registry.fedoraproject.org/fedora-minimal:36 -f ./images/Containerfile-sysstat -t collector-sysstat:latest* ------------------------------- ======== You will have noticed there is no `Containerfile` for exporter. That is because `quarkus-maven-plugin` can do just fine https://quarkus.io/guides/container-image[building an image on its own]. Just add the `jib` extension and tell it to push the image somewhere. [subs=+quotes] ------------------------------- $ *mvn package -Dquarkus.container-image.build=true -Dquarkus.container-image.push=true -Dquarkus.container-image.registry=foo* ------------------------------- === Building in OpenShift === ==== Collector Images ==== If building the images in OpenShift Container Platform, you must make sure an entitlement secret and corresponding RHSM certificate secret are mounted inside the build pod in order for packages to be found and installed. NOTE: The entitled system architecture needs to match the container host! The process is as follows. .Verify access to host entitlement data. [subs=+quotes] ------------------------------- $ **ls -l /etc/pki/entitlement/\*.pem /etc/rhsm/ca/*.pem** -rw-r--r--. 1 root root 3272 Oct 31 06:09 /etc/pki/entitlement/_6028779042203586857_-key.pem -rw-r--r--. 1 root root 149007 Oct 31 06:09 /etc/pki/entitlement/_6028779042203586857_.pem -rw-r--r--. 1 root root 2305 Sep 2 2021 /etc/rhsm/ca/redhat-entitlement-authority.pem -rw-r--r--. 1 root root 7411 Sep 2 2021 /etc/rhsm/ca/redhat-uep.pem ------------------------------- .Create corresponding secrets. [subs=+quotes] ------------------------------- $ *oc create secret generic etc-pki-entitlement \* *--from-file=/etc/pki/entitlement/_6028779042203586857_-key.pem \* *--from-file=/etc/pki/entitlement/_6028779042203586857_.pem* secret/etc-pki-entitlement created $ *oc create secret generic rhsm-ca \* *--from-file=/etc/rhsm/ca/redhat-entitlement-authority.pem \* *--from-file=/etc/rhsm/ca/redhat-uep.pem* secret/rhsm-ca created ------------------------------- .Make sure the BuildConfig mounts those secrets. [subs=+quotes] ------------------------------- apiVersion: build.openshift.io/v1 kind: BuildConfig ... strategy: type: Docker dockerStrategy: dockerfilePath: Containerfile-psacct from: kind: ImageStreamTag name: ubi-minimal:latest **volumes: - source: type: Secret secret: secretName: etc-pki-entitlement name: etc-pki-entitlement mounts: - destinationPath: /etc/pki/entitlement - source: type: Secret secret: secretName: rhsm-ca name: rhsm-ca mounts: - destinationPath: /etc/rhsm/ca** ------------------------------- `Containerfile` instructions are written such that they should work without modification regardless of whether the build is running in `podman` on an entitled host or inside a correctly configured OpenShift builder pod. NOTE: Key thing in `Containerfile` steps is to remove `/etc/rhsm-host` at some point unless `/etc/pki/entitlement-host` contains something (such as for example, valid entitlemets). Both are symlinks to `/run/secrets`. ==== Exporter Image ==== ===== Java Build ===== Java build is relatively simple. Figure out what OpenJDK image is available in the cluster and create a new build. [subs=+quotes] ------------------------------- $ *oc new-build openjdk-11-rhel8:1.0~https://github.com/benko/linux-metrics-exporter.git --context-dir=exporter* ------------------------------- Wait for the build to complete (it's going to take quite some time to download all deps) and that's it! If you're experimenting with the code, don't forget to mark the build as incremental. [subs=+quotes] ------------------------------- $ *oc patch bc/linux-metrics-exporter -p '{"spec": {"strategy": {"sourceStrategy": {"incremental": true}}}}'* ------------------------------- ===== Native Build ===== TBD // For the native build, you need a specific Mandrel image. Import it first. // // $ oc import-image mandrel --from=registry.redhat.io/quarkus/mandrel-21-rhel8:latest --confirm // imagestream.image.openshift.io/mandrel imported // ... ===== Publishing the Image ===== Make sure the internal OpenShift image registry is exposed if you want to copy the image somewhere else. [subs=+quotes] ------------------------------- $ *oc patch config.imageregistry/cluster --type=merge -p '{"spec": {"defaultRoute": true}}'* ------------------------------- Login to both source and target registries. [subs=+quotes] ------------------------------- $ *podman login quay.io* Username: *youruser* Password: *yourpassword* Login Succeeded! $ *oc whoami -t* sha256~8tIizkcLNroDEcWXJgoPMsVYUriK1sGnJ6N94WSveEU $ podman login default-route-openshift-image-registry.apps.your.openshift.cluster Username: _this-is-irrelevant_ Password: *token-pasted-here* Login Succeeded! ------------------------------- Then simply copy the image using `skopeo`. [subs=+quotes] ------------------------------- $ *skopeo copy \* *docker://default-route-openshift-image-registry.apps.your.openshift.cluster/project/linux-metrics-exporter:latest \* *docker://quay.io/youruser/yourimage:latest* ------------------------------- == Acknowledgements == Many thanks to Piotr Baranowski for the idea about running `sa1` in a DaemonSet.