Skip to content

fix: scrape infra and argo-workflows service monitors#1878

Open
tommartensen wants to merge 1 commit into
masterfrom
tm/fix-metrics-scraping
Open

fix: scrape infra and argo-workflows service monitors#1878
tommartensen wants to merge 1 commit into
masterfrom
tm/fix-metrics-scraping

Conversation

@tommartensen

@tommartensen tommartensen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Since about April 20, we didn't get any alerts into #acs-infra-alerts.
There are many failed clusters for which we didn't get alerts.
Root cause: Prometheus not scraping the infra and argo-workflows servicemonitors.

Validated on dev.infra.rox.systems

@tommartensen tommartensen self-assigned this Jul 9, 2026
@tommartensen tommartensen requested review from a team and rhacs-bot as code owners July 9, 2026 13:23
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Two ServiceMonitor resources in a Helm chart template were updated to add a metadata.labels.release field templated from {{ .Release.Name }}, affecting both the argo-workflows and infra-server monitors.

Changes

ServiceMonitor release label

Layer / File(s) Summary
Add release label to ServiceMonitors
chart/infra-server/templates/monitoring/servicemonitors.yaml
Added metadata.labels.release templated from {{ .Release.Name }} to the argo-workflows and infra-server ServiceMonitor resources.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Description check ❓ Inconclusive No description was provided, so the check cannot assess whether it matches the changeset. Add a brief pull request description summarizing the ServiceMonitor label change and why it is needed.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly matches the change by describing the ServiceMonitor scraping fix for infra and argo-workflows.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch tm/fix-metrics-scraping

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@rhacs-bot

Copy link
Copy Markdown
Contributor

A single node development cluster (infra-pr-1878) was allocated in production infra for this PR.

CI will attempt to deploy quay.io/rhacs-eng/infra-server: to it.

🔌 You can connect to this cluster with:

gcloud container clusters get-credentials infra-pr-1878 --zone us-central1-a --project acs-team-temp-dev

🛠️ And pull infractl from the deployed dev infra-server with:

nohup kubectl -n infra port-forward svc/infra-server-service 8443:8443 &
make pull-infractl-from-dev-server

🔓 You must go to to export an . Your token from the prod infra instance will not work with dev environments.

🚲 You can then use the dev infra instance e.g.:

bin/infractl -k -e localhost:8443 whoami

⚠️ Any clusters that you start using your dev infra instance should have a lifespan shorter then the development cluster instance. Otherwise they will not be destroyed when the dev infra instance ceases to exist when the development cluster is deleted. ⚠️

Further Development

☕ If you make changes, you can commit and push and CI will take care of updating the development cluster.

🚀 If you only modify configuration (chart/infra-server/configuration) or templates (chart/infra-server/{static,templates}), you can get a faster update with:

make helm-deploy

Logs

Logs for the development infra depending on your @redhat.com authuser:

Or:

kubectl -n infra logs -l app=infra-server --tail=1 -f

@tommartensen tommartensen requested a review from davdhacs July 10, 2026 08:36

@davdhacs davdhacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants