Kyle Brennan
696ec03449
[obs] fix alerts for workspace deployments ( #19502 )
...
The proper label to use with `kube_deployment_spec_replicas` is deployment.
`container` is okay with `kube_pod_container_status_restarts_total`
2024-03-05 01:49:14 +02:00
Gero Posmyk-Leinemann
bdad590b09
[server] Missing SCM access: Filter out user error on workspace start ( #19469 )
...
* [server] Missing SCM access: Filter out user error on workspace start
to prevent false alerts (EXP-1434)
* [proxy] api.: Handle /auth/*/callback
2024-02-27 09:13:08 +02:00
Gero Posmyk-Leinemann
d40119977c
GitpodWorkspaceTooManyRegularNotActiveMk2: add lower bound of 20 ( #19439 )
2024-02-19 11:53:00 +02:00
Huiwen
d2c0ed2882
[observability] update dashboard not all ready alert ( #19317 )
2024-01-15 12:01:52 +02:00
Kyle Brennan
31db761ae7
[obs] tune GitpodWorkspaceStuckOnStoppingMk2 alert ( #19299 )
...
This alert fired 8 times for us107 since Dec 23, none of the pages required action from the on-call operator.
Let's make it more difficult for the alert to fire, to avoid unnecessary escalations to on-call.
2023-12-31 22:30:38 +02:00
Kyle Brennan
47eef601a1
[ws-proxy] add SSH Summary to overview dashboard ( #19281 )
...
Related to ENG-1340 and ENG-1350
2023-12-18 21:08:25 +02:00
Huiwen
c18e049eb3
[observability] add DashboardStuckInPodInitState alert ( #19258 )
...
* [observability] add `DashboardStuckInPodInitState` alert
* Update expr
2023-12-15 10:19:22 +02:00
Anton Kosyakov
9bd532b474
add GitpodV1APIServerErrors alert ( #19212 )
2023-12-07 16:15:14 +02:00
Mads Hartmann
3ff9173705
This adds an alert for GitpodWorkspaceHighStartFailureRate ( #19099 )
2023-11-21 14:07:58 +02:00
Wouter Verlaek
110defe741
Use humanize1024 instead ( #19036 )
2023-11-08 11:19:45 +02:00
Wouter Verlaek
7af72bb539
Add GiB left in storage warning ( #19035 )
2023-11-08 10:45:45 +02:00
Kyle Brennan
5b13b510ec
[obs] remove GitpodImagebuildStartSuccess warning ( #19002 )
...
This expression has dips regularly, and does not provide value as a notification in its current form.
2023-11-02 22:09:40 +02:00
Kyle Brennan
f3cd71cc2d
[obs] remove noisy GitpodWorkspaceStuckOnStoppedMk2 ( #18998 )
...
Pods stuck in Stopped are removed after the 30m grace period: c346773e50/components/ws-manager-mk2/controllers/create.go (L54)
2023-11-02 00:05:39 +02:00
Kyle Brennan
c346773e50
[ops] change team from Workspace to Engine ( #18997 )
2023-11-01 22:46:39 +02:00
Wouter Verlaek
30a4280ab8
Add IPFS storage to overview dashboard ( #18949 )
2023-10-18 13:37:25 +03:00
Gero Posmyk-Leinemann
83d20c1415
[grafana] SpiceDB: add graph for request consistency ( #18904 )
2023-10-11 10:50:19 +03:00
Gero Posmyk-Leinemann
a84cca8e4c
[grafana] SpiceDB dashboard: add graphs for cache hit/miss rate and ratio ( #18897 )
2023-10-09 15:19:17 +03:00
Gero Posmyk-Leinemann
6c7f47dd7b
[alerts] Exclude reason "imageBuildFailedUser" from InstanceStartFailures ( #18768 )
2023-09-22 08:37:01 +02:00
Thomas Schubart
b57d8707e4
Include regular not active alerts in Dedicated ( #18702 )
2023-09-13 13:33:53 +02:00
Aleksandar
bcfa933865
[alerts] group by cluster for the NodePoolLoad alert ( #18663 )
2023-09-06 08:44:03 +02:00
Kyle Brennan
9cc716412d
[obs] demote GitpodImageBuildDurationAnomaly to a warning ( #18507 )
2023-08-14 14:17:40 +02:00
Kyle Brennan
285f11d234
[obs] reduce false positives for GitpodImageBuildDurationAnomaly ( #18496 )
2023-08-11 16:46:37 +02:00
Kyle Brennan
e12de07cf2
[obs] GitpodImageBuildDurationAnomaly appears to fire too often for Dedicated ( #18472 )
...
Remove to avoid alerting on-callers. Circle back after we have a better expression, or means to define criteria that is exclusive to Dedicated.
2023-08-09 20:57:35 +02:00
Kyle Brennan
d8b68fd515
[obs] delay triggering GitpodWorkspaceTooManyRegularNotActiveMk2 ( #18409 )
...
Related false alarm: https://gitpod.slack.com/archives/C01TNS8EVQT/p1690945869670219
2023-08-02 20:49:27 +08:00
Kyle Brennan
b90e12b7cb
[obs] re-enable regular not active alerts ( #18341 )
...
* [obs] Add back critical regular not active alerts
Related to ENG-15
Now that we have related data, we should resume triggering alerts if the data condition occurs.
* [obs] Fix runbook_url for GitpodImageBuildDurationAnomaly
Was getting 404
* [obs] Fix GitpodWorkspaceTooManyRegularNotActiveMk2 given https://www.gitpodstatus.com/incidents/bsrqgmsxw1gr
* [obs] share why regular not active is excluded from Dedicated
* [obs] consolidate runbook for regular not active alerts
2023-07-31 21:31:26 +08:00
Wouter Verlaek
5e7eff45d8
Warn when IPFS is running out of storage ( #18386 )
...
* Warn when IPFS is running out of storage
* Add critical alert
2023-07-31 19:54:26 +08:00
Milan Pavlik
3422cc7085
[messagebus] Remove alerts + dashboards ( #18337 )
2023-07-24 22:19:40 +08:00
Kyle Brennan
8266f6175c
Fix gitpod_workspace_regular_not_active_percentage_mk2, and temporarily disable related alerts ( #18332 )
...
We'll likely replace the alerts too, using one that detects anomalies given zscore.
Related to ENG-20
2023-07-24 19:40:40 +08:00
Kyle Brennan
2d3c03ee43
[obs] introduce workspace alerts for Dedicated ( #18331 )
...
* [obs] introduce GitpodImageBuildDurationAnomaly
Depends on https://github.com/gitpod-io/runbooks/pull/417
* [obs] Introduce GitpodImageBuilderCrashlooping
As per https://samber.github.io/awesome-prometheus-alerts/rules#rule-kubernetes-1-19
* [obs] Introduce GitpodImageBuilderReplicasMismatch
* [obs] use generic GitpodWorkspaceDeploymentCrashlooping for GitpodWsManagerCrashLoopingMk2
* Fix GitpodWsManagerCrashLoopingMk2
To avoid false positives
* Introduce GitpodWsManagerMk2ReplicasMismatch
* Fix syntax
* Fix GitpodWorkspaceDeploymentReplicaMismatch URL
* Introduce alerts for node-labeler and ws-proxy
* Fix severity and dedicated labels
* Fix proxy references
* Exclude ephemeral clusters
* Clean-up
2023-07-24 19:39:40 +08:00
Brad Harris
9d88f8d5e5
ignore 640 error codes ( #18249 )
2023-07-13 02:33:29 +08:00
Kyle Brennan
89797f48a4
[obs] add node variable and update panels in Node PSI dashboard to use it ( #18250 )
...
Related to WKS-303
2023-07-12 05:42:29 +08:00
Kyle Brennan
f3eb34242b
[obs] further restrict NodePoolLoad to avoid false positives ( #18222 )
...
Only trigger alerts when 4 or more nodes have high load average that is sustained over 1 for 60m.
2023-07-12 02:00:28 +08:00
Milan Pavlik
45a8c259a7
[server] Add Redis subscription/broadcasting to dashboard ( #18231 )
2023-07-11 19:18:28 +08:00
Wouter Verlaek
ad21ecb48e
Fix: remove ws-manager.json dashboard ( #18230 )
2023-07-11 18:02:27 +08:00
Milan Pavlik
9c26cb371b
[ws-man-bridge] Add events published to Redis to dashboard ( #18229 )
2023-07-11 17:27:27 +08:00
Wouter Verlaek
85a0e9a67c
[ws-manager-mk2] Fix metric labels ( #18220 )
2023-07-11 16:55:28 +08:00
Gero Posmyk-Leinemann
08f4a2737b
[alerts] First alerts for Dedicated ( #18207 )
2023-07-11 15:51:27 +08:00
Gero Posmyk-Leinemann
e7ec0eb90b
[alerts] Reduce the threshold for WebAppServicesCrashlooping ( #18201 )
...
* [alerts] Reduce the threshold for WebAppServicesCrashlooping
and add an internal version with even lower threshold for further improvements
* Review fix
Co-authored-by: Milan Pavlik <milan@gitpod.io>
* Review fix
Co-authored-by: Milan Pavlik <milan@gitpod.io>
* Review fix
Co-authored-by: Milan Pavlik <milan@gitpod.io>
---------
Co-authored-by: Milan Pavlik <milan@gitpod.io>
2023-07-10 15:22:26 +08:00
Kyle Brennan
13aa60d211
[obs] reduce noice related to GitpodWsManagerCrashLoopingMk2 ( #18181 )
...
When it triggers now, it's generally due to WKS-210, and is not valuable to gitpod.io or Dedicated in it's current form.
In other words, if ws-manager-mk2 restarts, it recovers and no action is needed. If it's unable to start, we won't be able to createWorkspace (and server should emit a signal).
Fixes WKS-288
2023-07-07 10:48:24 +08:00
Pudong
1f3f6d28e1
[obs] add ide alert for dedicated ( #18173 )
2023-07-05 15:38:23 +08:00
Mads Hartmann
2791ff7ed9
Add cell variable and panels which show the Gitpod version ( #18092 )
2023-06-29 16:53:47 +08:00
Wouter Verlaek
76b585e6e8
Add longest running reconcile to dashboards ( #18022 )
2023-06-22 20:38:12 +08:00
Thomas Schubart
ab8244040d
[workspace] UPdate regular not active alert ( #17997 )
2023-06-21 16:09:12 +08:00
Huiwen
a650082e40
[observability] add dashboard for local ssh IDE-154 ( #17963 )
...
* [observability] add dashboard for local ssh
* remove rate
2023-06-21 00:50:10 +08:00
Kyle Brennan
6755b23081
[obs] Fix workspace alerts to work in Grafana cloud ( #17944 )
...
* [obs] tag workspace alerts for Dedicated
* [obs] rename duplicate workspace-monitoring-rules
* [obs] fix duplicate ws-daemon-monitoring-rules
2023-06-15 22:01:06 +08:00
Kyle Brennan
aba2cfdfe4
[obs] remove ws-manager (mk1) alert references ( #17941 )
...
* [obs] remove ws-manager (mk1) alert references
This includes old alerts, old recording rules, what appear to be SLO alert rules that appear incomplete, and fixing a bug with BackupFailureBecauseOfGitpodWsDaemonCrash
* [obs] flip GitpodImagebuildDoneSuccess and GitpodImagebuildStartSuccess back to warning due to prior false positives
2023-06-15 22:00:06 +08:00
Kyle Brennan
4d5ea429ec
[obs] setup better names for alert rules ( #17930 )
...
* [obs] setup unique alert names
* [obs] Set unique name for vscode.yaml alert rules
2023-06-15 02:34:06 +08:00
Huiwen
aa36729030
[observability] add WorkspaceUpdates filter to supervisor ( #17804 )
2023-06-01 14:42:06 +08:00
Kyle Brennan
1422b8ea5a
[obs] fix and tune NodePoolLoad alert ( #17757 )
2023-05-26 19:12:00 +08:00
Kyle Brennan
2208a8792e
[obs] alert when cluster is under high load ( #17755 )
...
* [obs] formatting
* [obs] alert and inspect cluster due to high sustained load
2023-05-26 09:48:59 +08:00
