gke-ai-troubleshooting-tpu-mxla-hang
De la autor
Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).
Instalare
npx skills add google/skills --skill gke-ai-troubleshooting-tpu-mxla-hang -a claude-code
Se instalează în .claude/skills/gke-ai-troubleshooting-tpu-mxla-hang/ · Fiecare comandă descarcă skill-ul direct de pe GitHub cu utilitarul open source skills.
Identic octet cu octet cu commit-ul scanat, cu licența depozitului și un fișier SOURCE.txt.
Vrei ca agentul să-ți propună skill-uri în timp ce scrii? Vezi cum folosești skill-urile din IDE.
Verificări de siguranță pe SKILL.md
Niciun avertisment.
De ce 95
aiqbench calculează scorul de încredere din semnale publice. Nu este un audit de securitate, așa că citește un skill înainte să-l instalezi.
Skill-uri similare
Cele mai apropiate de acesta prin ce fac și când sunt folosite.
- gke-ai-troubleshooting-tpu-performance-degradationgoogle/skillsDiagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).OficialDevOps★ 21.5K
- gke-ai-troubleshooting-tpu-dynamic-slices-monitoringgoogle/skillsMonitors, troubleshoots, and manages GKE TPU Dynamic Slices custom resources. Use when checking TPU slice lifecycle states, troubleshooting slice provisioning failures, validating single-slice or multi-slice (JobSet) workload manifests, or safely patching stuck finalizers and disabling the slice controller. Don't use for generic GKE cluster node pool creation or standard non-TPU workload management (use gke-basics or gke-cluster-creation instead).OficialAltele★ 21.5K
- gke-ai-troubleshooting-handle-disruption-gpu-tpugoogle/skillsDiagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.OficialDevOps★ 21.5K
- gke-ai-troubleshooting-tpu-slice-mig-stuck-updatinggoogle/skillsDiagnose and resolve GKE TPU Dynamic Slice creation and JobSet scheduling failures blocked by underlying Compute Engine managed instance groups (MIGs) stuck in an updating or unstable state (isStable=False). Use when TPU slice creation fails with a SliceCreationFailed status reason, when JobSets remain stuck in pending, or when MIG status reflects isStable=False or versionTarget.isReached=False during TPU slice formation. Don't use for SliceCreationFailed causes unrelated to MIG stability (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring) or general pod scheduling failures unrelated to TPU slices.OficialAltele★ 21.5K
- gke-ai-troubleshooting-tpu-vm-repair-stuckgoogle/skillsDiagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).OficialAltele★ 21.5K
- gke-workload-troubleshootinggoogle/skillsDiagnoses GKE workload failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, etc.) via logs and events. Use when pods fail to start or crash repeatedly. Don't use for GKE cluster infrastructure provisioning, node pool creation, or non-Kubernetes Google Cloud services.OficialDevOps★ 21.5K
Note și recenzii
—
nicio recenzie
Ai încercat acest skill? Notează-l ca să-i ajuți pe alții. Autentifică-te
Discuție
Intră în discuție. Autentifică-te