Skip to main content

Notifications

Our notifications service routes alerts from across your Crusoe Cloud organization to your team's channels when important events occur. These notification events include infrastructure failures detected by Command Center, budget alerts for exceeded budget thresholds, and more. You can manage your notification preferences using the console, and configure webhooks for Slack—and other webhook endpoints—on the Svix dashboard, which you can access from the console.

How the notification service works​

Crusoe Cloud continuously monitors your infrastructure, usage, and services. When a significant event is detected, the event is published to a notification pipeline and routed to your configured channels with context.

Notification event categories​

Crusoe Cloud groups notification events into the following categories in the console. Review these categories in your Notification Settings page to understand what triggers a notification before configuring your delivery channels.

CategoryDescription
Resource HealthNotifications about hardware issues and automated remediation on your nodes.
User Account and Organization ManagementNotifications about changes to your organization's account settings and access controls.
BillingNotifications that help you monitor and control your cloud spending. See Budget Alerts for details.
Managed AINotifications about Managed AI service updates and issues. These notifications are exclusive to Managed AI users.

Access console and email notifications​

Console notifications are always enabled and don't require configuration.

View notifications in the Crusoe Cloud Console​

To view notifications in the console:

  1. From the console, click the bell icon in the top-right corner to view the latest, unread notifications. You can also dismiss notifications from this view.
  2. Select All Notifications from the modal to view your complete notification history. Previously dismissed notifications still appear in this view.

Receive email notifications​

When email notifications are enabled for a critical event that occurs, notifications are sent automatically to relevant team members.

important

Email notifications are disabled by default for some event types. To receive notifications for these event types, you must enable them in the Notification Settings page.

For critical resource health alerts, emails are sent to all users in an organization who have opted into receiving emails for this notification type through the Notification Settings page. These emails include:

  • VM ID and VM name of the affected node
  • Cluster name and cluster ID
  • A brief description of the detected issue and the action taken
  • A link to the relevant view in Crusoe Cloud Console (requires authentication)

Route notifications to Slack and webhooks​

To deliver notifications to Slack or an external webhook endpoint (for example, PagerDuty, Opsgenie, or custom automation), configure a delivery endpoint in the Svix dashboard (through the Crusoe Cloud Console).

Configure Slack notifications​

Route notifications to a Slack channel so your on-call team sees alerts in real time.

  1. From the console, click the bell icon in the top-right corner.
  2. Click All Notifications, then click Manage Slack/Webhook in the top-right corner. This links out to a separate page.
  3. On the new page, click Add Endpoint.
  4. Select Slack as the endpoint type.
  5. Provide your Slack incoming webhook URL or click Connect to Slack (requires authentication). To generate a webhook URL, follow the Slack documentation on incoming webhooks.
    note

    If the authorization to Connect to Slack or the incoming webhook URL requires approval from your enterprise Slack account, reach out to your IT department to authorize the Svix app for your account.

  6. Select which event types to subscribe to.
  7. Click Create.

Configure webhook notifications​

To integrate with PagerDuty, Opsgenie, custom automation, or other tools, configure a generic webhook endpoint.

  1. From the console, click the bell icon in the top-right corner.
  2. Click All Notifications, then click Manage Slack/Webhook in the top-right corner. This links out to a separate page.
  3. Click Add Endpoint.
  4. Select Webhook as the endpoint type.
  5. Provide your webhook endpoint URL.
  6. Select which event types to subscribe to.
  7. Click Create.

Webhook payloads are delivered as HTTP POST requests with a JSON body containing the event details. The exact structure varies by event type. An example JSON body for a resource.automated_remediation event (for a node replacement) is as follows:

{
  "status": "started",
  "org_name": "Org ABC 123",
  "org_id": "org-abc123",
  "project_name": "ml-training",
  "project_id": "proj-def456",
  "cluster_name": "training-cluster-prod",
  "cluster_id": "cluster-abc123",
  "resource_type": "VM",
  "resource_name": "worker-node-42",
  "resource_id": "vm-xyz789",
  "location": "us-east1-a",
  "workflow_type": "Node Replacement",
  "triggered_by": {
    "error_name": "XID 79: GPU has fallen off the bus"
  }
}

What's next​

  • Topology—Investigate affected nodes in the cluster topology
  • Logs—Review system logs for the nodes referenced in notifications
  • Metrics—Check performance trends around the time of the event
  • AutoClusters—Learn more about automated hardware remediation