Lab Definition Format

Overview

Lab Schema

Fields marked * are required.

Identity & Classification

Timing & Infrastructure

Learning Content

metadata block (optional)

Include this block when submitting — it helps the review team track authorship and version history.

metadata:
  version: 1.0.0
  author: your-name
  reviewed_by: []
  last_updated: 2026-06-14

environment block (optional)

Declare the infrastructure the lab requires. Used for cost estimation and catalog display.

environment:
  instance_type: t3.micro
  estimated_cost: "$0.01"
  aws_services:
    - ec2

deliverables block (optional)

Briefing & Evidence

The briefing: block gives incident and challenge labs their operational context. It replaces the description text with a structured scenario that reads like a real alert. The evidence: block (top-level, separate from briefing) adds flavour cards — PagerDuty alerts, Slack messages, tickets, and notes.

briefing fields

evidence items

briefing:
  severity: sev-2
  impact: Web service returning 500 on all endpoints
  title: Permission Chaos Detected
  narrative: |
    A junior engineer ran sudo chown -R root:root /var/www/html.
    nginx runs as www-data and can no longer read any files.
    Every request returns 500. Fix permissions — no restarts until you
    understand what broke.

evidence:
  - type: pagerduty
    title: "P2: nginx 500 on all requests"
    content: |
      CRITICAL: prod-web-01 — all requests returning 500
      nginx error log: Permission denied reading /var/www/html
  - type: slack
    title: "#ops-alerts"
    content: |
      [automated] nginx 500 rate: 100% — prod-web-01
      First seen: 14:23 UTC

Validation Checks

Checks run against the live EC2 instance when the student clicks Submit. Every check in the array must pass (or at least one with strategy: any) for the lab to be marked complete. Transient failures are retried up to 3 times automatically.

Validation Block Structure

validation:
  strategy: all        # all checks must pass (default). use 'any' for alternatives.
  default_timeout_seconds: 5
  checks:
    - id: nginx-running
      name: Nginx Running
      type: output_matches
      cmd: systemctl is-active nginx
      contains: active
      failure_hint: Run 'sudo systemctl restart nginx' and check 'journalctl -xe'

Check Fields

output_matches

Runs cmd via SSH. Passes if exit code is 0 and the output satisfies the match criteria. Covers 90% of validation needs.

- id: nginx-running
  name: Nginx Running
  type: output_matches
  cmd: systemctl is-active nginx
  contains: active
  failure_hint: Run 'sudo systemctl restart nginx' and check 'journalctl -xe'

- id: disk-below-90
  name: Disk Below 90%
  type: output_matches
  cmd: df / | awk 'NR==2 {gsub("%",""); print ($5 < 90) ? "ok" : "full"}'
  value: ok
  failure_hint: Run 'du -sh /* 2>/dev/null | sort -h' to find what is using space

ssh

Runs cmd via SSH. Passes if exit code is 0. Output is not checked — use for existence checks where you only care whether the command succeeds.

- id: config-exists
  name: Config File Exists
  type: ssh
  cmd: test -f /etc/myapp/config.yaml
  failure_hint: Create the config file at /etc/myapp/config.yaml

- id: service-unit-present
  name: Unit File Written
  type: ssh
  cmd: test -f /etc/systemd/system/myapp.service
  failure_hint: Write the unit file to /etc/systemd/system/myapp.service

http_get

Makes an HTTP GET to url. Passes if the response is 2xx, or matches expect_status if set.

- id: app-responding
  name: App HTTP Responding
  type: http_get
  url: "http://{{EC2_IP}}:8080/health"
  failure_hint: Check the app with 'systemctl status myapp' and 'journalctl -u myapp -n 50'

- id: auth-enforced
  name: Auth Required on Admin Route
  type: http_get
  url: "http://{{EC2_IP}}:8080/admin"
  expect_status: 401

multi_check

Runs a list of sub-checks. With operator: all every sub-check must pass. With operator: any at least one must pass.

- id: stack-healthy
  name: Full Stack Healthy
  type: multi_check
  operator: all
  checks:
    - id: nginx-active
      name: Nginx Active
      type: output_matches
      cmd: systemctl is-active nginx
      contains: active
    - id: app-active
      name: App Active
      type: output_matches
      cmd: systemctl is-active myapp
      contains: active

Writing Good Checks

Hint System

Every lab requires three hint levels. The levels are progressive — each reveals more than the last. Good hints are the difference between a student who learns and one who rage-quits.

hint_policy: show_level_1_automatically
hints:
  - level: 1
    text: |
      Something swept ownership across a directory it should not have touched.
      Find out which directory is affected and what nginx needs to read it.

  - level: 2
    text: |
      Check what nginx is trying to read:
        sudo journalctl -u nginx --since "5 min ago"
        ls -la /var/www/html

      nginx runs as www-data. Fix ownership:
        sudo chown -R www-data:www-data /var/www/html
        sudo chmod 755 /var/www/html

  - level: 3
    text: |
      # Confirm the problem
      sudo journalctl -u nginx --since "5 min ago" | grep "Permission denied"
      ls -la /var/www/html   # should show root:root — that is wrong

      # Fix ownership
      sudo chown -R www-data:www-data /var/www/html
      sudo chmod 755 /var/www/html
      sudo find /var/www/html -type f -exec chmod 644 {} \;

      # Verify nginx can now read the files
      sudo nginx -t
      curl -s -o /dev/null -w "%{http_code}" http://localhost

Terraform Modules

The terraform_module field tells the platform which infrastructure template to provision for your lab. Choose the module that matches your scenario — you do not write or modify Terraform.

Full Example

A complete incident-mode lab — all required fields, a briefing, evidence cards, deliverables, and validation checks:

schema_version: 1

metadata:
  version: 1.0.0
  author: your-name
  reviewed_by: []
  last_updated: 2026-06-14

id: linux-disk-full
title: "Disk Full: Production Filesystem Saturated"
description: >
  The root filesystem is at 100%. Every write is failing.
  Find what consumed the disk, clear space, and restore the service.

track: linux
module: linux-storage
sort_order: 80
difficulty: intermediate
engineer_level: L2
mode: incident

tags:
  - linux
  - disk
  - storage
  - troubleshooting
  - incident

estimated_minutes: 45
timeout_minutes: 90
terraform_module: labs/linux-ec2
scenario_id: disk-full

environment:
  instance_type: t3.micro
  estimated_cost: "$0.01"
  aws_services:
    - ec2

prerequisites:
  - linux-filesystem-anatomy

briefing:
  severity: sev-1
  impact: Service down — production impact
  title: Filesystem 100% — write errors
  narrative: |
    The production web server is returning 500 errors on every request.
    The app cannot write logs or temp files. The root filesystem hit 100%.
    Identify what consumed the disk, clear space, and restore the service.

evidence:
  - type: pagerduty
    title: "P1: Filesystem 100% — write errors"
    content: |
      CRITICAL: / at 100% on prod-web-01
      App is throwing ENOSPC on every write
      Services failing: nginx, app, cron
  - type: slack
    title: "#ops-alerts"
    content: |
      [automated] ALERT: prod-web-01 disk usage 100%
      First seen: 14:23 UTC — still firing

objectives:
  - Identify what is consuming disk space
  - Remove the bloat without deleting production data
  - Restore service without a restart

success_criteria:
  - df / shows less than 90% used
  - nginx is active and responding
  - App is responding on port 8080

deliverables:
  - path: /tmp/disk-cleanup.log
    description: |
      A log of what you found and what you removed:
        - Output of du -sh showing the bloat
        - Commands run to free space
        - df -h after cleanup

hint_policy: show_level_1_automatically
hints:
  - level: 1
    text: |
      Something wrote a lot of data somewhere.
      Find out what is large. Find out what is growing.

  - level: 2
    text: |
      df -h — which filesystem is full?
      sudo du -sh /* 2>/dev/null | sort -h — what is largest?
      sudo journalctl --disk-usage — are logs the culprit?
      ls -lhS /tmp /var/log /var/cache — spot large files

  - level: 3
    text: |
      df -h
      sudo du -sh /* 2>/dev/null | sort -h
      sudo journalctl --vacuum-size=200M
      sudo find /var/log -name "*.gz" -delete
      sudo apt-get clean
      df -h
      sudo systemctl restart nginx

validation:
  strategy: all
  default_timeout_seconds: 5
  checks:
    - id: disk-below-90
      name: Disk Below 90%
      type: output_matches
      cmd: df / | awk 'NR==2 {gsub("%",""); print ($5 < 90) ? "ok" : "full"}'
      value: ok
      failure_hint: Run 'du -sh /* 2>/dev/null | sort -h' to find what is using space

    - id: nginx-running
      name: Nginx Running
      type: output_matches
      cmd: systemctl is-active nginx
      contains: active
      failure_hint: Start nginx with 'sudo systemctl start nginx'

    - id: app-responding
      name: App HTTP Responding
      type: http_get
      url: "http://{{EC2_IP}}:8080/health"
      failure_hint: Check app logs with 'journalctl -u myapp -n 50'

completion_message: |
  A full disk silently kills services that try to write.
  You can now isolate the cause, clean safely, and verify the fix.
  Set up disk monitoring so you see this before it hits 100%.

Submit a Lab

Good labs come from real incidents. If you have been through an outage and want to turn it into a lab, here is what to prepare: