# Converting Static Figures to Interactive D3 Charts

This guide explains how to convert PNG figures into interactive D3.js visualizations for this project.

## Overview

Each interactive chart consists of:
1. **JSON data file** in `app/public/data/` (served at `/data/filename.json`)
2. **HTML embed file** in `app/src/content/embeds/` (e.g., `chart-name.html`)
3. **MDX integration** using the `HtmlEmbed` component

## File Structure

```
app/
├── public/data/                     # JSON data (served at /data/*)
│   ├── overall_performance.json
│   ├── calibration_curves.json
│   └── ...
└── src/content/embeds/              # HTML chart implementations
    ├── banner.html                  # Example: scatter plot
    └── calibration-curves.html      # (to create)
```

## Step 1: Understand Your Data

Check the JSON structure in `app/public/data/`. Common patterns:

**Scatter plot** (`overall_performance.json`):
```json
{
  "models": [
    { "name": "Model A", "avg_score": 15.8, "avg_output_tokens_per_turn": 5253, "color": "#FF6B00", "is_open": false }
  ]
}
```

**Line chart / Calibration** (`calibration_curves.json`):
```json
{
  "models": [
    {
      "name": "Model A", "color": "#FF6B00",
      "calibration_points": [
        { "confidence_level": 5, "actual_success_rate": 0.041, "sample_count": 73 }
      ]
    }
  ]
}
```

**Histogram** (`confidence_distribution.json`):
```json
{
  "models": [
    {
      "name": "Model A", "color": "#FF6B00", "total_guesses": 579,
      "distribution": [
        { "confidence_level": 5, "proportion": 0.024, "count": 14 }
      ]
    }
  ]
}
```

## Step 2: Create the HTML Embed

Create a new file in `app/src/content/embeds/`. Use this template:

```html
<div class="d3-CHART-NAME"></div>
<style>
  /* Scoped styles - prefix everything with .d3-CHART-NAME */
  .d3-CHART-NAME {
    width: 100%;
    margin: 10px 0;
    position: relative;
    font-family: system-ui, -apple-system, sans-serif;
  }

  .d3-CHART-NAME svg {
    display: block;
    width: 100%;
    height: auto;
  }

  /* Use CSS variables for theme support */
  .d3-CHART-NAME .axes path,
  .d3-CHART-NAME .axes line {
    stroke: var(--axis-color, var(--text-color));
  }

  .d3-CHART-NAME .axes text {
    fill: var(--tick-color, var(--muted-color));
    font-size: 11px;
  }

  .d3-CHART-NAME .grid line {
    stroke: var(--grid-color, rgba(0,0,0,.08));
  }

  /* Use specific selector to override .axes text */
  .d3-CHART-NAME .axes text.axis-label {
    font-size: 14px;
    font-weight: 500;
    fill: var(--text-color);
  }

  .d3-CHART-NAME .axes text.chart-title {
    font-size: 16px;
    font-weight: 600;
    fill: var(--text-color);
  }

  /* Adjust tick label spacing if needed */
  .d3-CHART-NAME .x-axis text {
    transform: translateY(4px);
  }

  /* Tooltip */
  .d3-CHART-NAME .d3-tooltip {
    position: absolute;
    top: 0; left: 0;
    transform: translate(-9999px, -9999px);
    pointer-events: none;
    padding: 10px 12px;
    border-radius: 8px;
    font-size: 12px;
    line-height: 1.4;
    border: 1px solid var(--border-color);
    background: var(--surface-bg);
    color: var(--text-color);
    box-shadow: 0 4px 24px rgba(0,0,0,.18);
    opacity: 0;
    transition: opacity 0.12s ease;
    z-index: 10;
  }
</style>
<script>
  (() => {
    // D3 loader - reuses existing if already loaded
    const ensureD3 = (cb) => {
      if (window.d3 && typeof window.d3.select === 'function') return cb();
      let s = document.getElementById('d3-cdn-script');
      if (!s) {
        s = document.createElement('script');
        s.id = 'd3-cdn-script';
        s.src = 'https://cdn.jsdelivr.net/npm/d3@7/dist/d3.min.js';
        document.head.appendChild(s);
      }
      const onReady = () => { if (window.d3 && typeof window.d3.select === 'function') cb(); };
      s.addEventListener('load', onReady, { once: true });
      if (window.d3) onReady();
    };

    const bootstrap = () => {
      // Find container (handles multiple instances)
      const scriptEl = document.currentScript;
      let container = scriptEl ? scriptEl.previousElementSibling : null;
      if (!(container && container.classList && container.classList.contains('d3-CHART-NAME'))) {
        const candidates = Array.from(document.querySelectorAll('.d3-CHART-NAME'))
          .filter((el) => !(el.dataset && el.dataset.mounted === 'true'));
        container = candidates[candidates.length - 1] || null;
      }
      if (!container) return;
      if (container.dataset) {
        if (container.dataset.mounted === 'true') return;
        container.dataset.mounted = 'true';
      }

      // Tooltip setup
      container.style.position = container.style.position || 'relative';
      const tip = document.createElement('div');
      tip.className = 'd3-tooltip';
      container.appendChild(tip);

      // SVG setup
      const svg = d3.select(container).append('svg');
      const gRoot = svg.append('g');

      // Chart groups (order matters for layering)
      const gGrid = gRoot.append('g').attr('class', 'grid');
      const gAxes = gRoot.append('g').attr('class', 'axes');
      const gContent = gRoot.append('g').attr('class', 'content');

      // State
      let data = null;
      let width = 800;
      let height = 450;
      const margin = { top: 40, right: 120, bottom: 56, left: 72 };

      // Scales
      const xScale = d3.scaleLinear();
      const yScale = d3.scaleLinear();

      // Data loading - single path since we use public/data/
      const DATA_URL = '/data/YOUR_DATA_FILE.json';

      function updateSize() {
        width = container.clientWidth || 800;
        height = Math.max(300, Math.round(width / 1.78)); // 16:9 aspect ratio
        svg.attr('width', width).attr('height', height).attr('viewBox', `0 0 ${width} ${height}`);
        gRoot.attr('transform', `translate(${margin.left},${margin.top})`);
        return {
          innerWidth: width - margin.left - margin.right,
          innerHeight: height - margin.top - margin.bottom
        };
      }

      function showTooltip(event, d) {
        const rect = container.getBoundingClientRect();
        const x = event.clientX - rect.left;
        const y = event.clientY - rect.top;

        tip.innerHTML = `
          <div style="font-weight: 600; color: ${d.color}">${d.name}</div>
          <div>Value: ${d.value}</div>
        `;

        const tipWidth = tip.offsetWidth || 150;
        const tipHeight = tip.offsetHeight || 80;
        let tipX = x + 12;
        let tipY = y - tipHeight / 2;

        if (tipX + tipWidth > width) tipX = x - tipWidth - 12;
        if (tipY < 0) tipY = 8;
        if (tipY + tipHeight > height) tipY = height - tipHeight - 8;

        tip.style.transform = `translate(${tipX}px, ${tipY}px)`;
        tip.style.opacity = '1';
      }

      function hideTooltip() {
        tip.style.opacity = '0';
        tip.style.transform = 'translate(-9999px, -9999px)';
      }

      function render() {
        if (!data) return;
        const { innerWidth, innerHeight } = updateSize();

        // TODO: Implement your chart rendering here
        // - Update scales with data extent
        // - Draw grid lines
        // - Draw axes
        // - Draw data elements (lines, bars, points, etc.)
      }

      // Initialize
      fetch(DATA_URL, { cache: 'no-cache' })
        .then(r => r.json())
        .then(json => {
          data = json;
          render();
        })
        .catch(err => {
          const pre = document.createElement('pre');
          pre.style.color = 'red';
          pre.style.padding = '16px';
          pre.textContent = `Error loading data: ${err.message}`;
          container.appendChild(pre);
        });

      // Resize handling
      if (window.ResizeObserver) {
        new ResizeObserver(() => render()).observe(container);
      } else {
        window.addEventListener('resize', render);
      }

      // Theme change handling (re-render on light/dark toggle)
      const observer = new MutationObserver(() => render());
      observer.observe(document.documentElement, {
        attributes: true,
        attributeFilter: ['data-theme']
      });
    };

    if (document.readyState === 'loading') {
      document.addEventListener('DOMContentLoaded', () => ensureD3(bootstrap), { once: true });
    } else {
      ensureD3(bootstrap);
    }
  })();
</script>
```

## Step 3: Key Implementation Details

### CSS Variables (Theme Support)

Always use CSS variables for colors that need to adapt to light/dark mode:

| Variable | Purpose |
|----------|---------|
| `var(--text-color)` | Main text, labels |
| `var(--muted-color)` | Secondary text, tick labels |
| `var(--border-color)` | Borders, outlines |
| `var(--surface-bg)` | Tooltip background |
| `var(--page-bg)` | Page background |

### D3 Patterns Used

**Scale setup:**
```javascript
const xExtent = d3.extent(data, d => d.x);
const xPadding = (xExtent[1] - xExtent[0]) * 0.1;
xScale.domain([xExtent[0] - xPadding, xExtent[1] + xPadding])
      .range([0, innerWidth])
      .nice();
```

**Grid lines:**
```javascript
gGrid.selectAll('.grid-x')
  .data(xScale.ticks(6))
  .join('line')
  .attr('class', 'grid-x')
  .attr('x1', d => xScale(d))
  .attr('x2', d => xScale(d))
  .attr('y1', 0)
  .attr('y2', innerHeight);
```

**Axes (basic):**
```javascript
gAxes.selectAll('.x-axis')
  .data([0])
  .join('g')
  .attr('class', 'x-axis')
  .attr('transform', `translate(0,${innerHeight})`)
  .call(d3.axisBottom(xScale).ticks(6));
```

**Axes with inner ticks:**
```javascript
const tickSize = 6;
gAxes.selectAll('.x-axis')
  .data([0])
  .join('g')
  .attr('class', 'x-axis')
  .attr('transform', `translate(0,${innerHeight})`)
  .call(d3.axisBottom(xScale)
    .ticks(6)
    .tickSizeInner(-tickSize)  // Negative = ticks point inward
    .tickSizeOuter(0));        // No outer ticks
```

**Custom shapes (5-point star):**
```javascript
const starPath = (cx, cy, outerR, innerR) => {
  const points = [];
  for (let i = 0; i < 10; i++) {
    const r = i % 2 === 0 ? outerR : innerR;
    const angle = (Math.PI / 2) + (i * Math.PI / 5);
    points.push([cx + r * Math.cos(angle), cy - r * Math.sin(angle)]);
  }
  return 'M' + points.map(p => p.join(',')).join('L') + 'Z';
};

// Use with path elements
gContent.selectAll('.point-star')
  .data(openModels)
  .join('path')
  .attr('d', d => starPath(xScale(d.x), yScale(d.y), radius * 1.2, radius * 0.5))
  .attr('fill', d => d.color);
```

**Data-join for elements:**
```javascript
gContent.selectAll('.point')
  .data(models)
  .join('circle')
  .attr('class', 'point')
  .attr('cx', d => xScale(d.x))
  .attr('cy', d => yScale(d.y))
  .attr('r', 8)
  .attr('fill', d => d.color)
  .on('mouseenter', showTooltip)
  .on('mousemove', showTooltip)
  .on('mouseleave', hideTooltip);
```

## Step 4: Integrate in MDX

In your `.mdx` file:

```mdx
import HtmlEmbed from "../../../components/HtmlEmbed.astro";

<HtmlEmbed
  src="chart-name.html"
  title="Chart Title"
  caption="<strong>Figure N:</strong> Description of what this shows."
/>
```

For frameless embedding (like the banner):
```mdx
<HtmlEmbed src="banner.html" frameless />
```

## Charts to Convert

| Figure | Data File | Chart Type | Status |
|--------|-----------|------------|--------|
| 1 | `overall_performance.json` | Scatter | Done (banner.html) |
| 2 | `calibration_curves.json` | Multi-line | Done (calibration-curves.html) |
| 3 | `confidence_distribution.json` | Grouped histogram | Done (confidence-distribution.html) |
| 4 | `score_vs_failed_guesses.json` | Scatter | TODO |
| 5 | `excess_caution.json` | Box plot | TODO |
| 5b | `tokens_by_turn.json` | Multi-line | Done (tokens-by-turn.html) |
| 6 | `caution_vs_failed_guesses.json` | Scatter | Done (caution-vs-failed-guesses.html) |
| 7 | `by_rule.json` | Strip plot | Done (by-rule.html) |
| 8 | `complexity_analysis.json` | Heatmap | Done (complexity-analysis.html) |
| 9 | `complexity_ratio.json` | Horizontal dot plot | Done (complexity-ratio.html) |

## Testing

1. Run dev server: `cd app && npm run dev`
2. Check the chart loads at the correct URL
3. Verify tooltip interactions
4. Toggle light/dark mode to check theme support
5. Resize the window to verify responsiveness

## Debugging Tips

- Open browser console to see data loading errors
- Check Network tab to verify `/data/filename.json` is being fetched
- If chart doesn't render, check `container.dataset.mounted` isn't already 'true'
- CSS scoping: always prefix selectors with `.d3-CHART-NAME`

## Common Gotchas

### Using `.style()` vs `.attr()` for Dynamic Colors

When setting fill/stroke colors dynamically in D3 based on data, use `.style()` instead of `.attr()`:

```javascript
// WON'T WORK - attr has lower specificity than CSS rules
.attr('fill', d => getContrastColor(d.color))

// USE THIS - inline styles have higher specificity
.style('fill', d => getContrastColor(d.color))
```

This is especially important for text labels where you need to calculate contrast colors dynamically. Example contrast function:

```javascript
function getContrastColor(hexColor) {
  const hex = hexColor.replace('#', '');
  const r = parseInt(hex.substr(0, 2), 16) / 255;
  const g = parseInt(hex.substr(2, 2), 16) / 255;
  const b = parseInt(hex.substr(4, 2), 16) / 255;
  const luminance = 0.299 * r + 0.587 * g + 0.114 * b;
  return luminance > 0.5 ? '#000000' : '#ffffff';
}

// Usage
gLabels.selectAll('.label')
  .data(items)
  .join('text')
  .style('fill', d => getContrastColor(d.color))
  .text(d => d.name);
```

### CSS Specificity for Axis Labels

The generic `.axes text` rule applies to ALL text inside the axes group, including axis labels. To style axis labels differently, use a more specific selector:

```css
/* This won't work - gets overridden by .axes text */
.d3-CHART-NAME .axis-label {
  font-size: 15px;
}

/* Use this instead - more specific */
.d3-CHART-NAME .axes text.axis-label {
  font-size: 15px;
  font-weight: 500;
  fill: var(--text-color);
}
```

### Adjusting Tick Label Position

To move X-axis tick labels down (add spacing from the axis line):

```css
.d3-CHART-NAME .x-axis text {
  transform: translateY(4px);
}
```

### Removing Chart Elements

When you don't need a title or legend:
1. Remove the rendering code from `render()`
2. Remove the CSS styles
3. Adjust margins accordingly (e.g., reduce `margin.top` if no title)