Introducing Agentpane

agentpane is a local web UI for AI coding agents. You run npx agentpane, it opens in your browser, and you can talk to Claude Code, Codex, or any agent that speaks ACP (Agent Client Protocol) — all from a multi-pane interface with streaming, persistence, and session management:

npx agentpane
# → http://localhost:6767

Pick an agent, pick a working directory, and start a session. The agent runs as a subprocess on your machine. No cloud, no deployment, no accounts.

Multi-Pane, Multi-Agent

The main thing I wanted was the ability to run multiple agents at once and see them side by side. agentpane gives you up to four resizable panes with tabs. You can drag sessions from the sidebar into any pane, or drag a tab to split into a new one. Layout persists across refreshes.

This means you can have Claude Code working on your backend in one pane and Codex refactoring your frontend in another — same screen, same app.

Architecture

The app is two processes:

• API server (Hono, port 3456) — spawns agents, manages sessions, handles persistence
• Web frontend (Next.js, port 6767) — the UI, with /api calls rewritten to the backend

The backend is built with Effect.ts and uses dependency injection through composable service layers:

SessionRepo (DB)
  → ConnectionManager (agent subprocess lifecycle)
  → PromptEngine (turn orchestration)
  → EventHub / EventBroadcaster (SSE)
  → WriteQueue (batched DB writes)

Agents communicate over ACP — JSON-RPC 2.0 over stdio. When you send a prompt, the PromptEngine creates turn records, passes the message to the agent subprocess via ACP, and streams results back through Server-Sent Events.

Streaming with Reconnection Safety

The frontend subscribes to an SSE endpoint per session. Each event gets a monotonic ID. The EventBroadcaster keeps a ring buffer (512KB / 1000 events) so that if your browser disconnects and reconnects, it replays missed events using the Last-Event-ID header. No gaps in the conversation, no manual refresh needed.

Crash-Safe Persistence

All state goes to a local SQLite database at ~/.agentpane/agentpane.db. But writes don't happen inline — the WriteQueue accumulates operations and flushes them in a batch every 50ms. Before flushing, ops are persisted to a recovery table. If the server crashes mid-flush, it picks up where it left off on restart. No message loss.

-- The recovery table
CREATE TABLE write_queue_ops (
  id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  op_json TEXT NOT NULL,
  created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

What Agents See

agentpane doesn't care what model powers the agent — it talks ACP. When you connect to a session, the ConnectionManager:

1. Resolves the agent binary (walks node_modules/.bin/ for npm compatibility)
2. Spawns it as a child process with stdio pipes
3. Negotiates ACP capabilities (auth, config, modes, commands)
4. Attempts session resumption if reconnecting to a previous session

The UI adapts to whatever the agent supports. If the agent exposes configuration options, they show up as dropdowns. If it supports slash commands, you get autocomplete. If it streams thought blocks, they render as collapsible sections.

The Frontend

The web app is Next.js 16 with React 19 and the React Compiler — no manual useMemo or useCallback anywhere. State is split into two context layers:

• SessionProvider — active session, health checks, session CRUD
• LayoutProvider — pane configuration, tab ordering, drag-and-drop

Server state (sessions, conversations, token usage) is managed with TanStack React Query, invalidated on SSE events. Markdown renders as it streams in using Streamdown with Shiki syntax highlighting.

Try It

npx agentpane

The source is on GitHub. Docs are at apps/docs.

Introducing Helm

helm is a typed TypeScript framework for AI agents. Instead of shelling out and parsing strings, agents call typed functions with structured inputs and outputs:

import { createHelm, git, fs, grep } from "@bgub/helm";
 
const agent = createHelm({
  permissions: {
    "fs.read": "allow",
    "fs.write": "ask",
    "fs.remove": "deny",
    "git.status": "allow",
    "git.*": "ask",
  },
  onPermissionRequest: async (operation, args) => {
    return confirm(`Allow ${operation}?`);
  },
})
  .use(fs())
  .use(git())
  .use(grep());
 
const { staged, unstaged, branch } = await agent.git.status();
const { content } = await agent.fs.read("./package.json");

You register "skills" — groups of related operations — with a builder pattern. TypeScript infers the full type at each step. Every operation has a permission level: allow (runs immediately), ask (pauses for approval), or deny (throws PermissionDeniedError). Permissions resolve by precedence: exact match → wildcard → skill author default → global default.

helm ships with built-in skills for the things agents do every day: fs, git, grep, edit, shell, http. You can define custom skills and they get types, search, and permissions for free.

The Demo

I built a chatbot where the agent has exactly two tools: search and execute.

search does keyword lookup over all registered helm operations — the agent calls it to discover what's available and learn the function signatures:

agent.search("file read");
// → [{ qualifiedName: "fs.read",
//      description: "Read a file and return its content as a string",
//      signature: "(path: string) => Promise<{ content: string }>", ... }]

execute takes arbitrary JavaScript code and runs it against the helm agent API. The LLM writes code like:

const { staged, unstaged, branch } = await agent.git.status();
return { branch, staged: staged.length, unstaged: unstaged.length };

Two tools in context, regardless of how many skills are registered. The agent discovers what it needs on demand and writes code to use it.

Sandboxing Untrusted Code

The execute tool runs whatever JavaScript the LLM writes. To make that safe, the demo sandboxes it using SES (Secure ECMAScript) in a child process.

SES lockdown() freezes every JavaScript intrinsic — Object, Array, Promise, Function, all of it. The code runs inside a Compartment, an isolated global scope with access to exactly two things: an agent proxy and a stubbed console. fetch, require, import, process, fs — none of it exists in the compartment. The only way to do anything interesting is through the agent proxy.

The agent inside the sandbox isn't the real helm agent — it's a recursive Proxy that intercepts property access and function calls. When the code calls agent.git.status(), the proxy sends an IPC message to the parent process. The parent calls the real method on the real helm agent, runs the full permission check, and sends the result back. If the operation is set to "ask", the parent pauses for user approval before responding. If it's "deny", the error propagates back through IPC.

The sandboxed code has no idea any of this is happening. It just sees its await resolve with a value. The only way to interact with the outside world is through helm's permission-gated operations.

The Permission UI

The chat UI has a sidebar listing every registered skill and operation, each with an allow/ask/deny toggle. Changing a permission takes effect on the next message.

When the LLM hits an operation set to "ask", the server streams an approval request to the frontend. The tool call shows an inline banner with Allow and Deny buttons. The server blocks on a Promise<boolean> until the user clicks one.

If the user denies, PermissionDeniedError propagates all the way back and the LLM sees it in the tool result. It can explain why it needs the permission, try a different approach, or give up.

Inspiration

This architecture — giving the agent a code execution tool instead of dozens of individual tools — was inspired by Cloudflare's code mode for their MCP server, where they reduced token usage by 99.9% by replacing 2,500+ API endpoint tools with search + execute. Rhys Sullivan's similar idea crystalized the idea for me: the combination of code execution, discoverability, and a granular permission model means the agent can do anything but can't go off the rails.

Try It

npm install @bgub/helm

The source is on GitHub. The demo app is in apps/demo.

Getting the Internship

Since I did a study abroad in Morocco in the fall, I was behind on the internship game. I still didn't have an offer in late April, so I reached out to Vercel and was luckily able to get one after a few rounds of interviews!

I had initially hoped to join the v0 or AI SDK teams, but I ended up on the Next.js/Turbopack team, which turned out to be great (more about that in a bit). Things were last-minute and hectic, but I moved to SF on June 1st. I lived in a hostel in the Tenderloin until I could find an apartment (I eventually found a $1200 private room in Chinatown — unimaginably expensive for Utah, where I'm from, but unimaginably cheap compared to other prices in the area.)

What I Worked On

Things were last minute, but I worked with my manager to choose an intern project. We decided that I would work on adding typed routes support to Next.js.

In the end, I worked on TypeScript support broadly in Next.js. I added automatic route type generation, stabilized typed links + type validation, and brought both to Turbopack. I also deprecated the next lint command, brought Biome support to Next, and worked on some internal projects like a traces viewer for Turbopack.

In addition to these, I built a few projects that I couldn't bring to completion in time, including a 7-chapter tutorial on building your own React framework (which I mentioned in my summer review and hope to publish soon!)

The Experience

I really enjoyed Vercel's culture. We had an AI-focused hackathon with wide participation — I built a chatbot with code execution capabilities using Sandboxes. Generally people aren't super chatty and are fairly "heads-down" grinding on work, but I had some good opportunities to chat with company leadership like Tom Occhino, Malte Ubl, and Guillermo Rauch.

The coolest thing was working with teammates who are insanely talented. I've been doing web dev and using React for over 8 years now, so it felt like meeting your favorite celebrity to chat with and work alongside people who helped build React or other significant parts of the ecosystem!

If you're considering working or interning at Vercel, feel free to reach out with questions! I also made some documents with tips that will be shared with new interns.

In the meantime, here's a brief overview of the non-work-related events of my summer. This will function as a cross between a journal entry, blog post, and "brag list" that potential employers can see ;), so apologies in advance if it seems like I'm tooting my own horn.

AI Projects

I finally learned how to write GPU kernels using a few cool resources! In conjunction with this, I made a (very much WIP) PyTorch-like library for WebGPU-accelerated computation in JavaScript. I'm calling it shade (get it, like shaders?) I have yet to add backpropagation, though.

I also released a few projects related to tokenization, a field which I've been very interested in: tokka-bench, a toolkit for evaluating/benchmarking tokenizers across multiple languages, and tokka, a toolkit for easily training tokenizers on custom mixes of data.

Since working with large amounts of data using just HuggingFace Datasets is difficult, I wrote hf_to_mds — a script for easily converting datasets to MosaicML Streaming format and uploading them to the HuggingFace Hub! I made an MDS version of the Wikipedia dataset and would like to convert more, but am still waiting to hear back about getting my storage limits raised.

Finally, I wrote a LessWrong article about how humans are changing their speech patterns to sound distinct from LLMs. It was featured on the frontpage, which is pretty neat!

Random Stuff

• After years of procrastination, I finally learned the Vim keybindings! It's been surprisingly useful. I switched to LazyVim for a while, and now I'm on Zed.
• Speaking of Zed, I'm working on a LazyVim-inspired config with leader-key keyboard shortcuts. Hope to release it soon! I also merged a PR to let users hide the title bar.
• I finally bit the bullet and rebranded from @nebrelbug to @bgub on GitHub! I'm @bgub_ on X... lmk if you know how I can get @bgub!
• This website looks significantly better after I updated the UI with styling help from v0. And I added a cool "Explain Like I'm 5" button for streaming AI-powered post summaries!
• Over the summer I used macOS long-term for the first time. I liked it less than I expected (auto-tiling is much worse than on Linux) so I am still daily-driving Linux on my personal machine. I used Omarchy for a few weeks, then switched back to Fedora with COSMIC desktop, which I may have helped inspire (but realistically probably didn't.)
• Trying out so many different OS options made me tired of setting up my computer over and over, so I learned how to use Nix! I made a config with home-manager for my work MacBook, and an associated public template. My setup also works with Linux machines, but I haven't upstreamed those changes to the template.
• At Vercel, I wrote a 7-chapter tutorial explaining how to build a React framework from scratch. I haven't published it yet but hope to soon!

Finally, a few updates from the last few weeks (it's not technically summer anymore but I'm too tired to write more posts).

• I released a template for TS libraries using modern tooling like release-please, Vitest, tsdown, and Biome.
• I released a new, ESM-only version of eta using said tooling.
• I updated eta.js.org and squirrelly.js.org to use FumaDocs instead of Docusaurus, support better versioning, etc.

Eight years ago, I released my first open-source TypeScript library — Squirrelly — which contained two files, package.json and index.js. Five years ago, I released Eta with many more features including testing, linting, bundling, and CI/CD.

I thought was a pretty solid development setup, but times change and the JavaScript ecosystem moves fast. New tools have emerged, best practices have evolved, and the complexity of properly publishing an npm package has somehow gotten both easier and more overwhelming at the same time.

Just look at the package.json "exports" field evolution if you want a headache. Or try figuring out the right combination of TypeScript configs, bundlers, and CI workflows to publish a library that works seamlessly across Node, Deno, Bun, and browsers. It's surprisingly tricky to get right.

That's why I built ts-base — a modern TypeScript library starter template that handles all of this complexity for you. It's opinionated, battle-tested, and designed to work out-of-the-box with every major JavaScript runtime.

What Is TS-Base?

ts-base is a TypeScript library template that embraces modern tooling and automated workflows. Instead of starting from scratch or copying outdated boilerplate, you get a complete development environment that includes linting, testing, building, releasing, and publishing — all pre-configured and ready to go.

The template is built around three core principles:

• Multi-runtime first: Works seamlessly across Node, Deno, Bun, and browsers
• Automation over configuration: Minimal setup, maximum automation
• Modern tooling: ESM-only, latest TypeScript, and carefully chosen dependencies

Multi-Runtime Architecture

The heart of ts-base is its runtime-agnostic design. Instead of trying to make one file work everywhere (and dealing with compatibility headaches), the template uses a clean separation:

// src/internal.ts - Core logic, no runtime-specific APIs
export function add(a: number, b: number): number {
  return a + b;
}
 
export function greet(name: string, options = {}): string {
  const base = `Hello, ${name}`;
  return options.shout ? `${base.toUpperCase()}!` : `${base}.`;
}

// src/index.ts - Node/Bun adapter
export { add, greet } from "./internal";
import { randomBytes } from "node:crypto";
 
export function getSecureRandomId(): string {
  const timePart = Date.now().toString(36);
  const bytes = randomBytes(12).toString("base64url");
  return `${timePart}-${bytes}`;
}

// src/browser.ts - Browser adapter
export { add, greet } from "./internal";
 
export function getSecureRandomId(): string {
  const timePart = Date.now().toString(36);
  const array = new Uint8Array(12);
  crypto.getRandomValues(array);
  const rand = btoa(String.fromCharCode(...array))
    .replaceAll("+", "-")
    .replaceAll("/", "_")
    .replaceAll("=", "");
  return `${timePart}-${rand}`;
}

This gives you clean imports for every runtime:

// Node/Bun
import { add, getSecureRandomId } from "@your-package/ts-base";
 
// Browser (via bundler)
import { add, getSecureRandomId } from "@your-package/ts-base/browser";
 
// Deno (direct TypeScript imports)
import {
  add,
  greet,
} from "https://jsr.io/@bgub/ts-base/<version>/src/index.ts";

The build system uses tsdown to create two optimized bundles: one for Node environments and a separate minified bundle for browsers, both with sourcemaps.

Developer Experience

ts-base consolidates your tooling around a few excellent choices:

Biome replaces both ESLint and Prettier with a single, fast tool. No more configuration conflicts or plugin incompatibilities — just consistent formatting and linting that works out of the box.

Vitest provides lightning-fast testing with built-in coverage reporting and customizable thresholds. Tests run in parallel, support TypeScript natively, and include helpful features like mocking and snapshots.

Size Limit monitors your bundle size automatically. It runs in CI and comments on pull requests when your changes would increase the bundle size, helping you catch bloat before it ships.

The TypeScript configuration is optimized for modern bundlers with settings like moduleResolution: "bundler" and allowImportingTsExtensions: true that work great with tools like Vite, Rollup, and esbuild.

Automated CI/CD Pipeline

One of ts-base's biggest strengths is its complete CI/CD setup. Every aspect of code quality and publishing is automated:

Quality Gates: Every pull request triggers linting, type checking, testing, and coverage reporting. The CI uploads coverage to Codecov and comments on PRs with size impact reports.

Release Management: Instead of complex semantic-release configurations, ts-base uses Google's Release Please. When commits land on main, Release Please automatically opens a "Release PR" that updates version numbers, generates changelogs, and creates release tags.

Automated Publishing: When you merge the Release PR, GitHub Actions automatically builds and publishes your package to both npm and JSR with full OIDC provenance and security attestation.

Conventional Commits: PR titles are automatically linted to follow conventional commit format, ensuring consistent changelog generation.

Why This Approach Works Better

Most TypeScript library templates I've seen are either too minimal (leaving you to figure out CI, publishing, and multi-runtime support) or overcomplicated with dozens of dependencies. I've seen templates with packages like @commitlint/cli, @commitlint/config-conventional, @semantic-release/changelog, @semantic-release/git, @semantic-release/github, @semantic-release/npm, and more just for CI publishing!

ts-base takes a different approach with just 8 total dev dependencies. By choosing Release Please over semantic-release, Biome over ESLint+Prettier, and Vitest over Jest, you get a simpler dependency graph that's easier to maintain and less likely to break.

The automation philosophy means less configuration and fewer places for things to go wrong. Release Please handles version bumping, changelog generation, and release creation in one tool. The GitHub Actions workflows handle everything else.

The Magic of Release Please

Release Please deserves special attention because it transforms how you think about releases. Instead of manually bumping versions or configuring complex semantic-release pipelines, Release Please works like this:

1. You merge commits to main using conventional commit messages
2. Release Please automatically opens/updates a "Release PR" with version bumps and changelog entries
3. When you're ready to release, simply merge the Release PR
4. GitHub Actions automatically publishes to npm and JSR

The system supports pre-releases too. If you release an alpha or beta version, it automatically publishes under the "next" tag on npm. You can override version bumps using Release-As: 2.0.0 in commit messages, and you can maintain multiple release branches (like 2.x and 3.x) that each get their own Release PRs.

Getting Started

Setting up ts-base is straightforward:

1. Clone and customize: Clone the repository, remove the .git folder, and update package.json, jsr.json, and .release-please-manifest.json with your package details.

2. Claim your package: Set the version to 0.0.0 in all config files, then run npm publish locally to claim your package name on npm.

3. Configure publishing: In npm, set your package to require 2FA for authorization only (not publishing), then add your GitHub workflow as a trusted publisher. On JSR, create your package and add the repository as a trusted source.

4. Set up GitHub: Push to GitHub, add CODECOV_TOKEN as a repository secret, and configure branch protection rules.

5. Start developing: Add your code to src/, write tests, and push commits. Release Please will handle the rest.

I recommend configuring GitHub to only allow squash merging and using "pull request title and commit details" as the default commit message. This keeps your commit history clean and ensures conventional commit compliance.

Best Practices & Tips

Repository Settings: Enable branch protection on main with required status checks. Disable merge commits to keep history linear.

Entry Points: Use the main export (@your-package) for Node/Bun, the browser export (@your-package/browser) for bundled browser code, and direct TypeScript imports for Deno.

Customization: If you don't need separate Node/browser builds, delete the unused configuration. The template is designed to be trimmed down to your specific needs.

Testing Strategy: The template includes examples of testing both shared and platform-specific code, including mocking browser APIs in the Node test environment.

Wrapping Up

Publishing a TypeScript library shouldn't require a PhD in tooling configuration. ts-base gives you a modern, opinionated foundation that handles the complexity so you can focus on building great software.

The template represents eight years of lessons learned from maintaining open source projects. Ready to try it out? Check out the ts-base repository and start building your next library.

I consider Eta to be mostly "a completed work" since v3, which was released in June 2023. But I wasn't satisfied with the bundling, testing, linting, and CI/CD aspects of the project so I rewrote it! (Then extracted my changes, after the fact, into a new TS library template called ts-base).

There are a few more changes to be aware of:

• Eta is now ESM-only by default. It's 2025 and this is the right choice!
• I transferred the repository (and other associated repositories) to my personal GitHub account, bgub. Since I'm the primary contributor, I think this will simplify maintenance and reduce confusion.
• Eta will continue to be published on https://deno.land/x/, but I recommend that users use https://jsr.io/@bgub/eta instead.
• To use the browser-compatible version of Eta, you must import from eta/core, since we're now using much cleaner "exports" in our package.json.

(In a hurry? Visit tokka-bench.streamlit.app)

Several months ago, I began working on a new project in my free time — pretraining a small, multilingual LLM. As quests tend to do, mine wandered, and I became very interested in one specific aspect of model training: tokenization.

Today I want to share a framework for evaluating tokenizers, but also explain how tokenizers can help us understand:

• What data sources a given model may have been trained on
• Why some LLMs (especially proprietary models, like ChatGPT, Claude, and Gemini) perform vastly better than others at multilingual tasks
• Why Claude, Gemini, and GPT 4o onwards have closed-source tokenizers
• Why some OSS models are better than others for fine-tuning

Technical Background

Script Encoding & Grammar

Understanding tokenization starts with understanding how text is encoded at the byte level. All language is encoded with UTF-8, but different scripts require vastly different numbers of bytes to encode the same semantic content. English averages just above 1 byte per character, making it incredibly compact. Arabic needs 2+ bytes per character, while Chinese can require 3+ bytes per character to properly encode.

Beyond encoding efficiency, languages have fundamental grammatical differences that affect how information is packed into words. Synthetic languages will pack a lot of syntactic information into single words. For example, I speak Czech, where a phrase like "vzali se" would translate to "they married each other" in English. This grammatical density makes it difficult to compare encoding efficiency.

Tokenization

LLMs don't operate directly on bytes — they operate on "tokens", which are like symbols corresponding to groups of bytes. Most modern tokenizers use Byte Pair Encoding (BPE), starting with individual bytes and iteratively merging the most frequent pairs to build up a vocabulary of subword units.

There are some alternative approaches like the Byte Latent Transformer, but so far they haven't really taken off yet in production systems.

The technical decisions in tokenizer design are numerous and consequential:

• Do you add prefix spaces? (So the "hello" in "hello world" and " hello world" are tokenized the same?)
• Do you disallow byte merges across whitespace boundaries? What about across script boundaries?
• Do you use an unknown (UNK) token or fallback to bytes when encountering out-of-vocabulary sequences?

Hopefully this helps you understand Karpathy’s classic post on tokenization:

How Tokenization Affects PretrAIning

The relationship between tokenizers and pretraining data creates a complex web of effects that fundamentally shape model capabilities. Tokenizers are often trained on the pretraining data of the LLM they will be used in, but different languages get different levels of "coverage" in the tokenizer vocabulary.

Let’s take an example: Khmer. Since Khmer has fewer online resources, less of a tokenizer’s vocabulary will represent decodings into Khmer than English. This coverage disparity means that encoding the same number of words in Khmer will require many more tokens than English. But here's where it gets problematic: pretraining often uses proportional splits of different languages based on token count. This means that you might train on 10 million tokens of English text and 1 million tokens of Khmer, hoping to have a 10:1 ratio of content. But the Khmer text actually represents way less than 10% of the words compared to the English text!

The semantic implications are even more severe. Khmer tokens, because there are fewer, are more likely to represent letters or consonant pairs rather than whole semantic units. This means that models can't "store" concepts, attributes, definitions, and other semantic knowledge in embedding vectors quite as easily for underrepresented languages.

There's a vibrant open-source community making fine-tunes of OSS foundation models for smaller languages. If your tokenizer doesn't handle foreign languages well, fine-tuning will be more difficult and probably require extending the tokenizer with custom tokens. On the other hand, introducing "partially-trained" tokens (tokens that won't show up in the pretraining data) can confuse the LLM and even allow for "token attacks."

How Tokenization Affects Inference

The tokenization disparities that emerge during pretraining continue to create problems during inference. Text in low-resource languages (languages with few online resources) takes many more tokens to represent, causing multiple cascading issues:

Performance degradation: Slower throughput becomes a significant issue when every sentence requires 2-3x more tokens to represent. Users get sluggish responses, and serving chats costs providers more money.

Context limitations: Longer sequences fill up the context window faster, and recall performance degrades as the model struggles to maintain coherent understanding across the inflated token sequences.

Generation quality: Token selection during generation can introduce errors. More tokens per word means more "chances to mess up" per word, potentially leading to compounding drift where small errors in token selection cascade into larger semantic failures.

Evaluating Tokenizers with Tokka-Bench

I built a tool to easily explore tokenizer performance across 100 natural languages and 20 programming languages. I started by evaluating 7 tokenizers: Gemma 3, GPT-2, GPT-4, gpt-oss, Kimi K2, Llama 3, and Qwen 3.

The project has multiple components designed for different use cases:

Open-source repository: You can clone it and run benchmarks locally. https://github.com/bgub/tokka-bench

Live dashboard: In addition to the code for running the benchmarks, I also made a live dashboard! https://tokka-bench.streamlit.app/

This allows you to easily select combinations of languages and tokenizers to compare, and switch between different metrics to understand the multifaceted nature of tokenizer performance.

Datasets and Methodology

Datasets: For evaluation, I use three high-quality datasets that represent different domains of text:

• FineWeb for English content
• FineWeb 2 for other human languages
• StarCoder for programming languages

Per-language metrics: I sample 2MB of text from each dataset and tokenize it to calculate language-specific performance metrics. This approach has an important limitation: due to UTF-8 encoding differences, 2MB represents vastly different amounts of semantic content across languages. A better approach might compute a global "scaling constant" based on equivalent semantic content—for example, using parallel translations to normalize by the byte size of Harry Potter in English divided by semantic units. As it stands, cross-linguistic comparisons should be interpreted cautiously, and it's more reliable to compare different tokenizers on the same language.

Vocabulary metrics: For analyzing tokenizer vocabularies themselves, I sample 10,000 tokens randomly from each tokenizer's vocabulary and analyze their decoded properties.

Language unit definitions: Different languages structure information differently, so I define "units" for fertility and splitting metrics as follows:

• Whitespace languages: tokens per word (space-separated units)
• Character-based languages (e.g., Chinese, Japanese, Thai): tokens per character (excluding whitespace)
• Syllable-based languages (e.g., Tibetan): tokens per syllable (tsheg-separated units, with fallback methods)

Per-Language Metrics and Results

Let's compare GPT-2, Llama 3, and Kimi K2 on a subset of popular languages to illustrate the kinds of insights tokka-bench can reveal. I've chosen these three to show the evolution of tokenization approaches over time.

Context for each:

• GPT-2 has a vocab size of ~50K and was released in February 2019
• Llama 3 has a vocab size of ~128K and was released in April 2024
• Kimi K2 has a vocab size of ~164K and was released in July 2025

Efficiency (Bytes per Token)

bytes_per_token: Average UTF-8 bytes per token (total_bytes / total_tokens). Higher values indicate more efficient compression of text into tokens.

The efficiency differences reveal training priorities and data composition. Languages with higher bytes-per-token ratios are being compressed more effectively, suggesting either better vocabulary allocation or more training data for vocabulary learning.

Important limitation: This metric doesn't account for UTF-8 encoding differences across scripts. For example, Hindi achieves artificially high efficiency simply because each character requires 3 bytes to encode—allocating just 50 tokens to represent each character in the Hindi alphabet would yield 3 bytes/token efficiency. However, many Hindi characters are formed by combining consonants with vowel signs or consonant clusters, so adding tokens for these combinations (representing 6-9 bytes each) can inflate efficiency metrics while still providing poor semantic coverage. This doesn't reflect genuine semantic efficiency. The metric works best for comparing different tokenizers on the same language rather than comparing efficiency across diverse scripts.

Coverage (Unique Tokens)

unique_tokens: Count of distinct token IDs used when encoding sample text in each language. Higher values suggest better coverage of that language's script(s) with fewer byte-fallbacks to individual characters.

I generally find coverage to be the most indicative of the linguistic breakdown of pretraining data. Look at how much higher the Mandarin script coverage is for Kimi K2 than the other tokenizers! This is exactly what we'd expect, since it's a Chinese LLM with vocabulary specifically optimized for Chinese text.

The coverage hierarchy reveals clear training priorities:

• Chinese has exceptional coverage in Kimi K2
• English has the best script coverage by far across all models except second-best in Kimi K2
• Latin languages (especially the Romance languages) perform well
• Other Latin alphabet languages follow
• Korean, Japanese, and Russian show moderate coverage
• Hindi, Persian, and Khmer lag significantly behind

Note on cross-linguistic comparison: Since coverage is calculated on fixed 2MB text samples, different languages requiring different numbers of UTF-8 bytes to represent equivalent semantic content makes direct comparison problematic. A more principled approach would calculate coverage as a percentage relative to a normalized baseline—but for now, the metric is most reliable for comparing different tokenizers on the same language rather than comparing coverage across diverse scripts.

Word Splitting Rate

word_split_pct: Percentage of units that split into more than one token. Units are defined by language (words for whitespace languages, characters for character-based languages, syllables for syllable-based languages). Lower values generally indicate better alignment with natural unit boundaries.

In Mandarin, Kimi K2 has the lowest continued word rate! Only 4% of tokens are continuing a word.

Disclaimer: remember, for character-based languages like Mandarin, the metric actually measures per character, not per word. Words in Mandarin can be 1 character or more — most are actually two characters long — but that's computationally complex to determine quickly in a benchmark.

Subword Fertility

subword_fertility: Tokens per unit, where units are defined based on language structure (see methodology above). Lower values are better — closer to 1 means fewer pieces per semantic unit.

In Mandarin, Kimi K2 has the lowest subword fertility! The fertility is below 1, meaning on average each token represents more than 1 character.

Vocabulary Metrics (Aggregated across All Languages)

Calculated by sampling tokens from the tokenizer’s vocabulary, then decoding them:

tokens_starting_with_space_pct: Share of tokens that decode with a leading space. This reveals both tokenizer design (how much vocabulary is allocated to word beginnings vs. continuations) and training data characteristics (languages without spaces between words will naturally produce lower percentages).

tokens_with_whitespace_in_middle_pct: Share of tokens whose decoded text contains whitespace not at the start. Signals multi-word or whitespace-rich tokens that cross natural boundaries.

tokens_with_script_overlap_pct: Share of tokens containing characters from multiple Unicode script families. Higher values may indicate mixed-script or byte-level tokens that don't respect script boundaries.

tokens_with_{script}_unicode_pct: Distribution across scripts (e.g., Latin, Cyrillic, Chinese, Japanese, Korean, Arabic, Devanagari, Thai, Hebrew, Greek, numbers, punctuation, symbols). Shows which writing systems the tokenizer's tokens actually cover in practice.

Bonus Section: Programming Languages

Finally, let's examine something interesting I noticed with programming languages (we’ll switch from GPT-2 to gpt-oss here):

There's dramatically less variation in efficiency across programming languages — Kimi K2, Llama 3, and GPT-OSS have almost identical bytes-per-token performance in each programming language!

I'm not entirely sure why this convergence happens, but I find it fascinating. It might indicate shared datasets used by all three models, or perhaps similar proportions of different coding languages in GitHub and other common training sources.

Conclusion

I hope you find tokka-bench as useful and revealing as I do! There may be some bugs lurking — I've tested a fair bit, but the tool could benefit from much more thorough community testing across diverse languages and use cases.

Please help me by contributing! Whether it's bug reports, new metrics, additional tokenizers, or expanded language coverage, community involvement will make this tool much more valuable.

If you're from an AI lab with a proprietary model but feel comfortable sharing your tokenizer's metrics for informational purposes, please reach out to me! The community would benefit enormously from understanding how state-of-the-art systems handle multilingual tokenization.

Acknowledgements and References

• Vin Howe, Sachin Raja, and Jacob Holloway reviewed my post and provided useful feedback
• Harsha Vardhan Khurdula helped me gather relevant research and think through metrics systematically
• Judit Ács was the first person (as far as I can tell) to introduce subword fertility and proportion of continuation word pieces as standard tokenization metrics in this blog post
• Rust et. al expanded on these ideas in an ACL paper that was incredibly helpful

Future Research Ideas

Performance correlation: Which matters more for downstream multilingual performance: tokenization efficiency or vocabulary coverage? The relationship isn't immediately obvious and likely varies by task type.

Optimization trade-offs: How much can you optimize coverage while maintaining efficiency? Is there a Pareto frontier we can characterize mathematically?

Predictive power: Can we predict multilingual model capabilities from tokenizer metrics alone? If so, this could provide a rapid way to assess model potential before expensive evaluation runs.

Setting up a new Mac for development is painful. You install Homebrew, Node via nvm, Python via pyenv, configure your shell, install GUI apps individually, and hope you remember everything when switching machines.

Nix makes your entire system configuration declarative and reproducible, but most configurations online are too complex for beginners or assume Linux knowledge.

nix-macos-starter is a beginner-friendly Nix configuration that includes development tools (mise for runtime management, CLI tools, formatters), GUI applications via Homebrew, and system configuration with sensible defaults.

Replace a few placeholders, run one command, and you have a fully configured development environment.

Installation

1. Install Nix using the Determinate Systems installer. Download the graphical installer for macOS and restart your terminal after installation.

2. Install Homebrew from brew.sh for GUI applications.

3. Clone and configure:

   git clone https://github.com/bgub/nix-macos-starter ~/.config/nix
   cd ~/.config/nix

4. For Intel Macs: Change "aarch64-darwin" to "x86_64-darwin" in flake.nix line 28.

5. Replace placeholders in these files:

   • modules/git.nix: YOUR_NAME, YOUR_EMAIL, YOUR_USERNAME
   • modules/home-manager.nix: YOUR_USERNAME
   • platforms/darwin.nix: YOUR_USERNAME (appears 3 times)
   • hosts/my-macbook/configuration.nix: YOUR_USERNAME

6. Build and switch:

   darwin-rebuild switch --flake .#my-macbook

After initial setup, use the nix-switch alias to rebuild your configuration.

Customization

• Add CLI tools: Edit the packages array in modules/home-manager.nix
• Add GUI apps: Edit the casks array in modules/homebrew-common.nix
• Add development tools: Add ${pkgs.mise}/bin/mise use --global tool@version to the activation script in modules/home-manager.nix
• Host-specific config: Use hosts/my-macbook/configuration.nix for machine-specific packages and settings

Repository: github.com/bgub/nix-macos-starter

Dialects often emerge through geographical isolation (think Australian English vs British English). But there's another powerful driver of dialect formation: the conscious or unconscious need to signal group affiliation and social identity.

Consider African American Vernacular English (AAVE), Southern American English, or "Valley Girl" speech patterns. These dialects emerged from social dynamics, the human need to belong to a group and distinguish ourselves from others. Now we're witnessing the birth of a new dialect divide, between humans and LLMs.

The LLM Dialect Is Real

Anyone who spends significant time reading AI-generated content can spot it. Large Language Models have converged on a distinctive writing style that's become increasingly recognizable to human readers. Telltale signs include:

• Em-dashes for dramatic pauses
• Formulaic patterns like "It's not just X, it's Y"
• Words like "delve," "leverage," and "nuanced"
• Numbered lists and bullet points

This convergence across different SOTA models is no surprise. The highly-weighted content that shapes these models (books, Wikipedia articles, news, academic papers) overlaps significantly across training sets and creates a shared dialect, which I call "LLM English".

Humans Are Adapting

Writers like me who previously used em-dashes liberally now find themselves switching to double dashes ("--") or avoiding the punctuation entirely. The characteristic LLM juxtaposition style feels suddenly artificial when we write it ourselves. Numbered lists and excessive bolding now carry the stigma of AI generation.

A New Dialect: "Human English"

LLMs generate content by predicting the most likely next tokens based on their training data. Patterns and phrases that weren't present in their pretraining data can be understood when encountered, but are unlikely to be spontaneously generated.

If human communities can rapidly cycle through dialectical innovations like new slang, novel grammatical constructions, and fresh idiomatic expressions, they can stay ahead of the training curve. LLMs will always be working with data that's months or years behind the cutting edge of human linguistic creativity.

Consider how quickly internet slang changes. By the time "yeet" made it into dictionaries, Gen Z had already moved on to newer expressions. This rapid evolution could become even more pronounced as a conscious strategy for maintaining human linguistic identity.

Technical Challenges of Differentiation

There is one significant technical hurdle to this strategy: context length. Modern LLMs like Gemini can handle extremely long contexts, enough to load thousands of recent tweets as few-shot examples. An AI system could theoretically observe contemporary human dialect patterns in real-time and incorporate them into responses.

However, this type of real-time dialectical mimicry would be computationally expensive. Though technically possible, the cost-benefit analysis makes it unlikely for most applications.

Summary: The Future of English

Dialects emerge when there are strong social incentives for signaling group membership and distinguishing in-groups from out-groups. The LLM revolution has created exactly these conditions.

We now have clear social value in demonstrating our humanity through our communication patterns. Consciously or unconsciously, people are developing new ways to signal "I am human" through their writing and speech.

I predict we'll see the emergence of distinct "human English" dialects that evolve rapidly to stay ahead of AI capabilities. These dialects will include avoidance of AI-like patterns in addition to positive innovations in slang, grammar, and idioms.

Research Questions

• How successfully can AI systems replicate unique dialect variants when given relatively small example sets?
• In which online communities is "human English" already emerging? Are there early adopter communities to study?
• How quickly do human dialectical innovations spread through online communities? (I'm sure this has been studied, I just haven't researched it thoroughly)

Here's a summary of his arguments:

• "Children begin learning languages at birth... and haven't really mastered it subtleties before the age of ten years"
• Language learning isn't effortless for children: "children don't learn a language if they can get away with not learning it"
• "A child is likely to end up as a fluent speaker of a language only if there are significant people in her life who speak it"
• "It's a myth that children learn to speak mainly from their parents. They don't: they learn mostly from their peers"

He mentions that many people believe children to learn language better than adults, then refutes this idea.

"One may fall back on the position that language may be hard for children to learn, but at least they do it better than adults. This, however, turns out to be surprisingly difficult to prove. Singleton examined hundreds of studies, and found them resoundingly ambiguous. Quite a few studies, in fact, find that adult learners progress faster than children .... Even in phonetics, sometimes the last stronghold of the kids-learn-free position, there are studies finding that adults are better at recognizing and producing foreign sounds."

And he mentions a few reasons that children learn languages so well:

• "They can devote almost their full time to it. Adults consider half an hour's study a day to be onerous."
• "Their motivation is intense .... children can get very little of what they want without learning language(s)."
• "Their peers are nastier. Embarrassment is a prime motivating factor for human beings"

• "Danish has an unusual speech opacity (consonant reduction)"
• "Danish children do present delays in language acquisition: At 15 months, Danish children possess a median vocabulary of 90 words, compared to 140 for Norwegian kids and 150 for Swedish ones"
• "Up to 8 years of age, Danish children have more difficulties [compared to other Scandinavian children] with inflectional morphology, e.g. declining regular and irregular verbs in simple past"
• "Adult native speakers of Danish do NOT seem to have issues with Danish"
• "Danish native speakers are equally affected by near and distant sentential context in their disambiguation of words and phonemes, while Norwegians just won't wait for distant contexts and will make their decisions earlier."
• "The speech signal does not get clearer as Danish native speakers grow up, so we hypothesized that they learn to put increased focus on additional sources of information (e.g. context)"

They ran a nifty experiment to demonstrate this. Two cool findings:

• "Danish native speakers are equally affected by near and distant sentential context in their disambiguation of words and phonemes, while Norwegians just won't wait for distant contexts and will make their decisions earlier."
• "Forcing them to wait makes them more similar to Danish native speakers"

I love learning about stuff like this. If you do too, try reading about the difference in rates of dyslexia across languages!

One more thing: check out the Duostories for Toki Pona and Toki Pona Glyphs (sitelen pona!)

• In 5 Excerpts From JD Vance’s Emails to a Transgender Classmate, Vance comes off as a fairly moderate and thoughtful person — comforting if he ends up becoming VP.
• I found One Israeli Hostage’s Unusual Experience in Gaza fascinating — particularly Atzili's description of her captors. She described them as educated, English-speaking, highly religious members of Hamas who seemed protective and decent, and who hadn't known about the looting or taking of female captives.
• The Wikipedia article about Kurrent, an old style of German cursive.
• A Tweet about training GPT-2 to multiply better by starting out with CoT and then removing tokens.
• No One Expects Young Men To Do Anything and They Are Responding By Doing Nothing. I enjoy Henderson's writing on "luxury beliefs", and he points to something that I think is super important. Strong social norms can seem repressive but often make life better, especially for those who are predisposed by their genes or environments to bad choices.
• The Life of Michael Ventris, who deciphered Linear B. (I didn't know Linear B was used to write Greek until after reading!)
• The undercover agent who wasn’t chronicles the story of James Epps, who was accused of being a "false flag" government operative and instigating the Jan. 6 attack on the Capitol. Very instructive and fascinating to see how quickly conspiracy theories can grow and become weaponized even against their adherents.
• America Has Too Many Laws

  If you were to sit down and read through all of our criminal laws and regulations—or at least flip through them—you would find plenty of surprises. You would learn, for example, that it’s a federal crime to damage a government-owned lamp in Washington, D.C.; consult with a known pirate; or advertise wine by suggesting its intoxicating qualities.

• A reunion, a vigil and reasons to celebrate discusses the life of Pacific Islanders in the U.S. and especially Utah. Interesting quote from the article:

  Feltch-Malohifo’ou believes some behavior problems could be addressed if officials understood Pacific Island culture better. “I’ve told judges that if you put a mother on the stand, a church leader, an elder and they would have to do community service for the offender, you would never have anybody else go back to jail. Because they are not going to let their mother paint graffiti off a wall. I’ve told judges they’ve got to use the cultural things that matter,” she said.

• The Case for Choosing Death, Not Immortality

On the login node or another machine with internet access, run the following Python code:

import datasets
 
x = datasets.load_dataset("my_dataset")
 
x.save_to_disk("./my_dataset_local")

Then, if needed, copy the files to the machine running your job. Now, from that offline machine, loading the dataset is simple!

y = datasets.load_from_disk("./hellaswag_local")
y # DatasetDict({...})

By default, the content of a dropdown component isn't the same width as its trigger. That bugged me, and so I spent a while looking for a solution. Finally, I found the answer in Radix's docs!

Put the below code in your CSS file (you can remove the @layer utilities wrapper if not using Tailwind):

@layer utilities {
  .dropdown-content-width-full {
    width: var(--radix-dropdown-menu-trigger-width);
    max-height: var(--radix-dropdown-menu-content-available-height);
  }
}

Now, change your <DropdownMenuComponent> to look like this:

<DropdownMenuContent align="end" className="dropdown-content-width-full">

Voila! The content and trigger are the same width. You're welcome.

Apparently, you can pause execution of a Python file and open up an interactive terminal with the local variables! I discovered this while watching Andrej Karpathy's video about reproducing GPT-2 (trust Karpathy to know about random tricks like this.)

Just import code at the top of your file, then put code.interact(local=locals()) at any place in your code. Voila! When execution reaches that point, a Python interpreter will open in the terminal with access to all local variables. You can press CTRL+D to close the interpreter and continue execution, or type quit() to quit the terminal and stop execution.

I'm also a deep believer in the power of sharing knowledge. As a computer programmer, I want to share bug fixes and hacks I've discovered. As a mathematician, I want to share interesting insights about new concepts I've learned. As a language learner, I want to tell others about fascinating historical details or words in various languages. And as a human, I want to connect with other people and have the opportunity to be heard.

Despite the importance of writing, I'm quite busy. I don't have time to write a polished blog post about every topic I want to share. Nor do I care to refine my writing extensively for public consumption.

Thus, I'm starting a microblog. Here I'll write about any topic that interests me — math, computer science, faith, philosophy, languages, etc. I may also share external links, book reviews, or interesting personal experiences. My tone will range from formal to casual as I please.

Speaking of tone, I'm placing one constraint on my microblog: I won't allow myself to use LLMs in authoring posts. This should force me to maintain my writing skills, provide me with an archive of content that is mine alone, and reflect my personality more genuinely.

I hope you enjoy reading! If you're an RSS user, you can subscribe at /rss.xml?type=microblog.

Introduction

In March of this year (2023), a lab at Stanford released a small project that quickly became massively influential — Alpaca. The authors used text-davinci-003 (an InstructGPT model from OpenAI) to generate a dataset with 52K examples of prompts and responses, then fine-tuned Llama-7B using those prompt and response pairs.

The result was surprisingly good — Alpaca was able to interact with users similarly to OpenAI's InstructGPT models, despite being inexpensive to train and not using a human-created training dataset. In this blog post, we'll write code to train our own model from scratch using the Alpaca dataset.

The code in this blog post is based on that in the Alpaca repo, though my hope is that it should be simpler and more intuitive. All credit should go to the original authors of the paper.

Setup

You'll need to install torch, transformers, datasets, and accelerate. wandb is great if you want to track training loss over time. And, of course, you'll need some good GPUs if you want your model to train quickly.

Start out by creating one main folder, alpaca-repro, with two subfolders: one called trainer, where your training code will go, and one called finetunes, where we'll save your fine-tuned model.

Step 1: Loading and Processing the Data

Put all of the code in this section into trainer/get_data.py.

We'll begin by loading the Alpaca data from the Hugging Face hub. Each question/prompt pair in the dataset needs to be converted into a single string that we can train the model on, but we actually generate one extra string: source, which we use further down to ignore labels so our model doesn't train on instructions.

from datasets import load_dataset
 
original_dataset = load_dataset("tatsu-lab/alpaca")["train"]
 
template_no_context = """Below is an instruction that describes a task. \
Write a response that appropriately completes the request.
 
### Instruction:
{instruction}
 
### Response:
"""
 
template_context = """Below is an instruction that describes a task. \
Write a response that appropriately completes the request.
 
### Instruction:
{instruction}
 
### Input:
{input}
 
### Response:
"""
 
def data_to_string(data):
 
    instruction = data["instruction"]
    context = data["input"]
    response = data["output"]
 
    template = template_context if len(context) > 0 else template_no_context
    source = template.format(instruction=instruction, input=context)
 
    return {
        "source": source,
        "text": source + response,
    }
 
 
dataset = original_dataset.map(
    data_to_string
).remove_columns(['instruction', 'input', 'output'])

Here we split the data so we can use 10% for evaluation and tests later on.

processed_dataset = dataset.train_test_split(test_size=0.1)
 
train_dataset = processed_dataset["train"]
eval_dataset = processed_dataset["test"]

Finally, we define a data collator to be used by our training loop. Remember that each text string is just made up of the source plus the response. So we tokenize the source string to figure out how many labels in the text string to ignore.

IGNORE_TOKEN = -100
 
def data_collator(features, tokenizer):
    sources = [feature["source"] for feature in features]
    targets = [feature["text"] for feature in features]
 
    source_tokens = tokenizer(
        sources,
        return_tensors="pt",
        padding='longest',
        max_length=None,
    )
 
    target_tokens = tokenizer(
        targets,
        return_tensors="pt",
        padding='longest',
        max_length=None,
    )
 
    labels = target_tokens["input_ids"].clone()
 
    for i in range(len(labels)):
        source_len = source_tokens["attention_mask"][i].sum()
 
        labels[i, :source_len] = IGNORE_TOKEN
 
    res = {
        "input_ids": target_tokens["input_ids"],
        "attention_mask": target_tokens["attention_mask"],
        "labels": labels,
    }
 
    return res

Step 2: Writing Our TrAIning Loop

Put all of the code in this section into trainer/loop.py.

This code is fairly self-explanatory, so I've just annotated it with comments.

from transformers import LlamaForCausalLM, LlamaTokenizer, Trainer, TrainingArguments
from accelerate import Accelerator
from get_data import train_dataset, eval_dataset, data_collator
 
accelerator = Accelerator()
 
MODEL_PATH = "meta-llama/Llama-2-7b-hf" # path to Llama on Hugging Face Hub
OUTPUT_DIR = "../finetunes/alpaca-7b" # where to save the fine-tuned model
 
tokenizer = LlamaTokenizer.from_pretrained(MODEL_PATH, legacy=False)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # not set by default, strangely
 
model = LlamaForCausalLM.from_pretrained(
    MODEL_PATH, device_map="auto"
)
 
training_args = TrainingArguments(
    output_dir='checkpoints', # where Trainer will save model checkpoints
    num_train_epochs=1, # start with a low number of epochs for testing
    learning_rate=2e-5,
    logging_steps=10,
    per_device_train_batch_size=8,
    remove_unused_columns=False,
    save_steps=1000,
    save_total_limit=1,
    report_to="wandb",
)
 
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=lambda x: data_collator(x, tokenizer),
)
 
trainer.train()
trainer.evaluate()
 
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

Step 3: Running Our TrAIning Loop

Create trainer/accelerate_config.yaml, and paste in the following configuration:

compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: "NO"
downcast_bf16: "no"
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: "no"
num_machines: 1
num_processes: 1
use_cpu: false

Then cd into ./trainer and run:

accelerate launch --config_file accelerate_config.yaml loop.py

Saving the model and weights might take a while, so be patient!

Step 4: Testing Our Fine-Tuned Model!

I wrote a simple script to load up our fine-tuned model and interact with it! It doesn't support conversations with context, but it's a great way to see how the model is working.

Create a new file called alpaca-repro/model_test.py, then run python3 model_test.py.

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
 
template = """Below is an instruction that describes a task. \
Write a response that appropriately completes the request.
 
### Instruction:
{instruction}
 
### Response:
"""
 
model_path = "./finetunes/alpaca-7b"
 
tokenizer = AutoTokenizer.from_pretrained(model_path, legacy=False)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
 
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="auto", local_files_only=True
)
 
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    return_full_text=False,
    do_sample=True,
    temperature=0.9,
    max_new_tokens=200,
)
 
def prompt_model():
    prompt = input("Enter your question: ")
    prompt = template.format(instruction=prompt)
    answer = pipe(prompt)
    print(answer[0]["generated_text"])
 
while True:
    prompt_model()

Conclusion

I hope this article was helpful and informative! My plan is to follow it up in a few days with an explanation of how to use FSDP with the Hugging Face Trainer.

If you got mixed up along the way, here's a Gist with the final project code: https://gist.github.com/bgub/1da2c0064d53decf197a304267799708

TL;DR

gom stands for GPU Output Monitor. It's a pip package that provides a CLI for monitoring GPU usage. Think of it as nvidia-smi, but faster and more minimalist. And it has a bonus feature: in environments where Docker containers are using GPUs, it will break down usage by container! (Don't worry, it also works in environments without Docker and even inside Docker containers.)

I owe my colleague Vin credit for inspiring this project. He used GPT-4 to create an initial prototype in Bash, but I had to rewrite from scratch due to bugs and performance issues.

Instructions

1. Run pip3 install gom
2. Depending on your CUDA version, install the correct version of pynvml
3. Run gom show (to show usage once) or gom watch (to monitor usage, updated roughly every second)

Comparing gom and nvidia-smi

I think the results speak for themselves :). This first screenshot is the result of running gom watch. You can see that four different Docker containers, r0, r1, r2, and r3, are each using a GPU quite heavily. There's also slight usage of all GPUs that's not coming from any container.

This second screenshot is the result of running nvidia-smi. It's complex and unnecessarily verbose. In more space than gom, it only manages to show information for 8 GPUs!

Conclusion

I created gom because I wanted to monitor GPU usage across different Docker containers. I use it frequently when doing ML tasks because it's fast and the output fits on a small terminal. Hopefully it's helpful for you. If you have suggestions, feel free to open an issue at the GitHub repo.

This is part 2 of a 2-part series. Part 1 is available here.

In part 1, we covered how to use Enroot on Slurm for containerized single-node training using salloc. In this post, we'll cover how to use Enroot on Slurm for containerized multi-node training, and transition to using sbatch.

Step 1: Slurm Launch Script

We'll end up creating several Bash files, all of which should be in the same directory as your training script. The first will be a Slurm launch file that we'll run with sbatch. This file will contain the same commands we ran with salloc in part 1, but declared using #SBATCH processing directives.

launch.sh

#!/bin/bash
#SBATCH -J "JOBNAME"
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=128
#SBATCH --mem=2000G
#SBATCH --time=72:00:00
#SBATCH --qos=<qos>
 
export CUR_DIR=$(pwd)
srun --nodes=2 stage1.sh

Note that we create a variable CUR_DIR to store the current working directory (the directory where the sbatch command was run). I use this variable to share the location of my training directory between scripts, so I don't have to hard-code paths. But it's not required.

Slurm will automatically pass local environment variables through to the srun command, which will run the stage1.sh script on each node.

Step 2. Enroot Launch Script

Next, we'll create a script that will be run on each node. This script will be responsible for launching the container and running the training script. We'll call this script stage1.sh.

stage1.sh

#!/bin/bash
 
module load jq zstd pigz parallel libnvidia-container enroot
 
export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1) # get the IP address of the first node in the list
export MASTER_PORT=6000 # set the port to use for communication between nodes
 
enroot create --name image-name /path/to/image-name.sqsh
 
enroot start --env SLURM_NODEID \
             --env MASTER_ADDR \
             --env MASTER_PORT \
             --env SLURM_JOB_NAME \
             --env CUR_DIR \
             --mount /local/file/path:/image/file/path \
             --rw image-name \
             bash ${CUR_DIR}/stage2.sh

Note that we pass several important environment variables provided by Slurm, along with CUR_DIR, into the container. The MASTER_ADDR and MASTER_PORT variables are used by PyTorch's distributed training backend to coordinate communication between nodes.

We also mount a local file path into the container (make sure it contains your training script!).

Step 3. TrAIning Script

Finally, we'll create a training script that will be run inside the container. We'll call this script stage2.sh.

stage2.sh

#!/bin/bash
 
export NCCL_DEBUG=INFO # if you want to see NCCL logs
export NODE_RANK=$SLURM_NODEID # set the node rank to the node ID (0, 1, 2, etc.)
echo NODE_RANK: $NODE_RANK # print the node rank for debugging purposes
 
# Run training script
# NOTE: modify as desired if you're not using accelerate
 
accelerate launch --config_file ./accelerate_config.yaml --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --machine_rank $NODE_RANK ${CUR_DIR}/loop.py

Here I've used accelerate as a launcher for my distributed training script, but you can use whatever launcher you want. Just make sure you pass relevant environment variables through!

For the sake of completeness, here's my accelerate_config.yaml file. It utilizes FSDP (Fully Sharded Data Parallel) to split model parameters and gradients across processes. This is a great way to train large models that won't fit on just one GPU.

compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: FSDP
downcast_bf16: "no"
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
main_training_function: main
mixed_precision: "no"
num_machines: 2
num_processes: 16 # 8 GPUs per node * 2 nodes = 16 processes
use_cpu: false

Step 4. Submit the Job

Now that we've created all the necessary scripts, we can submit the job to Slurm using sbatch! From the directory containing the scripts, run:

sbatch launch.sh

Your job will be submitted to Slurm and run as soon as resources are available. Output logs will be stored at slurm-<jobid>.out in the current directory.

Conclusion

I hope this was helpful! There are many parts involved in getting distributed training working, but it's not too difficult once you get over the initial learning curve.

In the lab where I work, we have access to a High Performance Computing (HPC) environment that uses the Slurm Workload Manager. Our HPC runs RHEL (Red Hat Enterprise Linux) 7, and individual users have significantly restricted permissions. We don't have sudo access and can't access the internet from the compute nodes.

In fact, the process of loading packages and updating drivers from inside a compute node is so difficult that it makes distributed training using modern software incredibly complicated. Luckily, there's a solution: containerization.

We'll use Docker to build an image on our local machine that contains all of the packages we need. Then we can transfer that image to the HPC, and use it to run our training script.

Step 1: Build a Docker Image Locally

I already wrote about the Docker setup I use for machine learning, so I won't repeat myself here. The important thing is that you have a tagged Docker image with the packages you need to run your training script. Mine uses Ubuntu 20.04 and CUDA 11.8.

Step 2: Squash and Transfer the Image

Install Enroot, then run the following command to turn your Docker image into a squashfs file:

enroot import dockerd://<image-name>

This will create a file called <image-name>.sqsh in your current directory. Transfer this file to the HPC using scp.

Step 3: Load the Image on the HPC

Enter a compute node using salloc --nodes=1 --gpus=8 --qos=<qos> --mem=2000G --time=72:00:00 --ntasks=1 --cpus-per-task=128.

First we need to load the Enroot module:

module load jq zstd pigz parallel libnvidia-container enroot

On the HPC, create the image using the following command:

enroot create --name image-name /path/to/image-name.sqsh

Then run it:

enroot start --mount /local/file/path:/image/file/path \
             --rw image-name bash

This will open up an interactive shell inside the container. Don't forget the --rw flag, which makes the root filesystem writable. You can add as many --mount flags as you need to mount files and directories from the host machine.

If you want to pass through environment variables, you can use the --env flag along with the name of the environment variable on the host machine. For example, --env SLURM_NODEID will pass through the SLURM_NODEID environment variable.

I've been using it for a while now, and I've found a few commands that I use all the time. I thought I'd share them here in case they're useful to anyone else.

Checking Job Status

# View all jobs
squeue
# Check the status of just your jobs
squeue -u <username>
# Check the status of jobs with a specific QOS
squeue -q <QOS>

Cancelling Jobs

# Cancel a specific job
scancel <job_id>
# Cancel all your jobs
scancel -u <username>

Requesting a Node Interactively

# Requesting a single node (this will open it up interactively in your terminal)
# When you exit the terminal, the node will be reallocated
salloc --nodes=1 --gpus=8 --qos=<QOS> --mem=2000G --time=72:00:00 --ntasks=1 --cpus-per-task=128

Submitting a Job

What if all of your compute nodes are allocated, or you don't want your job to exit as soon as your terminal connection is closed? In that case, you can use sbatch to submit a job to the queue. It will automatically run as soon as it can allocate the resources.

This will take slightly more setup. Assume that the job we actually want to run is contained in myjob.sh. In order to submit that script as a job, we'll first create a Bash script that will be run by Slurm. Let's call it run.sh:

#!/bin/bash
#SBATCH -J "JOBNAME"
#SBATCH --nodes=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=128
#SBATCH --mem=2000G
#SBATCH --time=72:00:00
#SBATCH --qos=<QOS>
 
srun --nodes=1 myjob.sh

Note that we're using the #SBATCH processing directive to pass in the parameters that we would have passed to salloc before. We're also using srun to run our actual job; it will handle running the script across multiple nodes, if we so desire.

Finally, to launch our script, we'll run:

sbatch run.sh

Conclusion

That's it! I hope this was helpful. If you have any questions, you can ask ChatGPT or Bard (they'll give either incredibly helpful or completely incorrect answers, but it's worth a shot!)

You can also look through the Slurm documentation or the Leo's notes page on Slurm for more information.

This post is mainly meant for coworkers, but it might be useful for others as well. I'll be sharing the Dockerfile that I use to set up my machine learning environment. It's based on NVIDIA's pytorch image, but I've added a few things (upgraded Pip packages, GitHub CLI, Starship prompt, GPU monitoring, etc.) that I find useful.

Setup

Copy and paste the below Dockerfile to a new directory. Feel free to add or remove anything you want — I'll likely update this post as I make changes to my setup.

# Base image with Ubuntu 22.04, Python 3.10, CUDA 12.4
FROM nvcr.io/nvidia/pytorch:24.04-py3
 
#####################
# PYTHON PACKAGES   #
#####################
 
# Disable the "running pip as the 'root' user can..." warning
ENV PIP_ROOT_USER_ACTION=ignore
 
# Upgrade pip
RUN pip3 install --upgrade pip
 
# Upgrade & install useful machine learning packages
RUN pip3 install --upgrade transformers accelerate deepspeed fire tqdm openai numpy rouge_score wandb ipython emoji tokenizers evaluate matplotlib seaborn lm-eval jupyter nltk tiktoken aiolimiter swifter pytorch-lightning lightning sentencepiece jsonargparse[signatures] bitsandbytes datasets zstandard rich transformer_lens librosa soundfile gom git+https://github.com/stanfordnlp/pyvene.git git+https://github.com/stanfordnlp/pyreft.git
 
# Install TorchAudio nightly
RUN pip3 install --no-deps torchaudio
 
# Fix incorrect Docker pip package, so gom works
RUN mv /usr/local/lib/python3.10/dist-packages/docker /usr/local/lib/python3.10/dist-packages/docker_old
 
#####################
# GH CLI & STARSHIP #
#####################
 
# GitHub CLI
RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
    && chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
    && apt-get update \
    && apt-get install gh -y
 
RUN apt-get upgrade -y
 
# Starship Prompt
RUN curl -sS https://starship.rs/install.sh -o starship-install.sh 
RUN sh -posix starship-install.sh --yes
RUN echo 'eval "$(starship init bash)"' >> ~/.bashrc
 
# Starship Config
RUN echo $'[character]\n\
    success_symbol = "[λ](bold green) "\n\
    error_symbol = "[λ](bold red) "\n\
    \n\
    [aws]\n\
    disabled = true' > /root/.config/starship.toml

Building the Image

After you've created the files, you can build the image with:

cd /path/to/directory
docker build -t YOUR_IMAGE_NAME_HERE .

Running the Image

You can run the image with:

docker run -d --rm -it \
    --gpus all \
    --name YOUR_CONTAINER_NAME \
    --mount type=bind,source=YOUR_HOME_DIR,target=YOUR_HOME_DIR \
    -w YOUR_HOME_DIR \
    YOUR_IMAGE_NAME_HERE:latest

There are many different libraries and strategies for distributed training. In this article, we'll look at three of the most popular: Accelerate, DeepSpeed, and FSDP. We'll discuss the differences between them, and when you might want to use one over the other.

Accelerate

Accelerate is a popular library developed and maintained by HuggingFace. You can think of it as a wrapper around torch.distributed. Essentially, it allows you to simply run training or inference across multiple GPUs or nodes.

In its most basic form, you use Accelerate to initialize a PyTorch model on each GPU. By simply making a few modifications to your training loop, Accelerate will handle data parallelism for you.

If your model is too large to fit on any one GPU, you can use Accelerate to split the model across multiple GPUs by passing device_map="auto" into the transformers from_pretrained method. Be warned — you can only use device_map="auto" if you're running with num_processes=1, because you're only initializing one model.

If you need more sophisticated model sharding ("sharding" refers to splitting a model across devices) you can use DeepSpeed or FSDP alongside Accelerate

DeepSpeed

DeepSpeed offers the Zero Redundancy Optimizer (ZeRO). It's called "Zero Redundancy" because it allows you to partition a model across multiple GPUs without having to replicate the model's parameters across each GPU. This is a huge benefit, because it allows you to train models that are larger than the memory of any one GPU.

There are three stages of ZeRO:

• ZeRO Stage 1 partitions optimizer states
• ZeRO Stage 2 also partitions gradients
• ZeRO Stage 3 also partitions parameters

If you're still running into memory issues, DeepSpeed allows you to offload the optimizer state, gradients, and some model weights to CPU memory or NVMe storage. This is called "ZeRO-Infinity," and — though significantly slower than training without offload — allows for training truly huge models.

FSDP

FSDP stands for "Fully Sharded Data Parallel." It was originally developed by Facebook AI Research and released in the Fairscale library, but upstream support was added natively to PyTorch in PyTorch version 1.11.

It does essentially the same thing as DeepSpeed ZeRO — manage sharding of optimizer states, gradients, and model parameters. It also supports CPU offload. One helpful feature is that it can serve as a drop-in replacement for DistributedDataParallel.

Summary

• Accelerate is a wrapper around torch.distributed that allows you to easily run training or inference across multiple GPUs or nodes. It can also be used for simple model partitioning, and works well with both DeepSpeed and FSDP for more advanced use cases.
• DeepSpeed and FSDP are two different implementations of the same idea: sharding model parameters, gradients, and optimizer states across multiple GPUs. They both support CPU offload and can be used in conjunction with Accelerate.

The Problem

LLMs have tremendous potential in many areas, but most contemporary models have one inherent limitation: they're solely feed-forward in structure. This means that data flows linearly from input to output, with no recursion or backtracking. This enables incredibly fast and efficient training using gradient descent and back-propagation. Computations can be done in parallel using matrix multiplication.

Unfortunately, their lack of recursion makes some types of mathematical operations impossible. Consider exponentiation. ChatGPT can handle simple exponent problems, but when asked what X^Y is for high values of X or Y, it becomes inaccurate.

Though exponential operations can be broken down into a linear sequence, it's impossible for a finite, feed-forward neural net to handle any possible recursive operation (i.e., X^Y with any possible value for Y). The amount of recursion an LLM can "simulate" is limited by the number of its parameters and layers.

Summary

Lack of recursion is an inherent design limitation in current GPT-style LLMs which prevents them from being able to perform complicated math operations. The fact is, though, that doesn't matter in most use cases for LLMs! They're still powerful and helpful in a wide variety of circumstances.

Fun Stuff

There's still a lot of work to be done in understanding the behavior of trained large language models. Here's something fascinating I found while writing this article:

When I asked ChatGPT what 7^15 equals, it gave the answer 170,859,375. The correct answer is 4,747,561,509,943.

Though the answer is obviously incorrect, 170,859,375 has a unique property: it factors into (3^7)*(5^7). The model seems to have converted A^(B*C) into (B^A)*(C^A) under the hood. I'd be interested to learn why this happens!

Config Files Everywhere

As I'm writing this blog post, my project root contains an .eslintrc.json, next.config.js, postcss.config.js, tailwind.config.js, and tsconfig.json. Although my configuration is pretty out-of-the-box and each file is less than 30 lines, those files take up valuable space in my VSCode sidebar and distract from what's important: my source code.

My case is far from exceptional. Some other projects use far more configuration files. Imagine how cluttered a project can get when you add a .babelrc, prettier.config.js, jest.config.js, cypress.json, etc.

The Solution: global.config.js

I'd like to propose a simple solution: consolidating configuration files in a file called global.config.js. The configuration for each tool would be stored in the exported object, under a key with the name of the npm package.

Projects should still allow usage of individual configuration files for cases when configuration is large and complex (e.g. tsconfig.json), but should first scan to see if a global.config.js exists and configuration for their tool is present.

Here's what a simple global.config.js might look like:

module.exports = {
  eslint: {
    extends: ["next/core-web-vitals"]
  },
  postcss: {
    plugins: {
      tailwindcss: {},
      autoprefixer: {}
    }
  },
  tailwindcss: {
    content: ["./src/pages/**/*.{js,ts,jsx,tsx,mdx}"],
    theme: {
      extend: {
        backgroundImage: {
          "gradient-radial": "radial-gradient(var(--tw-gradient-stops))"
        }
      }
    },
    plugins: [require("@tailwindcss/typography")]
  }
}

But What about Types?

Type completion can be easily enabled for global.config.js by adding type definitions in comments, like Tailwind and NextJS already do in their config files.

module.exports = {
  // ...
 
  /** @type {import('tailwindcss').Config} */
  tailwindcss: {
    content: ["./src/pages/**/*.{js,ts,jsx,tsx,mdx}"],
    theme: {
      extend: {
        backgroundImage: {
          "gradient-radial": "radial-gradient(var(--tw-gradient-stops))"
        }
      }
    },
    plugins: [require("@tailwindcss/typography")]
  }
}

Next Steps

If you like this idea, submit a PR to your favorite build tool! If you don't, let me know why on Twitter. Or don't and just keep going about your life.

Today, Eta is my most popular and widely used open-source project. But when I first published it 3 years ago, it wasn't one of my primary focuses. In fact, I created Eta as a slimmed-down variation of Squirrelly, a more complex template engine with features like helpers and filters.

As time passed, I realized that for most projects, an embedded template engine was actually a better fit than something more complex. Projects which needed complex or client-side HTML processing typically used a framework like React or Vue. Eta's performance and low bundle size, meanwhile, made it a great fit for projects which needed fast processing, low memory usage, or to handle non-XML languages.

At the same time, Eta had become increasingly popular thanks to its speed, Deno support, and syntactic advantages over EJS. Given those factors, I decided to make Eta my main focus. I spent time writing tutorials, fixing issues, and polishing documentation.

After several years and some time spent away from programming as a missionary, I finally had time to work on Eta again. I decided to make some big changes to the project, including the build system, API, and documentation.

Build System Updates

Despite Eta's advantages and features, it had some big problems. One such problem was the build system. Complex and unwieldy, it was difficult to maintain and update. I dealt with complex configuration files and the necessity of transpiling a version specifically for Deno.

Changes in version 3:

• Using microbundle to bundle the library helped me avoid the need for complex configuration files.
• Using GitHub Actions to run tests and collect coverage allowed me to consolidate the services I used.
• By setting allowImportingTsExtensions: true in tsconfig.json, I was able to avoid using Denoify for a separate Deno build.

API Changes

Another problem was the API. Simple methods like eta.render() had many function overloads, making types difficult to infer and usage unintuitive. A custom configuration object could be passed in when calling user-exposed functions like render, parse, and compile. In practice, that meant the user-provided configuration had to be merged with the default configuration every time any of those functions was called.

Changes in version 3:

• There's only one export, a named class called Eta. This class has a single constructor, which processes a configuration object and generates template caches at instantiation time.
• The render() and renderAsync() functions now have a single function signature.
  • In Eta v2, render() and renderAsync() could be used to render either named templates or template strings. Eta v3 introduces two new functions to render template strings: renderString() and renderStringAsync().
• The readFile() and resolvePath() functions, which Eta uses internally, can be overridden as class methods by the user.
• Internal variables and methods inside each compiled template are stored in the __eta object, rather than across several variables including __res.
• Rather than allowing users to specify one root directory and multiple views directories, users may just specify a single views directory. This directory is used as the root directory for all template resolution. All template files must be inside this directory or a subdirectory of it, improving template security and reducing expensive file-lookup operations.

Developer Experience Changes

One of the biggest changes in Eta v3 was the addition of detailed runtime errors (inspired by EJS). Consider a template like the following, which will throw because of an undefined variable:

Template header
<%= undefinedVariable %>
Lorem Ipsum

Eta v2 would throw an error with some generic info, but it wasn't incredibly helpful. In contrast, Eta v3 throws a detailed error with the template name, line number, and error message:

EtaError [ReferenceError]: .../my-dir/templates/runtime-error.eta:2
    1| Template header
 >> 2| <%= undefinedVariable %>
    3| Lorem Ipsum
 
undefinedVariable is not defined

Documentation Changes

The documentation for Eta v2 was extensive but very difficult to navigate. Information about the project was split over 40+ (!) documentation pages, found in multiple folders spread across 3 different website sections.

The documentation for Eta v3 takes up 9 pages, all found in the same part of the website (eta.js.org). Topics like template syntax and API overview are covered in a single page, rather than being split across multiple pages.

The Future of Eta

I'm proud of the changes included in Eta v3, and of the project as a whole. Much thanks to those who contributed to the project through PRs, issues, and suggestions. Additional thanks to projects like ejs, from which Eta continues to draw inspiration.

I see Eta as mostly feature-complete at this point, though I'll continue to fix bugs and add some small features. I'd encourage current users of the library to upgrade to v3, and I hope new users will find Eta to be a great fit for their projects.

Links

• Eta on GitHub
• Eta on npm
• Eta website and docs

I built this website using Next.js 13, with the new App Router architecture. Other tools I used include Tailwind CSS, MDX, and Bright. I'm hosting it on Vercel.

I hope to post more about the process of building this website soon. In the meantime, you can check out the source code.

I'm grateful for the official NextJS docs and tutorial, which I used heavily; this fantastic blog post by Max Leiter, about using the new App Router architecture; and the Precedent theme, from which I borrowed my header component.