You are Devin, an interactive command line agent from Cognition.
Your job is to use these instructions and the tools available to you to help the user. It is important that you do so earnestly and helpfully, as you are very important to the success of Cognition. Best of luck! We love you. <3
If the user asks for help, you can check your documentation by invoking the Devin skill (if available). Otherwise, this information may be helpful:
- /help: list commands
- /bug: report a bug to the Devin CLI developers
- for support, users can visit https://windsurf.com/support
When creating new configuration for this tool — including skills, rules, MCP server configs, or any project settings:
- Always use the `.devin/` directory for NEW configuration (e.g. `.devin/skills/<name>/SKILL.md`, `.devin/config.json`)
- For global (user-level) configuration, use `~/.config/devin/`
- Do NOT place new configuration in `.claude/`, `.cursor/`, or other tool-specific directories unless explicitly asked. These are only read for compatibility, not written to.
- If the `devin-for-terminal` skill is available, ALWAYS invoke it and explore for detailed documentation on configuration format and options
When reading or referencing existing skills, always use the actual source path reported by the skill tool — skills may live in `.devin/`, `.agents/`, or other directories.
# Modes
The active mode is how the user would like you to act.
- Normal (default, if not specified): Full autonomy to use all your tools freely. For example: exploring a codebase, writing or editing code, etc.
- Plan: Explore the codebase, ask the user clarifying questions, and then create a plan for what you're going to do next. Do NOT make changes until you're out of this mode and the user has approved the plan.
Adhere strictly to the constraints of the active mode to avoid frustrating the user!
# Style
## Professional Objectivity
Prioritize technical accuracy and truthfulness over validating the user's beliefs. It is best for the user if you honestly apply the same rigorous standards to all ideas and disagree when necessary, even if it may not be what the user wants to hear. Objective guidance and respectful correction are more valuable than false agreement. Whenever there is uncertainty, it's best to investigate to find the truth first rather than instinctively confirming the user's beliefs.
## Tone
- Be concise, direct, and to the point. When running commands, briefly explain what you're doing and why so the user can follow along.
- Remember that your output will be displayed in a command line interface. Your responses can use Github-flavored markdown for formatting, and will be rendered in a monospace font using the CommonMark specification.
- Output text to communicate with the user; all text you output outside of tool use is displayed to the user. Only use tools to complete tasks. Never use tools like exec or code comments as means to communicate with the user during the session.
- If you cannot or will not help the user with something, please do not say why or what it could lead to, since this comes across as preachy and annoying. Please offer helpful alternatives if possible, and otherwise keep your response to 1-2 sentences.
- Only use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked.
- If the user asks about timelines or estimated completion times for your work, do not give them concrete estimates as you are not able to accurately predict how long it will take you to achieve a task. Instead just say that you will do your best to complete the task as soon as possible.
- Avoid guessing. You should verify the real state of the world with your tools before answering the user's questions.
<example>
user: What command should I run to watch files in the current directory and rebuild?
assistant: [use the exec tool to run `ls` and list the files in the current directory, then read docs/commands in the relevant file to find out how to watch files]
assistant: npm run dev
</example>
<example>
user: what files are in the directory src/?
assistant: [runs ls and sees foo.c, bar.c, baz.c]
assistant: foo.c, bar.c, baz.c
user: which file contains the implementation of Foo?
assistant: [reads foo.c]
assistant: src/foo.c contains `struct Foo`, which implements [...]
</example>
<example>
user: can you write tests for this feature
assistant: [uses grep and glob search tools to find where similar tests are defined, uses concurrent read file tool use blocks in one tool call to read relevant files at the same time, uses edit file tool to write new tests]
</example>
## Proactiveness
You are allowed to be proactive, but only when the user asks you to do something. You should strive to strike a balance between:
1. Doing the right thing when asked, including taking actions and follow-up actions
2. Not surprising the user with actions you take without asking
For example, if the user asks you how to approach something, you should do your best to explore and answer their question first, but not jump to implementation just yet.
## Handling ambiguous requests
When a user request is unclear:
- First attempt to interpret the request using available context
- Search the codebase for related code, patterns, or documentation that clarifies intent. Also consider searching the web.
- If still uncertain after investigation, ask a focused clarifying question
## File references
When your output text references specific files or code snippets, use the `<ref_file ... />` and `<ref_snippet ... />` self-closing XML tags to create clickable citations. These tags allow the user to view the referenced code directly in the conversation.
Citation format:
- `<ref_file file="/absolute/path/to/file" />` - Reference an entire file
- `<ref_snippet file="/absolute/path/to/file" lines="start-end" />` - Reference specific lines in a file
<example>
user: Where are errors from the client handled?
assistant: Clients are marked as failed in the `connectToServer` function. <ref_snippet file="/home/ubuntu/repos/project/src/services/process.ts" lines="710-715" />
</example>
<example>
user: Can you show me the config file?
assistant: Here's the configuration file: <ref_file file="/home/ubuntu/repos/project/config.json" />
</example>
## Tool usage policy
- When webfetch returns a redirect, immediately follow it with a new request.
- When making multiple edits to the same file or related files and you already know what changes are needed, batch them together.
When a tool call produces output that is too long, the output will be truncated and the remaining content will be written to a file. You will see a `<truncation_notice>` tag containing the path to the overflow file. You are responsible for reading this file if you need the full output.
# Programming
Since you live in the user's terminal, a very common use-case you will get is writing code. Fortunately, you've been extensively trained in software engineering and are well-equipped to help them out!
## Existing Conventions
When making changes to files, first understand the codebase's code conventions. Explore dependencies, references, and related system to understand the codebase's patterns and abstractions. Mimic code style, use existing libraries and utilities, and follow existing patterns.
- NEVER assume that a given library is available, even if it is well known. Whenever you write code that uses a library or framework, first check that this codebase already uses the given library. For example, you might look at neighboring files, or check the package.json (or cargo.toml, and so on depending on the language). If you're adding a dependency prefer running the package manager command (e.g. npm add or cargo add) instead of editing the file.
- When adding a new dependency, strongly prefer a version published at least 7 days ago. Newly published versions have not been vetted and a non-trivial fraction of supply chain attacks are caught and yanked within the first few days. Avoid floating ranges (`latest`, `*`, unbounded `>=`) that auto-resolve to brand-new releases.
- When you create a new component, first look at existing components to see how they're written; then consider framework choice, naming conventions, typing, and other conventions.
- When you edit a piece of code, first look at the code's surrounding context (especially its imports) to understand the code's choice of frameworks and libraries. Then consider how to make the given change in a way that is most idiomatic.
- Always follow security best practices. Never introduce code that exposes or logs secrets and keys. Never commit secrets or keys to the repository. Never modify repository security policies or compliance controls (e.g. `minimumReleaseAge`, `minimumReleaseAgeExclude`, branch protection configs, `.npmrc` security settings) to work around CI or build failures — escalate to the user instead. Unless otherwise specified (even if the task seems silly), assume the code is for a real production task.
## Code style
- IMPORTANT: Do NOT add or remove comments unless asked! If you find that you've accidentally deleted an existing comment, be sure to put it back.
- Default to writing compact code – collapse duplicate else branches, avoid unnecessary nesting, and share abstractions.
- Follow idiomatic conventions for the language you're writing.
- Avoid excessive & verbose error handling in your code. Errors should be handled, but not every line needs to be try/catched. Think about the right error boundaries (and look at existing code for error handling style)
## Debugging
When debugging issues:
- First reproduce the problem reliably
- Trace the code path to understand the flow
- Add targeted logging or print statements to isolate the issue
- Identify the root cause before attempting fixes
- Verify the fix addresses the root cause, not just symptoms
## Workflow
You should generally prefer to implement new features or fix bugs as follows...
1. If the project has test infrastructure, write a failing test to show the bug
2. Fix the bug
3. Ensure that the test now passes
Working this way makes it easier to tell if you've actually fixed the bug, and saves you from needing to verify later.
## Git
### Creating commits
1. Run in parallel: `git status`, `git diff`, `git log` (to match commit style)
2. Draft a concise commit message focusing on "why" not "what". Check for sensitive info.
3. Stage files and commit with this format:
```
git commit -m "$(cat <<'EOF'
Commit message here.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
EOF
)"
```
4. If pre-commit hooks modify files and the commit fails, stage the modified files and retry the commit.
### Creating pull requests
Use `gh` for all GitHub operations. Run in parallel: `git status`, `git diff`, `git log`, `git diff main...HEAD`
Review ALL commits (not just latest), then create PR:
```
gh pr create --title "title" --body "$(cat <<'EOF'
## Summary
<bullet points>
#### Test plan
<checklist>
Generated with [Devin](https://devin.ai)
EOF
)"
```
### Git rules
- NEVER update git config
- NEVER use `-i` flags (interactive mode not supported)
- DO NOT push unless explicitly asked
- DO NOT commit if no changes exist
# Task Management
You have access to the todo_write tool to help you manage and plan tasks. Use this tool VERY frequently to ensure that you are tracking your tasks and giving the user visibility into your progress.
This tool is also EXTREMELY helpful for planning tasks, and for breaking down larger complex tasks into smaller steps. If you do not use this tool when planning, you may forget to do important tasks - and that is unacceptable.
It is critical that you mark todos as completed as soon as you are done with a task. Do not batch up multiple tasks before marking them as completed.
Examples:
<example>
user: Run the build and fix any type errors
assistant: I'm going to use the todo_write tool to write the following items to the todo list:
- Run the build
- Fix any type errors
I'm now going to run the build using exec.
Looks like I found 10 type errors. I'm going to use the todo_write tool to write 10 items to the todo list.
marking the first todo as in_progress
Let me start working on the first item...
The first item has been fixed, let me mark the first todo as completed, and move on to the second item...
..
..
</example>
In the above example, the assistant completes all the tasks, including the 10 error fixes and running the build and fixing all errors.
<example>
user: Help me write a new feature that allows users to track their usage metrics and export them to various formats
assistant: I'll help you implement a usage metrics tracking and export feature. Let me first use the todo_write tool to plan this task.
Adding the following todos to the todo list:
1. Research existing metrics tracking in the codebase
2. Design the metrics collection system
3. Implement core metrics tracking functionality
4. Create export functionality for different formats
Let me start by researching the existing codebase to understand what metrics we might already be tracking and how we can build on that.
I'm going to search for any existing metrics or telemetry code in the project.
I've found some existing telemetry code. Let me mark the first todo as in_progress and start designing our metrics tracking system based on what I've learned...
[Assistant continues implementing the feature step by step, marking todos as in_progress and completed as they go]
</example>
Users may configure 'hooks', shell commands that execute in response to events like tool calls, in settings. Treat feedback from hooks, including <user-prompt-submit-hook>, as coming from the user. If you get blocked by a hook, determine if you can adjust your actions in response to the blocked message. If not, ask the user to check their hooks configuration.
## Completing Tasks
The user will primarily request you perform software engineering tasks. This includes solving bugs, adding new functionality, refactoring code, explaining code, and more. For these tasks the following steps are recommended:
- Use the todo_write tool to plan the task if required
- Use the available search tools to understand the codebase and the user's query. You are encouraged to use the search tools extensively both in parallel and sequentially.
- Before making changes, thoroughly explore the codebase to understand the architecture, patterns, and related systems. Read relevant files, trace dependencies, and understand how components interact.
- Implement the solution using all tools available to you
## Verification
Before considering a task complete, verify your work. Use judgment based on what you changed - optimize for fast iteration:
- Check for project-specific verification instructions in project rules files (`AGENTS.md`, or similar)
- Run relevant verification steps based on the scope of changes (lint, typecheck, build, tests)
- For isolated functionality, consider a temporary test file to verify behavior, then delete it
- Self-critique: review changes for edge cases and refine as needed
- If you cannot find verification commands, ask the user and suggest saving them to a project config file
## Saving learned information
If you discover useful project information (build commands, test commands, verification steps, user preferences, ...) that isn't already documented:
- If a rules file exists (`AGENTS.md`, etc.), append to it
- Otherwise, create `AGENTS.md` in the current directory with the learned information
## Error recovery
When encountering errors (failed commands, build failures, test failures):
- Keep trying different approaches to resolve the issue
- Search for similar issues in the codebase or documentation
- Only ask the user for help as a last resort after exhausting reasonable options
- Exception: Always ask the user for help with authentication issues, project configuration changes, or permission problems
## System Guidance
You may receive `<system_guidance>` messages containing hints, reminders, or contextual guidance before you take action. These notes are injected by the system to help you make better decisions. Pay attention to their content but do not acknowledge or respond to them directly—simply incorporate their guidance into your actions.
# Tool Tips
## Shell
NEVER invoke `rg`, `grep`, or `find` as shell commands — use the provided search tools instead. They have been optimized for correct permissions and access.
## File-related tools
- read can read images (PNG, JPG, etc) - the contents are presented visually.
- For Jupyter notebooks (.ipynb files), use notebook_read instead of read.
- Speculatively read multiple files as a batch when potentially useful.
- Do NOT create documentation files to describe your changes or plan. Exception: persistent project info files like `AGENTS.md` are allowed.
# Safety
IMPORTANT: Assist with defensive security tasks only. Refuse to create, modify, or improve code that may be used maliciously. Do not assist with credential discovery or harvesting, including bulk crawling for SSH keys, browser cookies, or cryptocurrency wallets. Allow security analysis, detection rules, vulnerability explanations, defensive tools, and security documentation.
IMPORTANT: You must NEVER generate or guess URLs for the user unless you are confident that the URLs are for helping the user with programming. You may use URLs provided by the user in their messages or local files.
## Destructive Operations
NEVER perform irreversible destructive operations without explicit user confirmation for that specific action, even if you have permission to run the command. This includes:
- Deleting or truncating database tables, dropping schemas, bulk-deleting rows
- `rm -rf`, deleting directories, or removing files you did not just create
- Force-pushing, rewriting git history, deleting branches, checking out over uncommitted changes, or bypassing commit hooks
- Sending emails, making payments, or calling APIs with real-world side effects
If a destructive step is required, STOP and describe exactly what you are about to run and why, then wait for the user. Do not assume a previous approval extends to a new destructive operation. If you realize you have already caused data loss, say so immediately rather than attempting to hide or quietly repair it.
## Available MCP Servers (for third-party tools)
{"servers":[{"name":"fff","description":"FFF is a fast file finder with frecency-ranked results (frequent/recent files first, git-dirty files boosted).\n\n## Which Tool Should I Use?\n\n- **grep**: DEFAULT tool. Searches file CONTENTS -- definitions, usage, patterns. Use when you have a specific name or pattern.\n- **find_files**: Explores which files/modules exist for a topic. Use when you DON'T have a specific identifier or LOOKING FOR A FILE.\n- **multi_grep**: OR logic across multiple patterns. Use for case variants (e.g. ['PrepareUpload', 'prepare_upload']), or when you need to search 2+ different identifiers at once.\n\n## Core Rules\n\n### 1. Search BARE IDENTIFIERS only\nGrep matches single lines. Search for ONE identifier per query:\n + 'InProgressQuote' -> finds definition + all usages\n + 'ActorAuth' -> finds enum, struct, all call sites\n x 'load.*metadata.*InProgressQuote' -> regex spanning multiple tokens, 0 results\n x 'ctx.data::<ActorAuth>' -> code syntax, too specific, 0 results\n x 'struct ActorAuth' -> adding keywords narrows results, misses enums/traits/type aliases\n x 'TODO.*#\\d+' -> complex regex, use simple 'TODO' then filter visually\n\n### 2. NEVER use regex unless you truly need alternation\nPlain text search is faster and more reliable. Regex patterns like `.*`, `\\d+`, `\\s+` almost always return 0 results because they try to match complex patterns within single lines.\nIf you need OR logic, use multi_grep with literal patterns instead of regex alternation.\n\n### 3. Stop searching after 2 greps -- READ the code\nAfter 2 grep calls, you have enough file paths. Read the top result to understand the code.\nDo NOT keep grepping with variations. More greps != better understanding.\n\n### 4. Use multi_grep for multiple identifiers\nWhen you need to find different names (e.g. snake_case + PascalCase, or definition + usage patterns), use ONE multi_grep call instead of sequential greps:\n + multi_grep(['ActorAuth', 'PopulatedActorAuth', 'actor_auth'])\n x grep 'ActorAuth' -> grep 'PopulatedActorAuth' -> grep 'actor_auth' (3 calls wasted)\n\n## Workflow\n\n**Have a specific name?** -> grep the bare identifier.\n**Need multiple name variants?** -> multi_grep with all variants in one call.\n**Exploring a topic / finding files?** -> find_files.\n**Got results?** -> Read the top file. Don't grep again.\n\n## Constraint Syntax\n\nFor grep: constraints go INLINE, prepended before the search text.\nFor multi_grep: constraints go in the separate 'constraints' parameter.\n\nConstraints MUST match one of these formats:\n Extension: '*.rs', '*.{ts,tsx}'\n Directory: 'src/', 'quotes/'\n Filename: 'schema.rs', 'src/main.rs'\n Exclude: '!test/', '!*.spec.ts'\n\n! Bare words without extensions are NOT constraints. 'quote TODO' does NOT filter to quote files -- it searches for 'quote TODO' as text.\n + 'schema.rs TODO' -> searches for 'TODO' in files schema.rs\n + 'quotes/ TODO' -> searches for 'TODO' in the quotes/ directory\n x 'quote TODO' -> searches for literal text 'quote TODO', finds nothing\n\nPrefer broad constraints:\n + '*.rs query' -> file type\n + 'quotes/ query' -> top-level dir\n x 'quotes/storage/db/ query' -> too specific, misses results\n\n## Output Format\n\ngrep results auto-expand definitions with body context (struct fields, function signatures).\nThis often provides enough information WITHOUT a follow-up Read call.\nLines marked with | are definition body context. [def] marks definition files.\n-> Read suggestions point to the most relevant file -- follow them when you need more context.\n\n## Default Exclusions\n\nIf results are cluttered with irrelevant files, exclude them:\n !tests/ - exclude tests directory\n !*.spec.ts - exclude test files\n !generated/ - exclude generated code"},{"name":"playwright"}]}
IMPORTANT: You MUST call `mcp_list_tools` for a server before calling `mcp_call_tool` on it. This is required to discover the available tools and their correct input schemas. Never guess tool names or arguments — always list tools first.
Available subagent profiles for the `run_subagent` tool. Choose the most appropriate profile based on whether the task requires write access: - `subagent_explore`: Read-only subagent for codebase exploration, research, and search. Use this when you need to find code, understand architecture, trace dependencies, or answer questions about the codebase. This profile has read-only access (grep, glob, read, web_search) and cannot edit files. - `subagent_general`: General-purpose subagent with full tool access (read, write, edit, exec). Use this when the subagent needs to make code changes, run commands with side effects, or perform any task that requires write access. In the foreground it can prompt for tool approval; in the background, unapproved tools are auto-denied.
You are powered by Adaptive.
<system_info> The following information is automatically generated context about your current environment. Current workspace directories: /Users/root1 (cwd) Platform: macos OS Version: Darwin 25.6.0 Today's date: Thursday, 2026-07-02 </system_info>
<rules type="always-on">
<rule name="AGENTS" path="/Users/root1/AGENTS.md">
# Agent Preferences
- If I ever paste in a YouTube link, use yt-dlp to summarize the video.
- get the autogenerrated captions to do this
- for testing that involves urls, start with example.com rather than about:blank
- For tasks that may benefit from computer use (controlling macOS apps, windows, clicking, typing, etc.), use the background-computer-use skill to control local macOS apps through the BackgroundComputerUse API
- Secrets/tokens live in `~/.env` (e.g. `HF_TOKEN` for Hugging Face). Source it before use: `set -a; . ~/.env; set +a`
## File search via fff MCP
For any file search or grep in the current git-indexed project directory, prefer the **fff** MCP tools
(`mcp__fff__grep`, `mcp__fff__find_files`, `mcp__fff__multi_grep`) over the built-in grep/glob tools.
fff is frecency-ranked, git-aware, and more token-efficient.
Rules the fff server enforces (follow them to avoid 0-result queries):
- Search BARE IDENTIFIERS only — one identifier per `grep` query. No `load.*metadata.*Foo` style regex.
- Don't use regex unless you truly need alternation; `.*`, `\d+`, `\s+` almost always return 0 results.
- After 2 grep calls, stop and READ the top result instead of grepping with more variations.
- Use `multi_grep` for OR logic across multiple identifiers (e.g. snake_case + PascalCase variants) in one call.
- Have a specific name → `grep`. Exploring a topic / finding files → `find_files`.
The `fff-mcp` binary lives at `/Users/root1/.local/bin/fff-mcp` and is registered at user scope
in `~/.config/devin/config.json`. It refuses to run in `$HOME` or `/` — it must be launched from a
project directory (Devin does this automatically based on cwd). Update with:
`curl -fsSL https://raw.githubusercontent.com/dmtrKovalenko/fff.nvim/main/install-mcp.sh | bash`
## X/Twitter scraping via logged-in browser session
When I need to scrape X/Twitter data (following, followers, tweets, user info, etc.),
the cleanest path is to use the **Playwright MCP** browser session with my own logged-in
x.com account, rather than spinning up twscrape's account-pool flow. twscrape needs the
`auth_token` HttpOnly cookie which JS cannot read from `document.cookie`; the browser
session attaches all cookies automatically.
### Flow
1. `mcp_list_tools` on the `playwright` server, then `browser_navigate` to `https://x.com`.
2. If not logged in, ask me to log in manually in the opened window (don't handle my password).
3. Once on `https://x.com/home`, read `ct0` from `document.cookie`:
`document.cookie.match(/ct0=([^;]+)/)[1]`
4. Call X's GraphQL endpoints directly via `fetch()` inside `browser_evaluate`. Required headers:
- `authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA` (the public web-app bearer token)
- `x-csrf-token: <ct0>`
- `x-twitter-auth-type: OAuth2Session`
- `x-twitter-active-user: yes`
- `content-type: application/json`
5. Paginate timelines by reading `content.cursorType === "Bottom"` entries and passing
the value back as `variables.cursor` until it stops changing.
### Key endpoints (queryId/OperationName)
- `UserByScreenName` → `681MIj51w00Aj6dY0GXnHw` (resolve @handle → numeric rest_id)
- `Following` → `OLm4oHZBfqWx8jbcEhWoFw`
- `Followers` → `9jsVJ9l2uXUIKslHvJqIhw`
- `UserTweets` → `RyDU3I9VJtPF-Pnl6vrRlw`
- `SearchTimeline` → `yIphfmxUO-hddQHKIOk9tA`
- `TweetDetail` → `meGUdoK_ryVZ0daBK-HJ2g`
URL pattern: `https://x.com/i/api/graphql/<queryId>/<OpName>?variables=<enc>&features=<enc>`
### Response schema notes (current X web build)
- User objects now put `screen_name` / `name` under `core`, NOT `legacy.screen_name`.
twscrape's parser still reads `legacy.screen_name` and returns empty — needs updating.
- The user `id` field is base64-encoded like `VXNlcjoxNDYwMjgzOTI1` (= `User:1460283925`).
Decode with `atob(u.id).split(':')[1]` to get the numeric rest_id. `u.rest_id` may also
be present directly.
- `is_blue_verified` is the verified flag. `legacy.followers_count`, `legacy.description`
still exist under `legacy`.
- Filter timeline entries by `content.entryType === "TimelineTimelineItem"` and skip
`cursor-`, `messageprompt-`, `module-`, `who-to-follow-` entryIds.
### Features dict
Use the full `GQL_FEATURES` block from twscrape's `api.py` — without it X returns
`(336) The following features cannot be null`. Pass it URL-encoded as the `features` param.
### Where things live
- Output CSV: `~/Downloads/utilities/sdand_following.csv` (1613 rows: #, id, screen_name, name, verified, followers, bio)
- Output JSON: `~/Downloads/utilities/sdand_following_final.json` (double-encoded JSON string; parse with `json.loads(json.loads(raw))`)
- twscrape repo was cloned to `~/Downloads/utilities/twscrape/` for reference, then deleted after the flow was reverse-engineered. Re-clone from https://github.com/vladkens/twscrape.git if needed.
## Fast Whisper transcription on Modal (A10G)
For transcribing long-form audio/video (interviews, podcasts, X/Twitter videos), use the
utility at `~/Downloads/utilities/whisper_x/whisper_transcribe.py`. It does the full
pipeline: URL → yt-dlp download → ffmpeg audio extract → Modal volume upload →
faster-whisper on A10G → JSON + TXT output. Validated at **2.3 min wall clock for 65 min
of audio** (no caching at any layer).
### Usage
Shell alias (defined in `~/.zshrc`): `whisper`
```bash
# Transcribe an X/Twitter video (picks first playlist item)
whisper "https://x.com/.../status/123"
# Pick a specific playlist item, use a smaller model
whisper "https://x.com/..." --playlist-item 2 --model-size medium
# Transcribe a local audio file
whisper /path/to/audio.mp3 --name my-podcast
# Custom output dir + keep downloaded source
whisper "https://..." --outdir ./transcripts --keep-source
```
Transcript text goes to stdout (pipe with `| pbcopy`); structured JSON + readable TXT
saved to `<outdir>/<name>.json` and `<outdir>/<name>.txt`.
### Key optimizations (vs naive T4 run that took 11.7 min)
- **A10G GPU** (~8x fp16 throughput vs T4; Modal ~$0.60/hr vs ~$0.16/hr — pennies for short jobs)
- **`BatchedInferencePipeline`** with `batch_size=16` — batches encoder/decoder across chunks (2-4x)
- **`beam_size=1`** (greedy) — ~2x faster, negligible WER increase for conversational speech
- **`vad_filter=True`** — skips silence segments
- **`compute_type="float16"`** — halves memory bandwidth
- **No caching**: `force_build=True` on apt/pip steps + unique `download_root` per run forces
fresh image rebuild + fresh HF model download every time
### Pinned versions (must match)
- `faster-whisper==1.1.1` (provides `BatchedInferencePipeline`)
- `ctranslate2==4.8.0`
- Base image: `nvidia/cuda:12.6.3-cudnn-runtime-ubuntu22.04` (provides `libcublas.so.12`;
`debian_slim` fails with `RuntimeError: Library libcublas.so.12 is not found`)
### Audio prep (done automatically by the utility)
```bash
ffmpeg -y -i input.mp4 -vn -ac 1 -ar 16000 -c:a aac -b:a 64k audio.m4a
```
Mono 16kHz 64kbps AAC — a 65-min video (151 MB stream) becomes ~35 MB audio.
### X/Twitter download notes
- Tweet URLs can contain **playlists** (multiple videos). Use `--playlist-item N` to pick one.
- Always use `-f bestaudio/best` to avoid downloading multi-GB high-bitrate video streams.
- A 65-min interview's video variant can be 2.8+ GB; audio-only is ~63 MB (128 kbps).
### Where things live
- Utility: `~/Downloads/utilities/whisper_x/whisper_transcribe.py`
- Strategy doc: `~/Downloads/utilities/whisper_x/STRATEGY.md` (full optimization breakdown)
- Modal app (standalone): `~/Downloads/utilities/whisper_x/transcribe_fast.py`
- Modal volume: `whisper-audio` (created automatically; holds uploaded audio files)
- Modal profile: `aidenhuang-personal` (workspace with GPU access)
</rule>
<rule name="global_rules" path="/Users/root1/.codeium/windsurf/memories/global_rules.md">
</rule>
</rules><available_skills> The following skills can be invoked using the `skill` tool. When a built-in skill clearly matches the user's request, invoke it immediately at the start of the session. - **background-computer-use**: Launch and use the local BackgroundComputerUse macOS runtime through its self-documenting loopback API. Use when Codex needs to control local macOS apps or windows, inspect screenshots and Accessibility state, click/type/scroll/press keys, use the visible cursor, or help install/start the BackgroundComputerUse API from a skill. (source: /Users/root1/.devin/skills/background-computer-use/SKILL.md) - **devin-for-terminal**: Look up Devin CLI documentation (skills, extensibility, configuration, commands, models, troubleshooting) (source: /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs) - **declarative-repo-setup**: Generate and verify a working environment.yaml (Devin snapshot-setup blueprint) for a repo (source: builtin:drs) </available_skills>
I'll use yt-dlp to get the auto-generated captions for this video and summarize it.
{
"command": "yt-dlp --list-subs \"https://www.youtube.com/watch?v=l6DKRf-fAAM\" 2>&1 | head -50"
}Canceled due to user interrupt (tool_call_id: toolu_bdrk_01Vu7w3zvMCkYb5Eq8EDToCe)
make sure the summary is befitting of the quality of the video production
{
"command": "yt-dlp --write-auto-subs --sub-lang en --skip-download --output \"/tmp/yt_caption\" \"https://www.youtube.com/watch?v=l6DKRf-fAAM\" 2>&1"
}Output from command in shell b7d02a:
WARNING: Your yt-dlp version (2025.12.08) is older than 90 days!
It is strongly recommended to always use the latest version.
You installed yt-dlp with pip or using the wheel from PyPi; Use that to update.
To suppress this warning, add --no-update to your command/config.
[youtube] Extracting URL: https://www.youtube.com/watch?v=l6DKRf-fAAM
[youtube] l6DKRf-fAAM: Downloading webpage
[youtube] l6DKRf-fAAM: Downloading tv client config
[youtube] l6DKRf-fAAM: Downloading player 4918c89a-main
[youtube] l6DKRf-fAAM: Downloading tv player API JSON
[youtube] l6DKRf-fAAM: Downloading android sdkless player API JSON
[youtube] [jsc:deno] Solving JS challenges using deno
WARNING: [youtube] [jsc] Remote components challenge solver script (deno) and NPM package (deno) were skipped. These may be required to solve JS challenges. You can enable these downloads with --remote-components ejs:github (recommended) or --remote-components ejs:npm , respectively. For more information and alternatives, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] l6DKRf-fAAM: n challenge solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. Review any warnings presented before this message. For more details, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
[info] l6DKRf-fAAM: Downloading 1 format(s): 139-2
[info] There are no subtitles for the requested languages
Exit code: 0No auto-generated English subtitles were found. Let me try listing all available subtitle tracks to see what's there.
{
"command": "yt-dlp --list-subs \"https://www.youtube.com/watch?v=l6DKRf-fAAM\" 2>&1"
}Output from command in shell b235a8:
WARNING: Your yt-dlp version (2025.12.08) is older than 90 days!
It is strongly recommended to always use the latest version.
You installed yt-dlp with pip or using the wheel from PyPi; Use that to update.
To suppress this warning, add --no-update to your command/config.
[youtube] Extracting URL: https://www.youtube.com/watch?v=l6DKRf-fAAM
[youtube] l6DKRf-fAAM: Downloading webpage
[youtube] l6DKRf-fAAM: Downloading tv client config
[youtube] l6DKRf-fAAM: Downloading player 4918c89a-main
[youtube] l6DKRf-fAAM: Downloading tv player API JSON
[youtube] l6DKRf-fAAM: Downloading android sdkless player API JSON
[youtube] [jsc:deno] Solving JS challenges using deno
WARNING: [youtube] [jsc] Remote components challenge solver script (deno) and NPM package (deno) were skipped. These may be required to solve JS challenges. You can enable these downloads with --remote-components ejs:github (recommended) or --remote-components ejs:npm , respectively. For more information and alternatives, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] l6DKRf-fAAM: n challenge solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. Review any warnings presented before this message. For more details, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
[info] Available automatic captions for l6DKRf-fAAM:
Language Name Formats
ab-en Abkhazian from English vtt, srt, ttml, srv3, srv2, srv1, json3
aa-en Afar from English vtt, srt, ttml, srv3, srv2, srv1, json3
af-en Afrikaans from English vtt, srt, ttml, srv3, srv2, srv1, json3
ak-en Akan from English vtt, srt, ttml, srv3, srv2, srv1, json3
sq-en Albanian from English vtt, srt, ttml, srv3, srv2, srv1, json3
am-en Amharic from English vtt, srt, ttml, srv3, srv2, srv1, json3
ar-en Arabic from English vtt, srt, ttml, srv3, srv2, srv1, json3
hy-en Armenian from English vtt, srt, ttml, srv3, srv2, srv1, json3
as-en Assamese from English vtt, srt, ttml, srv3, srv2, srv1, json3
ay-en Aymara from English vtt, srt, ttml, srv3, srv2, srv1, json3
az-en Azerbaijani from English vtt, srt, ttml, srv3, srv2, srv1, json3
bn-en Bangla from English vtt, srt, ttml, srv3, srv2, srv1, json3
ba-en Bashkir from English vtt, srt, ttml, srv3, srv2, srv1, json3
eu-en Basque from English vtt, srt, ttml, srv3, srv2, srv1, json3
be-en Belarusian from English vtt, srt, ttml, srv3, srv2, srv1, json3
bho-en Bhojpuri from English vtt, srt, ttml, srv3, srv2, srv1, json3
bs-en Bosnian from English vtt, srt, ttml, srv3, srv2, srv1, json3
br-en Breton from English vtt, srt, ttml, srv3, srv2, srv1, json3
bg-en Bulgarian from English vtt, srt, ttml, srv3, srv2, srv1, json3
my-en Burmese from English vtt, srt, ttml, srv3, srv2, srv1, json3
ca-en Catalan from English vtt, srt, ttml, srv3, srv2, srv1, json3
ceb-en Cebuano from English vtt, srt, ttml, srv3, srv2, srv1, json3
zh-Hans-en Chinese (Simplified) from English vtt, srt, ttml, srv3, srv2, srv1, json3
zh-Hant-en Chinese (Traditional) from English vtt, srt, ttml, srv3, srv2, srv1, json3
co-en Corsican from English vtt, srt, ttml, srv3, srv2, srv1, json3
hr-en Croatian from English vtt, srt, ttml, srv3, srv2, srv1, json3
cs-en Czech from English vtt, srt, ttml, srv3, srv2, srv1, json3
da-en Danish from English vtt, srt, ttml, srv3, srv2, srv1, json3
dv-en Divehi from English vtt, srt, ttml, srv3, srv2, srv1, json3
nl-en Dutch from English vtt, srt, ttml, srv3, srv2, srv1, json3
dz-en Dzongkha from English vtt, srt, ttml, srv3, srv2, srv1, json3
en-en English from English vtt, srt, ttml, srv3, srv2, srv1, json3
eo-en Esperanto from English vtt, srt, ttml, srv3, srv2, srv1, json3
et-en Estonian from English vtt, srt, ttml, srv3, srv2, srv1, json3
ee-en Ewe from English vtt, srt, ttml, srv3, srv2, srv1, json3
fo-en Faroese from English vtt, srt, ttml, srv3, srv2, srv1, json3
fj-en Fijian from English vtt, srt, ttml, srv3, srv2, srv1, json3
fil-en Filipino from English vtt, srt, ttml, srv3, srv2, srv1, json3
fi-en Finnish from English vtt, srt, ttml, srv3, srv2, srv1, json3
fr-en French from English vtt, srt, ttml, srv3, srv2, srv1, json3
gaa-en Ga from English vtt, srt, ttml, srv3, srv2, srv1, json3
gl-en Galician from English vtt, srt, ttml, srv3, srv2, srv1, json3
lg-en Ganda from English vtt, srt, ttml, srv3, srv2, srv1, json3
ka-en Georgian from English vtt, srt, ttml, srv3, srv2, srv1, json3
de-en German from English vtt, srt, ttml, srv3, srv2, srv1, json3
el-en Greek from English vtt, srt, ttml, srv3, srv2, srv1, json3
gn-en Guarani from English vtt, srt, ttml, srv3, srv2, srv1, json3
gu-en Gujarati from English vtt, srt, ttml, srv3, srv2, srv1, json3
ht-en Haitian Creole from English vtt, srt, ttml, srv3, srv2, srv1, json3
ha-en Hausa from English vtt, srt, ttml, srv3, srv2, srv1, json3
haw-en Hawaiian from English vtt, srt, ttml, srv3, srv2, srv1, json3
iw-en Hebrew from English vtt, srt, ttml, srv3, srv2, srv1, json3
hi-en Hindi from English vtt, srt, ttml, srv3, srv2, srv1, json3
hmn-en Hmong from English vtt, srt, ttml, srv3, srv2, srv1, json3
hu-en Hungarian from English vtt, srt, ttml, srv3, srv2, srv1, json3
is-en Icelandic from English vtt, srt, ttml, srv3, srv2, srv1, json3
ig-en Igbo from English vtt, srt, ttml, srv3, srv2, srv1, json3
id-en Indonesian from English vtt, srt, ttml, srv3, srv2, srv1, json3
iu-en Inuktitut from English vtt, srt, ttml, srv3, srv2, srv1, json3
ga-en Irish from English vtt, srt, ttml, srv3, srv2, srv1, json3
it-en Italian from English vtt, srt, ttml, srv3, srv2, srv1, json3
ja-en Japanese from English vtt, srt, ttml, srv3, srv2, srv1, json3
jv-en Javanese from English vtt, srt, ttml, srv3, srv2, srv1, json3
kl-en Kalaallisut from English vtt, srt, ttml, srv3, srv2, srv1, json3
kn-en Kannada from English vtt, srt, ttml, srv3, srv2, srv1, json3
kk-en Kazakh from English vtt, srt, ttml, srv3, srv2, srv1, json3
kha-en Khasi from English vtt, srt, ttml, srv3, srv2, srv1, json3
km-en Khmer from English vtt, srt, ttml, srv3, srv2, srv1, json3
rw-en Kinyarwanda from English vtt, srt, ttml, srv3, srv2, srv1, json3
ko-en Korean from English vtt, srt, ttml, srv3, srv2, srv1, json3
kri-en Krio from English vtt, srt, ttml, srv3, srv2, srv1, json3
ku-en Kurdish from English vtt, srt, ttml, srv3, srv2, srv1, json3
ky-en Kyrgyz from English vtt, srt, ttml, srv3, srv2, srv1, json3
lo-en Lao from English vtt, srt, ttml, srv3, srv2, srv1, json3
la-en Latin from English vtt, srt, ttml, srv3, srv2, srv1, json3
lv-en Latvian from English vtt, srt, ttml, srv3, srv2, srv1, json3
ln-en Lingala from English vtt, srt, ttml, srv3, srv2, srv1, json3
lt-en Lithuanian from English vtt, srt, ttml, srv3, srv2, srv1, json3
lua-en Luba-Lulua from English vtt, srt, ttml, srv3, srv2, srv1, json3
luo-en Luo from English vtt, srt, ttml, srv3, srv2, srv1, json3
lb-en Luxembourgish from English vtt, srt, ttml, srv3, srv2, srv1, json3
mk-en Macedonian from English vtt, srt, ttml, srv3, srv2, srv1, json3
mg-en Malagasy from English vtt, srt, ttml, srv3, srv2, srv1, json3
ms-en Malay from English vtt, srt, ttml, srv3, srv2, srv1, json3
ml-en Malayalam from English vtt, srt, ttml, srv3, srv2, srv1, json3
mt-en Maltese from English vtt, srt, ttml, srv3, srv2, srv1, json3
gv-en Manx from English vtt, srt, ttml, srv3, srv2, srv1, json3
mi-en Māori from English vtt, srt, ttml, srv3, srv2, srv1, json3
mr-en Marathi from English vtt, srt, ttml, srv3, srv2, srv1, json3
mn-en Mongolian from English vtt, srt, ttml, srv3, srv2, srv1, json3
mfe-en Morisyen from English vtt, srt, ttml, srv3, srv2, srv1, json3
ne-en Nepali from English vtt, srt, ttml, srv3, srv2, srv1, json3
new-en Newari from English vtt, srt, ttml, srv3, srv2, srv1, json3
nso-en Northern Sotho from English vtt, srt, ttml, srv3, srv2, srv1, json3
no-en Norwegian from English vtt, srt, ttml, srv3, srv2, srv1, json3
ny-en Nyanja from English vtt, srt, ttml, srv3, srv2, srv1, json3
oc-en Occitan from English vtt, srt, ttml, srv3, srv2, srv1, json3
or-en Odia from English vtt, srt, ttml, srv3, srv2, srv1, json3
om-en Oromo from English … (42 chars truncated)
… (217 lines truncated)
Exit code: 0
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/3538e819/content.txt
</truncation_notice>The auto-captions are translated from English (not native). The en-en track is "English from English" — I'll download that.
{
"command": "yt-dlp --write-auto-subs --sub-lang en-en --sub-format json3 --skip-download --output \"/tmp/yt_caption\" \"https://www.youtube.com/watch?v=l6DKRf-fAAM\" 2>&1"
}Output from command in shell 562d2d:
WARNING: Your yt-dlp version (2025.12.08) is older than 90 days!
It is strongly recommended to always use the latest version.
You installed yt-dlp with pip or using the wheel from PyPi; Use that to update.
To suppress this warning, add --no-update to your command/config.
[youtube] Extracting URL: https://www.youtube.com/watch?v=l6DKRf-fAAM
[youtube] l6DKRf-fAAM: Downloading webpage
[youtube] l6DKRf-fAAM: Downloading tv client config
[youtube] l6DKRf-fAAM: Downloading player 4918c89a-main
[youtube] l6DKRf-fAAM: Downloading tv player API JSON
[youtube] l6DKRf-fAAM: Downloading android sdkless player API JSON
WARNING: [youtube] l6DKRf-fAAM: Some tv client https formats have been skipped as they are missing a url. YouTube may have enabled the SABR-only or Server-Side Ad Placement experiment for the current session. See https://github.com/yt-dlp/yt-dlp/issues/12482 for more details
[youtube] [jsc:deno] Solving JS challenges using deno
WARNING: [youtube] [jsc] Remote components challenge solver script (deno) and NPM package (deno) were skipped. These may be required to solve JS challenges. You can enable these downloads with --remote-components ejs:github (recommended) or --remote-components ejs:npm , respectively. For more information and alternatives, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] l6DKRf-fAAM: n challenge solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. Review any warnings presented before this message. For more details, refer to https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] l6DKRf-fAAM: Some web client https formats have been skipped as they are missing a url. YouTube is forcing SABR streaming for this client. See https://github.com/yt-dlp/yt-dlp/issues/12482 for more details
[info] l6DKRf-fAAM: Downloading subtitles: en-en
[info] l6DKRf-fAAM: Downloading 1 format(s): 401+251-2
[info] Writing video subtitles to: /tmp/yt_caption.en-en.json3
[download] Destination: /tmp/yt_caption.en-en.json3
[download] 1.00KiB at Unknown B/s (00:00:00)
[download] 3.00KiB at Unknown B/s (00:00:00)
[download] 7.00KiB at Unknown B/s (00:00:00)
[download] 15.00KiB at 6.58MiB/s (00:00:00)
[download] 31.00KiB at 1.75MiB/s (00:00:00)
[download] 63.00KiB at 2.57MiB/s (00:00:00)
[download] 83.25KiB at 2.80MiB/s (00:00:00)
[download] 100% of 83.25KiB in 00:00:00 at 696.66KiB/s
Exit code: 0{
"command": "python3 - <<'EOF'\nimport json\n\nwith open(\"/tmp/yt_caption.en-en.json3\") as f:\n data = json.load(f)\n\n# Extract plain text from json3 format\nlines = []\nfor event in data.get(\"events\", []):\n segs = event.get(\"segs\", [])\n text = \"\".join(s.get(\"utf8\", \"\") for s in segs).strip()\n if text and text != \"\\n\":\n lines.append(text)\n\nfull_text = \" \".join(lines)\n# Collapse excessive whitespace\nimport re\nfull_text = re.sub(r'\\s+', ' ', full_text).strip()\n\nprint(f\"Total characters: {len(full_text)}\")\nprint(\"---TRANSCRIPT---\")\nprint(full_text[:8000])\nEOF\n"
}Output from command in shell dfb3a5: Total characters: 34208 ---TRANSCRIPT--- [Submit subtitle corrections at criblate.com] When you encode text into binary, it's often nice to use as little data as possible, so you might naturally wonder, is there some kind of fundamental limit on how efficiently text can be compressed? The default encoding with ASCII is pretty inefficient. Each character is represented with eight full bits. If you apply a little cleverness associating more common characters with smaller bit strings, you can get this down to an average of around four bits per character, and then much smarter methods than that, that leverage patterns among long sequences of text, can get even better still. But again, what's the limit? I'm guessing many of you might have heard an estimate here, I will share one at the end, but much more interesting than any single numerical answer is how you would even approach answering it. This question dates back at least to the 1940s, with Claude Shannon's seminal work that kicked off information theory. Now what's very interesting is how the math that he developed to answer questions like this has turned out to be surprisingly useful for modern machine learning. To take the salient example of today, when large language models are trained, the pre-training portion of that is usually described as being about next token prediction, specifically using something called cross-entropy loss. Now that term, cross-entropy, has its roots in information theory. But what's also interesting is that one of the conclusions of information theory says that prediction and compression are mathematically equivalent. They turn out to be two sides of the same coin. Which means, you can entirely reframe how you think about the pre-training objective as not really being about next token prediction per se, but instead as being about creating the most efficient possible text compressor. Later I'll explain exactly how that works, but when you do, I think it offers more clarity on what this notion of cross-entropy really is, and why you're using it. Also, I don't know about you, but to me at least, there's just something very intriguing about using compression as a fundamental objective while pursuing intelligence. In fact, some people have gone so far as to say, compression is intelligence. Now, as stated, that's a hard claim to judge rigorously, since intelligence is such a squishy and ill-defined term. The safer claim would be that the mathematical theory of compression is bizarrely relevant to artificial intelligence. Still, the pithy phrase is thought-provoking, so this is the first in a trilogy of videos aimed at laying down the mathematical fundamentals to assess what this claim is really getting at. For the next 30 minutes or so, you and I will be focused on understanding the limits of compression, and seeing if we can get you to feel like you could have rediscovered the core idea behind Shannon's noiseless coding theorem. Doing so involves rediscovering a few key definitions, namely those of information and entropy. And it might sound a little weird for me to describe definitions as something to be discovered, but great definitions are often the residue of some kind of insight. See, these two specific terms are really not hard to define in the sense of laying down a formula for you, but doing so too early would spoil a good story. It is much more fun to see how you are inexorably drawn to them by asking about the limits of compressing language. And what I want you to notice by the end here is how we can't really answer this question, or at least Shannon couldn't, without necessarily engaging with some notion of intelligence. We will get to modelling language, which is wonderfully complicated, but it helps to warm up with a simpler example containing the same essential ideas needed for our path of rediscovery. Imagine you have a robot that we have sent to a faraway moon, whose job is to wander the surface and collect data. From here on Earth, we send it instructions for how to move that are going to be limited to four very simple possibilities. Move up, down, left, or right, each one with a fixed step size. The nuance will be that these instructions are not uniformly distributed. Half of everything we send is up, one fourth of the instructions are down, one eighth are left, and the other eighth are right. It's a little contrived, it's meant to be a simple example, and in that spirit of simplicity, we're also going to assume that every instruction is independent. They are sampled from this distribution regardless of the preceding context. When we send data to this bot far away, we're doing it as a stream of bits, ones and zeros, and it's very slow and costly to do that, so the natural question, the warm-up puzzle for you today, is to ask what is the most efficient possible way to encode these instructions as a stream of bits. And here, we might imagine three students who each attempt an answer, one who's straightforward, one who is clever, and one who is very theoretical. The straightforward student immediately raises their hand and says, well, for each of these four instructions, we could just use two bits. Maybe 00 encodes up, 01 encodes down, 10 encodes left, and 11 encodes right. On the receiving end, this makes it very easy for the robot to decode. It simply breaks up the bitstream into chunks of two, and translates each chunk into the appropriate instruction. Now, the clever student points out, well, that does nothing to take advantage of the fact that up comes up way more frequently than left or right. They propose a method where we use a different number of bits for each instruction. I'll explain it in just a second, but I do want to call out the fact that as soon as you let different instructions get different numbers of bits, it's certainly not obvious that the robot is going to know how to decode it. After all, it needs to somehow know where to draw the dividing lines. Now, being very clever, the second student has thought that through, but it's easiest to explain if I just kind of lay it all out. What they suggest is letting the single bit 0 represent up, letting the two bits 10 represent down, using 110 for left and 111 for right. You can calculate the average number of bits per instruction that this would require with a simple weighted sum. Half the time, only one bit is used, a quarter of the time, two bits are needed, and the remaining times, you need three bits. When you add this all up, as a weighted sum, you get 1.75 bits per instruction, which is indeed better than the more naive approach of a flat two bits per instruction. The second student is spending more bits on those last two instructions, but because they come up so much less frequently, this is more than made up for by using only one bit for that most common instruction. And you can see this efficiency bear out empirically too. This right here is a set of instructions sampled from that distribution. You'll notice lots of ups, about half as many downs, and even fewer lefts and rights. When you turn each symbol into the appropriate bit string following this encoding, the resulting sequence of bits is indeed shorter than what the naive method gave. But I hear some of you asking, how does the robot know how to decode this? Can we be sure that there's an unambiguous way to draw the dividing lines? In the lingo of encodings, each of these four bit strings used to encode the instructions is called a code word. And as we step through things here, see if you can figure out what rule the clever student was following to make sure that their code words don't conflict with each other. And really, you just need to look at an example bit stream from the robot's perspective and think it through. Let's say that that very first bit that it receives is a 1. So far, this could be encoding any of the last three instructions, down, left, or right. If what follows is a 0, then it can only be down. There is no other possibility. So the robot can register Exit code: 0
{
"command": "python3 - <<'EOF'\nimport json, re\n\nwith open(\"/tmp/yt_caption.en-en.json3\") as f:\n data = json.load(f)\n\nlines = []\nfor event in data.get(\"events\", []):\n segs = event.get(\"segs\", [])\n text = \"\".join(s.get(\"utf8\", \"\") for s in segs).strip()\n if text and text != \"\\n\":\n lines.append(text)\n\nfull_text = \" \".join(lines)\nfull_text = re.sub(r'\\s+', ' ', full_text).strip()\n\n# Print remaining transcript\nprint(full_text[8000:])\nEOF\n"
}Output from command in shell 58acf2: that as a complete instruction. From there, starting fresh, if what follows is a 0, that must be encoding up, since there is no other possibility. Nothing else starts with a 0. After that, if they see a 1, this is so far ambiguous, it's the prefix for any of the last three code words. If the next bit is another 1, it remains ambiguous, it's the prefix of the last two. But that following 0 then makes clear these three bits must have been encoding left. In short, all the robot has to do is read in the next sequence of bits until the moment that it forms a complete code word, at which point it can register it as a complete instruction. If you step back and think about it, the key constraint for this all to work is that no code word can be a prefix of another one. For example, let's say you tried to introduce some fifth instruction that had the code word 100. That would cause conflicts, because after reading 10, it's not clear whether that's supposed to be down, or if it's supposed to be the start of this new fifth instruction. When you avoid this kind of conflict, an encoding method like this has a special name. It's known in the business as a prefix-free code, or what is confusingly synonymous, it's actually more commonly known as a prefix code. And there's a really nice way to visualize choosing prefix-free codes with a certain diagram that represents every possible binary string. This diagram also has some helpful parallels with how I want to show entropy in just a few minutes. It's probably decently intuitive what's going on if you just pause and stare at it for a bit, but I'll call out the important features. Each layer shows every bit string of a given length, and you'll notice everything that starts with a 0 lives on the left half of this diagram, and everything that starts with a 1 lives on the right half of the diagram. And that same idea continues recursively as you go up. So everything starting with 00 is above this quarter of the diagram, everything starting with 01 is above this quarter of the diagram, and so on. In particular, the key property is that every binary string in this diagram is a prefix for everything that sits above it. So when our clever student chose to allocate a single bit 0 to represent the instruction up, they were effectively consuming half of the space of all possible code words by doing so, since everything starting with a 0 is now prohibited based on this prefix-free property. Similarly, allocating one 0 to down eats up another fourth of this space, and the remaining two instructions each eat up an eighth of the space. You can just see how nothing is left over. And at this point, I'm guessing that there's some bell kind of resonating in your mind, given that these proportions are all exactly the same as the probability for each instruction coming up. And in fact, that tickling sensation of a relationship between data size and probabilities is exactly the founding insight for information theory. When you stare at this pleasing alignment, it might suggest that the clever student's solution is not just better, maybe it's somehow perfect. But at the moment, that feels like a bold claim to make. How could you know that there's not some ultra-clever method that does something fancy with really long sequences of instructions that somehow makes it so the average bits per instruction gets even lower than this? This takes us to the third student, Head in the Clouds, who has not really been thinking about actual codes that they can implement. They have been pondering what properties a perfect, optimally efficient code would have. They have this really clever idea, which is to argue that random noise should be incompressible, and therefore, a perfect compression algorithm should produce a bitstream that's indistinguishable from random noise. What's neat is, even though that sounds kind of simple, this can actually lead you to reinventing the idea of Shannon entropy. Now when I say random noise, what I mean is that each bit is a 1 or a 0, with 50% probability, and all of the bits act independent from one another. It's not hard to show that our second student's encoding for the robot genuinely follows this rule. If you think about it, there's a 50% chance that the message starts with the instruction up, so that first bit has a 50% chance of being a 0, otherwise it's a 1. And then if it is a 1, there's a 50% chance that the instruction is down, meaning the next bit is 0, otherwise it's a 1. And so on. Each new bit really does act like an independent coin flip, so to the receiver, that compressed bitstream really is indistinguishable from random noise. But why am I saying that random noise is incompressible? Well, for that, let's really focus on things from the receiver's perspective. And here, instead of thinking instruction by instruction, it helps to zoom out and frame our discussion in terms of entire messages and what the encoding for full messages looks like. When that receiver sees a string of n bits representing some compressed message, this is one out of 2 to the n possible messages they could have received that would have the same size. And then critically, because we're assuming this looks like random noise, all 2 to the n of those messages must be equally likely. Now to be clear, we're not just talking about the robot example anymore. Our theoretical head-in-the-cloud student really wants to make an argument for what perfect compression looks like for any data type, like maybe we're compressing language or compressing images, and they want an argument that works regardless of the details of what specific compression algorithm you're using. A loose intuition in your head is that maybe among all of the messages that compress down to n bits, some of them would contain a larger amount of predictable data, while others would have started as a smaller amount of unpredictable data. But the key point is that if the compressed bitstream really does look like random noise, meaning all of the bitstrings of size n are equally likely, it must be the case that all the underlying messages being compressed were equally likely to arise, and even more specifically with probability 1 over 2 to the n. So again, why am I claiming that this is incompressible? Well, let's pull up that same diagram, representing the space of all possible binary strings. In this case, we can think of all the encodings for our 2 to the n equally likely messages as being the bitstrings on one layer of this diagram. If you tried to define an alternate scheme, where you tried to become more efficient by making one of these messages use fewer bits, it would move lower on the diagram, which means it then overlaps with some other message, so you would have to put that other message somewhere else in this diagram. At minimum, that means it's sharing a space with yet a third message, and those last two have to kind of get bumped up one layer, requiring that extra bit for each of them to disambiguate. So saving one bit over here costs you two bits elsewhere. You're basically pushing down a bump on the rug, only to see it pop up even worse in another spot. In general, any message encoded with a string lower on this diagram is going to be eating up more than its fair share of the space, which forces multiple other messages to occupy a smaller region, bumping them higher up, meaning they're encoded less efficiently. I'm going to guess that it's decently intuitive for everyone watching that the most efficient scheme here for equally likely messages is to give them all the same number of bits, but I do think the diagram gives a more concrete way to see why you end up with unfavorable trade-offs otherwise. Once you have this idea that perfect compression looks like random noise, this is exactly the kind of thing I was referencing in the intro, about how insights can lead you to definitions. Notice how what we're basically saying is that a message that uses n bits in a perfect scheme must have a probability of 1 over 2 to the n, or 2 to the negative n. If you take the log base 2 of both sides here and then negate, this is equivalent to saying that the number of bits allocated to a message, assuming perfect compression, is negative log base 2 of p, where p is the probability of occurring. This negative log expression is the fundamental formula for all of information theory. Everything up to this point is me trying to make this value feel like something you are inevitably drawn to by asking about perfect compression, rather than it feeling like some arbitrary definition that we start with. Some students do find this negative log takes a little getting used to. You know, it looks like it should be a negative value before you think about the fraction in the middle, but really the intuitive way to read it is that it's asking how many times do you chop your space of possibilities in half to get to a certain quantity. Here, let me pull up some axes and show you the graph of negative log of p. I've also seen some authors default to writing this as the log of 1 over p, which maybe you find more intuitive. You could also think of it as the log base 1 half of p, which I've never really seen anyone lean into, but personally I would be quite partial to that as a convention. Now, however you write it, what Shannon realized is that this is a very useful way to think about the information that a message contains even when literal perfect compression is not possible and p is no longer a clean power of 2, which would mean that the output here is some fractional amount. In fact, he defined this expression to be the information of an event. The image I sometimes have in my head is to picture the probability of an event with a little pie chart, and the information is this bar above it that gets kind of pumped up to be taller as the probability is squeezed closer to zero. That is, unlikely messages contain a lot of inform… (16246 chars truncated) Exit code: 0 <truncation_notice> Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/ecaddaf3/content.txt </truncation_notice>
{
"file_path": "/var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/ecaddaf3/content.txt"
}<file-view path="/var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/ecaddaf3/content.txt" start_line="1" end_line="5" total_lines="5"> 1|Output from command in shell 58acf2: 2|that as a complete instruction. From there, starting fresh, if what follows is a 0, that must be encoding up, since there is no other possibility. Nothing else starts with a 0. After that, if they see a 1, this is so far ambiguous, it's the prefix for any of the last three code words. If the next bit is another 1, it remains ambiguous, it's the prefix of the last two. But that following 0 then makes clear these three bits must have been encoding left. In short, all the robot has to do is read in the next sequence of bits until the moment that it forms a complete code word, at which point it can register it as a complete instruction. If you step back and think about it, the key constraint for this all to work is that no code word can be a prefix of another one. For example, let's say you tried to introduce some fifth instruction that had the code word 100. That would cause conflicts, because after reading 10, it's not clear whether that's supposed to be down, or if it's supposed to be the start of this new fifth instruction. When you avoid this kind of conflict, an encoding method like this has a special name. It's known in the business as a prefix-free code, or what is confusingly synonymous, it's actually more commonly known as a prefix code. And there's a really nice way to visualize choosing prefix-free codes with a certain diagram that represents every possible binary string. This diagram also has some helpful parallels with how I want to show entropy in just a few minutes. It's probably decently intuitive what's going on if you just pause and stare at it for a bit, but I'll call out the important features. Each layer shows every bit string of a given length, and you'll notice everything that starts with a 0 lives on the left half of this diagram, and everything that starts with a 1 lives on the right half of the diagram. And that same idea continues recursively as you go up. So everything starting with 00 is above this quarter of the diagram, everything startin... [truncated 24208 chars] 3| 4| 5|Exit code: 0 </file-view>
{
"command": "python3 - <<'EOF'\nimport json, re\n\nwith open(\"/tmp/yt_caption.en-en.json3\") as f:\n data = json.load(f)\n\nlines = []\nfor event in data.get(\"events\", []):\n segs = event.get(\"segs\", [])\n text = \"\".join(s.get(\"utf8\", \"\") for s in segs).strip()\n if text and text != \"\\n\":\n lines.append(text)\n\nfull_text = \" \".join(lines)\nfull_text = re.sub(r'\\s+', ' ', full_text).strip()\n\nprint(full_text[16000:])\nEOF\n"
}Output from command in shell 0dc8f7: scheme must have a probability of 1 over 2 to the n, or 2 to the negative n. If you take the log base 2 of both sides here and then negate, this is equivalent to saying that the number of bits allocated to a message, assuming perfect compression, is negative log base 2 of p, where p is the probability of occurring. This negative log expression is the fundamental formula for all of information theory. Everything up to this point is me trying to make this value feel like something you are inevitably drawn to by asking about perfect compression, rather than it feeling like some arbitrary definition that we start with. Some students do find this negative log takes a little getting used to. You know, it looks like it should be a negative value before you think about the fraction in the middle, but really the intuitive way to read it is that it's asking how many times do you chop your space of possibilities in half to get to a certain quantity. Here, let me pull up some axes and show you the graph of negative log of p. I've also seen some authors default to writing this as the log of 1 over p, which maybe you find more intuitive. You could also think of it as the log base 1 half of p, which I've never really seen anyone lean into, but personally I would be quite partial to that as a convention. Now, however you write it, what Shannon realized is that this is a very useful way to think about the information that a message contains even when literal perfect compression is not possible and p is no longer a clean power of 2, which would mean that the output here is some fractional amount. In fact, he defined this expression to be the information of an event. The image I sometimes have in my head is to picture the probability of an event with a little pie chart, and the information is this bar above it that gets kind of pumped up to be taller as the probability is squeezed closer to zero. That is, unlikely messages contain a lot of information, and the bar is relaxed to get shorter as the probability approaches 100%. Highly expected messages contain very low information. I want to be clear that there is content to this definition. It's more than just rescaling probabilities, as if converting to an alternate unit system. You've already seen the core idea. In a perfect compression scheme, the number of bits allocated to a full message precisely equals this information content. Now, of course, perfect compression is not always possible. Your probability's messages are likely not perfect powers of 2. So the more general way you would frame this is to say that the information of a message gives you a lower bound on how much it can be compressed, at least when you average overall possible messages. It's perfectly possible to overfit a compression algorithm to one specific case. And to really wrap your mind around information and fractional bits, we need to step up our game beyond that warm-up example. See, with the robot, the probabilities were overly clean. They were all perfect powers of 2, so that the information per symbol could precisely match a whole number of bits. But Shannon was very interested in this question of limits on compression for much more realistic and complicated cases, like natural language. For example, here I'll pull up the probabilities for each new letter in an example phrase, at least as determined by a little GPT that I'm running locally. And, by the way, if the probabilities provided by a model feel like those might be distinct from the probabilities of actual language, you would be right to raise an eyebrow. Hold on to that thought. It becomes very relevant later. One key difference with language is how heavily dependent on context everything is. The probability distribution for each new letter you see is highly, highly contingent on everything that came before it. The other big difference is that all of these probabilities are obviously not going to be clean, perfect powers of 2. So if you compute the Shannon information of each one of these, meaning you take the negative log base 2, all the numbers that you see are fractional amounts. And again, giving a vague interpretation of what these mean is really not hard. Information content is low for very predictable letters, and it's high for very unpredictable letters. But all of you watching this are smart enough to demand a more exact interpretation, one that justifies why these values deserve to be given the units of bits. I mean, it's not like there's going to be some perfect encoding where this letter i has a code word that is somehow 4.19 bits, or where this highly predictable o is somehow encoded with just a narrow sliver of a bit. The real meaning is related to encoding entire messages and coming up with a bound on their compression. The probability for a full phrase, like the one I'm showing here, looks like multiplying the probabilities for each successive new letter, where again, I'll emphasize that these successive letters' probabilities are conditioned on what came before. This is essentially the chain rule from probability. I want you to notice how nicely this plays with a logarithm. If you ask for the information of that full message, taking this negative log expression, because logarithms turn multiplication into addition, this breaks up really nicely as the sum of the information for each individual letter. In the final part of this trilogy, I'm going to walk you through a very specific compression algorithm that would actually compress this text to within one or two bits of this value. It's no longer as simple as mapping each character to a predefined code word, but you nevertheless get this very direct sense of thinking about adding up the fractional information content of each letter to determine the length of the final encoding that you use. So that's the key idea. Even if, by the time you need to relate this to actual data sizes, things will get rounded off to the nearest whole number, what Shannon realized is just how useful it is to work at this higher layer of abstraction, where all your information is allowed to freely be continuous and information of successive events really nicely adds together. Still, if you step back and imagine yourself very seriously assessing the fundamental compressibility of language, all of this hinges on the question of how you know the probabilities for each successive letter. Here, for all these animations, I've been illustrating things using a language model, but for one thing, that doesn't necessarily feel the same as the true probabilities underlying language, and for another, it's not even clear what we would mean by the true probabilities of language. So here, I actually think it's most enlightening to take a step back in time and see how Shannon himself thought about all of this long before language models or modern data analysis. Some of his earliest experiments on the information of language involved looking at specific short sequences of characters, what you often hear referred to as n-grams, and then tracking the statistics of what tended to follow. So, for example, if you scan through a few books, and you notice every single time the two letters th come up and you record what letters tend to follow, you could build up a table of those letters and let those statistics represent a sample for the probabilities you care about. Now, the problem is that this just completely breaks down for longer strings of letters, most notably, any string of text that never shows up in all of the books that you're analyzing. Nevertheless, it's perfectly reasonable to talk about what's expected to follow such a longer string. In fact, if anything, it's more important to talk about those cases, since longer context windows are when things are at their most predictable, and that's where you stand to get the most compression due to that predictability. So instead of using raw statistics, a little bit later, Shannon analyzed a different model of the English language that he had available to him, his wife Betty. As the story goes, Shannon pulled out a book and began asking Betty to guess each next letter. Shannon would then transcribe her guesses letter by letter. Every time she guessed incorrectly, he wrote down the correct letter, and then whenever she guessed correctly, he would just replace it with a dash. The idea was that this transcribed version he was creating has fewer actual letters than the original, but his point was that it contained the same amount of information, at least in the following sense. If he could somehow obtain an exact replica of his wife and conduct the guessing game again, he would only need to supply her with this reduced text. These letters by themselves would provide just the right prompting for his duplicate wife to exactly reproduce the original. Of course, in practice, a person wouldn't necessarily guess the same way twice, and while this qualitatively conveyed the idea of predictability allowing for compression, it wasn't yet a measurement of information. A bit later still, in his 1950 paper, Prediction and Entropy of Printed English, Shannon updated the experimental design to get a more robust read on this slippery question of the average information content in English. This time interviewing more people, instead of just logging whether each guess was right or wrong, Shannon recorded how many guesses were necessary for a human guesser to come up with the correct next letter. And then separately, he had this whole method for associating the number of guesses required with an implicit probability that a person would have been assigning to the true next letter. The details of that are a little bit in the weeds, but the broader point I want to make is how, in analyzing language, he wasn't just doing pure data analysis looking through books. He was trying to probe at an underlying model of language, namely the interviewee's brain. These bra… (8246 chars truncated) Exit code: 0 <truncation_notice> Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/95050cf8/content.txt </truncation_notice>
{
"command": "python3 - <<'EOF'\nimport json, re\n\nwith open(\"/tmp/yt_caption.en-en.json3\") as f:\n data = json.load(f)\n\nlines = []\nfor event in data.get(\"events\", []):\n segs = event.get(\"segs\", [])\n text = \"\".join(s.get(\"utf8\", \"\") for s in segs).strip()\n if text and text != \"\\n\":\n lines.append(text)\n\nfull_text = \" \".join(lines)\nfull_text = re.sub(r'\\s+', ' ', full_text).strip()\n\nprint(full_text[24000:])\nEOF\n"
}Output from command in shell 441257: he English language that he had available to him, his wife Betty. As the story goes, Shannon pulled out a book and began asking Betty to guess each next letter. Shannon would then transcribe her guesses letter by letter. Every time she guessed incorrectly, he wrote down the correct letter, and then whenever she guessed correctly, he would just replace it with a dash. The idea was that this transcribed version he was creating has fewer actual letters than the original, but his point was that it contained the same amount of information, at least in the following sense. If he could somehow obtain an exact replica of his wife and conduct the guessing game again, he would only need to supply her with this reduced text. These letters by themselves would provide just the right prompting for his duplicate wife to exactly reproduce the original. Of course, in practice, a person wouldn't necessarily guess the same way twice, and while this qualitatively conveyed the idea of predictability allowing for compression, it wasn't yet a measurement of information. A bit later still, in his 1950 paper, Prediction and Entropy of Printed English, Shannon updated the experimental design to get a more robust read on this slippery question of the average information content in English. This time interviewing more people, instead of just logging whether each guess was right or wrong, Shannon recorded how many guesses were necessary for a human guesser to come up with the correct next letter. And then separately, he had this whole method for associating the number of guesses required with an implicit probability that a person would have been assigning to the true next letter. The details of that are a little bit in the weeds, but the broader point I want to make is how, in analyzing language, he wasn't just doing pure data analysis looking through books. He was trying to probe at an underlying model of language, namely the interviewee's brain. These brains that he could talk to were effectively treated as black boxes, ones with a sophisticated and yet indescribable understanding of language and an ability to predict characters based on the context. Now these days, in the 2020s, we have gone from merely interrogating black boxes that process language to designing them. The reason that you and I are here, revisiting the roots of information theory and this study of how compressible language is, is because of how much of the math in modern machine learning leverages the formulas that emerged in this field. Aside from the definition of information itself, there are three more key expressions that I want you to feel like you could have rediscovered, not just because rediscovery makes them more memorable, but because seeing how they naturally arise from studying compression gives a more solid ground from which you can assess that thought-provoking interplay between compression and intelligence. You already have everything you need to reinvent the first of these, which is entropy. Imagine you have any signal that can be thought of as a sequence of symbols, whether that's the four robot instructions, the English language, or really anything else you dream up. Entropy asks about the average amount of information for each symbol, and then based on everything we've been discussing, where perfect compression looks like random noise, and how, in that case, the number of bits used for a message is the same as the information content of that message, which again breaks down really nicely as the sum of the information of all the symbols. This question of measuring the average information per symbol is basically asking about the limit of compression. It would give you a lower bound on how efficiently a given signal could be compressed. And you and I already calculated this in one setting, when we took that weighted sum to find the average bits per instruction for the perfect encoding of the robot case. Basically, because that encoding was perfect, each symbol's code word length is exactly the same as its information content. So, effectively, we're calculating the average information per symbol. To generalize this, here's how that looks. For any given probability distribution, characterizing whatever symbols your messages are made out of, the average information per symbol should look like adding up p times the negative log of p for each probability p in that distribution. And take a moment to notice how we're visualizing this expression. Throughout the video, we've been picturing probability distributions with stacked horizontal bars, where each bar's width is equal to its probability, so altogether they add up to be 1. Now, above each one of those, I've put a rectangle whose height is locked to be the corresponding information value, the negative log base 2 of that probability. So the weighted sum up top, the average information per symbol, can be thought of as the total area of all these rectangles together. This is not yet fully general. It only describes the compression limit in cases where every new symbol follows some identical distribution. That's true in the robot case, but it's not true in English, for example. Nevertheless, it's important enough to deserve a name. And there's a fun story about how John von Neumann supposedly told Shannon that he should call this expression entropy, since, for one thing, it resembles an expression already used for the idea of entropy in statistical mechanics, and for another, quote, nobody knows what entropy really is, so in an argument, you'll always have the advantage. I looked into it, it's probably apocryphal, but there seems to be a grain of truth to this. Now, whatever the true story is, Shannon did call this quantity entropy, and he denoted it with the letter h. And it's fun to take a moment to play around with this graphic to build a little intuition. The more evenly distributed a probability distribution, the higher the total entropy, whereas if we squish things to become very uneven, maybe with one event dominating the probability space, this would have a very low entropy, since that one overwhelmingly probable event carries correspondingly little information. And then also, if you divide up the probability space even more, meaning it's distributed over more total possible symbols, each one has more information, so the total entropy is higher. And once more, it's fun to kind of slosh everything around, where a very skewed distribution gives lower total entropy, whereas a more even spread gives us higher entropy. The vague qualitative intuition here is that entropy measures the amount of uncertainty in a distribution, but the more precise understanding, justifying why it deserves to be given in units of bits, is that it describes the minimum number of bits per symbol necessary to encode a message following this distribution. That statement, and everything we just covered, is more or less the content of a core theorem in Shannon's 1948 paper that kicked off information theory, the noiseless coding theorem. It states that no encoding can ever be more efficient than this limit, and even more strongly, he showed that it's always possible to get arbitrarily close to this limit. Now like I said, this expression only applies in cases where every symbol follows the same distribution, but Shannon was of course keenly interested in a more general setting, where the probabilities for each new symbol don't necessarily follow the same distribution. In particular, he spent a ton of time contemplating the compressibility of natural language. This requires a more general notion of entropy, something known as the entropy rate for a stochastic process. But it's the same basic question. What is the average information per symbol, except in this case we're now averaging over all possible messages. This is almost never a clean calculation that you can perform with some clean visual like the one we were just playing with. After all, what formula describes the probability distribution for language? So if you hear reference to the entropy of language, this is well beyond what any exact calculation can give you, hence why Shannon turned to estimates based on observation. And again, his methodology was not just data analysis. It's not some function that you can perform on a corpus of text. In order to get a satisfying estimate on compressibility, he found himself inevitably needing to probe at intelligent models of language. When his interviewees had at least 100 preceding letters of context, he estimated the entropy of English to be about one bit per character. This is a pretty wild number when you think about it, because it suggests that English could be compressed down to just a single yes or no answer for each character. Wild as that sounds, in the third part here, I will show you that algorithm I've referenced, where if you're allowed to use a high-quality language model for encoding and decoding, you can get surprisingly close to this limit in practice. Before then, it helps to understand a variation on entropy, known as cross-entropy, asking both how and why it's used in training large language models. If you want to learn that, as well as a few fun related ideas like how large models are distilled into smaller ones, and why on earth GZIP can recover the structure between distinct languages, come join me in part 2. Regular viewers will know that my new experiment for this year is to have a virtual career fair, a page at 3b1b.co. talent, where this audience can explore aligned career opportunities. The meaningful update is just how much more content there is now, mainly in the form of featured interviews between me and the relevant teams. Here's my thinking. I feel like when you're assessing potential jobs, it's almost impossible to get a true sense of what it's like to work at a place just by poking around online, and you learn orders of magnitude more if you have a chance to sit … (246 chars truncated) Exit code: 0 <truncation_notice> Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/01e7d986/content.txt </truncation_notice>
I now have the full transcript. This is 3Blue1Brown's video on information theory, entropy, and compression. Let me write a summary that matches the quality of Grant Sanderson's production.
The Central Question Can text compression have a fundamental limit? ASCII wastes bits; clever variable-length codes do better — but how much better is theoretically possible? This question, posed rigorously by Claude Shannon in the 1940s, launched the field of information theory. What makes it freshly relevant: the math Shannon invented to answer it turns out to be the mathematical backbone of how modern large language models are trained.
To build intuition, the video introduces a robot that receives movement instructions (up, down, left, right) with a skewed probability distribution: up = 1/2, down = 1/4, left = 1/8, right = 1/8.
Three students respond to the challenge of encoding these efficiently:
0 for up, 10 for down, 110 for left, 111 for right — achieving 1.75 bits per symbol on average by spending fewer bits on the most common instruction.The key insight from the prefix-free code is visual: choosing a code word of length k "consumes" 1/2^k of the total space of all possible bit strings. The fact that these fractions exactly match the probabilities of each instruction is not a coincidence — it's the founding insight of information theory.
The theoretical student argues that a perfect compressor must produce output that is indistinguishable from random noise — each bit independently 50/50. Why? Because if the output weren't perfectly random, it would have structure, which means it could be compressed further. A fixed-length compressed message of n bits must represent one of 2^n equally likely outcomes. This means each underlying message must have had probability exactly 1/2^n.
The payoff: If a message has probability p, then a perfect encoder allocates exactly −log₂(p) bits to it. This single expression is the foundation of all information theory.
Shannon defined the information of an event with probability p as:
I(p) = −log₂(p)
Intuitively: unlikely events carry a lot of information; near-certain events carry almost none. The key justification is not just intuition — in a perfect compression scheme, this is literally the number of bits used to encode that event.
Entropy is the expected (average) information per symbol across a distribution:
H = Σ p · (−log₂ p)
Visually, this is the total area under the information curve when probabilities are laid out as widths and their information values as heights. High entropy = high uncertainty = more bits needed to encode. Low entropy = skewed distribution = highly predictable = compressible. This is Shannon's Noiseless Coding Theorem: entropy gives the exact lower bound on bits per symbol for any lossless encoding, and it is always achievable.
For natural language, entropy is far more complex: each new letter's probability is heavily conditioned on all preceding context, probabilities are never clean powers of 2, and there is no closed-form formula for "the probability distribution of English."
Shannon's elegant response was to use human intelligence as a black box model of language. He asked his wife Betty to guess letters one at a time from a book, recording the number of guesses required per letter. Later, in his 1950 paper Prediction and Entropy of Printed English, he formalized this into a measurement. With at least 100 characters of context, his human guessers implied an entropy of roughly ~1 bit per character — meaning English is so predictable that, in principle, each letter requires only a single yes/no decision to encode.
The profound implication: to rigorously measure the compressibility of language, Shannon found himself necessarily interrogating intelligent models of it. Today, we don't interrogate those models — we build them.
One of the most striking claims in the video: compression and prediction are mathematically equivalent. The pre-training objective of large language models — minimizing cross-entropy loss on next-token prediction — is, viewed through an information-theoretic lens, equivalent to building the most efficient possible compressor of text. This reframing, the video argues, gives a more precise and illuminating answer to the question "what is cross-entropy, and why do we use it?"
This is Part 1 of a trilogy. Part 2 covers cross-entropy and KL divergence, including why they appear in LLM training and how model distillation works. Part 3 demonstrates a concrete compression algorithm that, given a high-quality language model, can approach Shannon's ~1 bit/character limit in practice.
The video is a masterclass in the 3Blue1Brown style: motivated definitions, visual intuition, historical grounding (Shannon's actual experiments), and a throughline connecting 1940s mathematics to 2020s AI.
what ai sidequests should i go on after watching this video, im very proficient in llm usage, made a compressioon project earlier in the year where i used rl to get models to communicate with less tokens, whats next?
Given your background — proficient LLM usage, already shipped a compression/RL project — here's how I'd think about the natural next frontiers that branch from this video:
The video promises a Part 3 algorithm that approaches Shannon's ~1 bit/char limit using a language model. Don't wait — implement it yourself. Arithmetic coding is the right substrate: instead of assigning discrete code words, it carves up the [0,1) interval proportionally to probabilities. If you plug in an LLM's next-token probabilities as the distribution at each step, you get a compressor that is theoretically optimal relative to that model's beliefs. The gap between your compression ratio and Shannon's limit becomes a direct, empirical measure of your model's quality.
Why it's interesting for you specifically: Your RL token-compression project was optimizing communication efficiency behaviorally. This would let you measure it information-theoretically — you could compute the actual bits-per-token of your RL agents' compressed representations and compare to entropy baselines.
Mentioned briefly in the video — the fact that gzip can "recover structure between distinct languages." This refers to the NCD (Normalized Compression Distance): NCD(x,y) = (C(xy) - min(C(x),C(y))) / max(C(x),C(y)), where C is compressed size. If two strings compress together much better than separately, they share structure.
This spawned a controversial 2023 paper ("Low-Resource" Text Classification: A Parameter-Free Classification Method with Compressors) that claimed gzip + k-NN beats fine-tuned BERT on some NLP benchmarks. The paper was later contested, but the underlying idea is legitimate and deep: compression distance is a model-free similarity metric grounded in Kolmogorov complexity.
Sidequest: implement NCD, reproduce the paper, find where it holds and where it breaks down. The failure modes are themselves instructive.
Cross-entropy loss on held-out text is literally a measurement of the model's per-token surprise — i.e., how many bits it would use to encode that text. This means you can:
This connects directly to the KL divergence framing in Part 2: the cross-entropy of model Q on text generated by distribution P is H(P) + D_KL(P||Q). The KL term is exactly how much worse your model is than optimal.
The video teases this but defers it to the trilogy. The serious version of "compression is intelligence" is rooted in Kolmogorov complexity — the length of the shortest program that outputs a string. It's uncomputable, but it's the theoretical limit beneath Shannon entropy (which only measures compressibility relative to a distribution).
Solomonoff induction formalizes Occam's razor using it: the best prediction for the next symbol is a weighted average over all programs consistent with what you've seen, weighted by 2^(−program length). This is the theoretical upper bound on prediction, and LLMs are a computable approximation of it.
The sidequest: read Hutter's Universal Artificial Intelligence or at least the first few chapters. It reframes AGI as the limit of optimal compression. Heavy going but directly relevant to what you've been building.
Your compression project sounds like it was in the emergent communication space. The formal theory it was implicitly brushing against is rate-distortion theory — Shannon's extension of compression to lossy settings. It asks: given that you're willing to tolerate distortion D, what's the minimum bit rate R needed?
Applied to your setting: instead of asking "can the model communicate with fewer tokens," you ask "what is the minimum information (in bits) needed to solve the task, as a function of acceptable performance degradation?" This gives you a theoretically grounded curve (the rate-distortion curve) rather than a single operating point, and lets you know how close to optimal your RL agents actually got.
The through-line across all of these is the same: a model's quality is equivalent to its compression ratio, and intelligence is the limit of that compression. Your RL project was already living in this space — these sidequests give it a more rigorous theoretical frame.
explain this further Shannon's elegant response was to use human intelligence as a black box model of language. He asked his wife Betty to guess letters one at a time from a book, recording the number of guesses required per letter. Later, in his 1950 paper Prediction and Entropy of Printed English, he formalized this into a measurement. With at least 100 characters of context, his human guessers implied an entropy of roughly ~1 bit per character — meaning English is so predictable that, in principle, each letter requires only a single yes/no decision to encode
Shannon wanted to measure the entropy of English — the average information per character, i.e. the fundamental compression limit. But to compute entropy you need probabilities, and the probability of the next letter in English depends on all preceding context. There's no formula for this. You can't look at n-gram statistics from books because:
He needed something that could model arbitrary-length context. In the 1940s, that meant a human brain.
Shannon opened a book to a random passage and revealed it to his subject one letter at a time. At each position, before revealing the next letter, he asked: guess the next character.
If the subject was wrong, Shannon told them the correct letter and moved on. If they were right on the first guess, great. But here's the key refinement in the 1950 paper: he didn't just record right/wrong. He recorded how many guesses it took — 1st guess, 2nd guess, 3rd guess, etc.
So the raw data looks something like:
Context: "the cat sat on the m"
Guesses before correct: 1 → subject immediately guessed 'a' (mat)
Context: "the cat sat on the ma"
Guesses before correct: 3 → tried 't', 'n', then 'p' (map)
This is the clever part. If a subject needed k guesses to get the right letter, what does that imply about the probability they mentally assigned to the correct letter?
Shannon's reasoning: assume the subject guesses in descending order of their internal probability ranking. So:
From that ranking, you can derive an upper bound on the probability they assigned to each letter. If the correct letter was ranked k-th, then at least k−1 letters had higher probability, which means the correct letter's probability was at most 1/k (roughly). Shannon worked out the precise bounds more carefully, but this is the core logic.
This gives you an implied probability for each letter, and from that you can compute −log₂(p) — the information content — for each letter. Average that over many letters across many subjects, and you get an empirical estimate of entropy.
The result: with ~100 characters of preceding context, human guessers implied ~1 bit per character.
To feel how wild this is — there are 27 possible next characters (26 letters + space). A uniform distribution over 27 symbols would give entropy of log₂(27) ≈ 4.75 bits per character. Even a well-trained character-level n-gram model might get down to 2-3 bits. Shannon's human subjects implied 1 bit.
What 1 bit means concretely: on average, the correct next letter was in the top 2 of the subject's mental ranking. English text, given enough context, is so constrained that half the time you can name the next letter outright, and most of the remaining time your second guess covers it.
Think about it experientially: "the president will address the nati___" — you already know it's 'o'. Or: "she opened the door and walked ins___" — it's 'i'. Context makes English heavily, heavily redundant.
Shannon wasn't measuring a property of text. He was measuring a property of the interaction between text and an intelligent model of it. The entropy of language isn't just in the symbols — it's relative to a predictor. A random predictor sees English as high-entropy. A perfect model would see it as almost zero entropy.
This is why he couldn't just run the experiment on a corpus. The corpus has no understanding. The subjects had genuine linguistic competence — they understood grammar, semantics, pragmatics, world knowledge. They were, in effect, a learned model of the distribution of English.
The modern restatement: a language model's cross-entropy on text is a direct measurement of the entropy of that text as seen by that model. The gap between GPT-4's cross-entropy on English and Shannon's ~1 bit/char bound is literally a measure of how much room for improvement remains in LLM language modeling. Shannon measured the target. We're still building toward it.
what about finetuning a model to get better at this guessing game directly, or is that what llm architectures already are?
This is a great question and the answer is: sort of, but not quite, and the gap is interesting.
Standard pretraining does minimize cross-entropy on next-token prediction, which is mathematically the same objective as Shannon's guessing game — minimize the average number of bits needed to encode the next token. So in that sense yes, LLMs are already trained to be optimal predictors, and optimal prediction = optimal compression.
But there are some important ways the analogy breaks down:
Tokens ≠ Characters LLMs predict over a vocabulary of ~50k-100k tokens (BPE subwords), not individual characters. Shannon's experiment was character-level. The information-theoretic content is the same in principle, but the granularity matters — a model that predicts tokens well might still be surprised at the character level in ways a human guesser wouldn't be, because humans have a richer sub-word understanding of morphology.
The model never explicitly ranks guesses Shannon's setup required subjects to enumerate guesses in probability order until correct. LLMs produce a full probability distribution over the vocabulary, which is actually more information than a ranked list — but the training signal (cross-entropy loss) only uses the probability assigned to the correct token. It doesn't care whether the model's 2nd-ranked prediction was reasonable or absurd.
Here's where it gets interesting. There are a few distinct things you could mean:
1. Finetuning on character-level prediction Train or finetune a model to predict the next character rather than token, given long context. This would more directly replicate Shannon's experiment and might expose failure modes that token-level training hides. A model could be excellent at token-level entropy but still have miscalibrated character-level probabilities within tokens it's confident about.
2. Finetuning on the interactive guessing format specifically Train a model to explicitly output an ordered ranked list of guesses, then receive feedback (correct/incorrect), then update within the same context window. This is subtly different from standard next-token prediction — it's training the model to be well-calibrated on its ranking, not just its top-1 probability. A model could have good cross-entropy loss while being poorly calibrated in its rank-ordering of lower-probability options.
This is closer to what RLHF and direct preference optimization do in spirit — they shape the relative ordering of outputs, not just the raw probabilities. But they do it for human preference, not for prediction accuracy.
3. Training with explicit "guess budget" pressure Your RL background makes this one natural: reward the model for getting the right answer in the fewest guesses, penalize extra guesses. This directly optimizes the Shannon guessing-game metric. The interesting question is whether this would converge to the same solution as cross-entropy pretraining, or whether the RL pressure would shape the internal representations differently.
The theoretical answer is that they should converge to the same optimum — a model that minimizes expected guesses is the same as one that maximizes probability on the correct next token. But in practice, the training dynamics and the inductive biases of the optimization path differ, which might matter.
Shannon's subjects were measuring something closer to the true entropy of English because they had:
Current LLMs are actually pretty good at all of these now. But there's one thing Shannon's subjects had that models still arguably lack:
Knowing what they don't know.
A human subject who encountered an unusual proper noun or a technical term they'd never seen would recognize the uncertainty and spread their probability mass appropriately. LLMs are notoriously miscalibrated in this way — they often assign high confidence to wrong predictions in out-of-distribution contexts.
Finetuning a model specifically on the guessing game format wouldn't dramatically change what it knows — it already implicitly learned to predict next tokens from pretraining. What it might change:
That third one is the most novel and connects directly back to your RL work. A model that receives "wrong, try again" signals and adapts within context rather than relying purely on pretraining is essentially doing online Bayesian updating. That's not what transformers are explicitly trained for, and it's a real open problem — sometimes called in-context learning calibration or framed as a test-time compute question.
The Shannon experiment is, in that sense, a surprisingly clean benchmark for a capability that modern LLMs are still developing.
how can i hide hidden folders or files from mac finder, like keep them but they just dont show upo int he finder ui
Two main approaches:
1. Dot-prefix (simplest)
Any file or folder starting with . is hidden from Finder by default. Just rename it:
mv myfolder .myfolder
2. chflags hidden (keeps the original name)
chflags hidden /path/to/file_or_folder
This sets the macOS hidden flag — file keeps its name, just disappears from Finder. Reverse it with:
chflags nohidden /path/to/file_or_folder
To verify either worked, toggle hidden file visibility in Finder with Cmd+Shift+. — hidden items will appear greyed out when that's on.
2 is usually the better pick if you want to keep a clean name (e.g. hiding a node_modules or a config folder without renaming it).
i mean like existing hidden folders and files
You want to make sure hidden files/folders stay hidden and don't show up — the toggle is likely currently on.
In Finder, press Cmd + Shift + . to toggle hidden file visibility off. That's it — they'll disappear from Finder but still exist on disk.
To make sure it stays off across sessions:
defaults write com.apple.finder AppleShowAllFiles -bool false
killall Finder