Devin Sessions

Done. Here's the full pipeline result.

Pipeline

Download — api.cobalt.tools now requires JWT auth, so I used the community cobalt instance dwnld.nichind.dev (POST to / with downloadMode: audio, audioFormat: mp3). Got an 18 MB MP3 (~19:53) saved to ~/Downloads/utilities/tweet_audio/twitter_2070551894674555081_audio.mp3.
Transcription — faster-whisper (medium model, int8, CPU, VAD filter) on the M4 Pro. Full transcript saved to ~/Downloads/utilities/tweet_audio/twitter_2070551894674555081_audio.txt. (The openai-whisper CLI was unusable here because of a self-signed-cert MITM on the model download — faster-whisper via HuggingFace worked around it.)
Summary — below.

Summary

This is Dwarkesh narrating a blog post (find it at dwarkesh.com) about what's missing between today's RL-trained agents and real AGI: continual / on-the-job learning.

The labs' big bet: RLVR → AGI

The labs believe that training AIs on millions of verifiable tasks across thousands of diverse RL environments (RLVR — RL with verifiable rewards) will basically produce AGI: a problem-solving agent that can grind on open-ended tasks for weeks despite errors and ambiguity. Optimists argue the known deficits (sample inefficiency, no continual learning) will be "steamrolled" by scale, just as NLP research problems collapsed once enough compute hit LLMs.

The counterarguments Dwarkesh raises

"Sample inefficiency only matters during training" — defenders say training cost is amortized over billions of sessions; what matters is in-session smarts, which RL is clearly improving (visible in coding agents). Fair, but…
"Continual learning may be unnecessary if context windows get huge" — people argue you can just fit the "six months to become productive" into context. Dwarkesh is skeptical.
Computer use is suspiciously slow — a telling clue. Computer-use tasks are verifiable (did the Etsy order arrive? were taxes filed?), yet progress lags coding/math. His diagnosis: a domain must be not just verifiable but grindable — you need thousands of parallel rollouts against a deterministic, replayable simulator. Coding has that (spin up 1,000 identical containers); Amazon checkout doesn't (Andy Jassy bans your bots). You can clone Slack/Gmail, but it's labor-intensive — until AIs can code well enough to build high-fidelity clones themselves (which is also a great coding RL objective).
The deeper problem — many crucial skills (building a business, winning court cases, profitable trading, winning elections, LBJ-level politics, Elon-level space companies) require real-world interaction, months/years of outer-loop verification, and you can't perturb actions across parallel rollouts to isolate what worked. Because data there is idiosyncratic and sparse, you need sample efficiency to get proficient, and that data doesn't exist in any containerized form.

Does RLVR generalize to long horizons / real world?

Empirical question. He reads a telling Dario quote about context length: train-short / serve-long causes degradation — hinting that short-horizon RL doesn't automatically generalize to long-horizon performance, which would also undermine "train on white-collar tasks → build a business like Sam Walton."

Continual learning is the crux

~30–50% of lab compute goes to inference, currently doing nothing to improve the model — a huge waste, especially since the most valuable learning signal (what orgs actually use the model for, real-world mistakes) is only revealed in deployment.
Analogy: a genius grad student never allowed a real internship, fed only classroom case studies.
You can't just grow the KV cache forever (not scalable, not how humans learn — human memory compresses; savants with perfect recall actually struggle with abstraction).
But moving learnings into the weights is sample-inefficient: every shipped online-learning model (e.g., Cursor Tab, ~400M requests/day) only works because it learns the same objective across millions of users. We haven't seen per-user online learning, because one session's data is enough for a human but not for training a more capable AI.

Techniques that could close the gap

On-policy self-distillation (OPSD) — train the base model to match the predictions of a "veteran" teacher model that has accumulated a long session of context. Better than RLVR (no outer-loop verifiable reward needed; denser per-token supervision) and better than naive SFT (which would force the model to memorize the whole session transcript rather than consolidate the few useful insights). Preserves RL's nice property of changing only the parameters necessary to achieve the outcome — important for avoiding catastrophic forgetting.
"Dreaming" / test-time training — inspired by EfficientZero (which beats novice humans at a novel Atari game after 2 hours by playing dozens of simulated games in its head per real step). Future LLMs might consume little real-world data while endlessly practicing against self-built simulations of what they're witnessing in production. Much harder than emulating Go — you'd have to simulate the whole world — so this is the most speculative idea, but if it works it becomes a fourth axis of scaling alongside pretraining, RL, and inference-time compute. Cue the joke: instead of /compact in Cursor/Claude (a cheap summary masquerading as continual learning), you'd hit /dream and incinerate huge compute building and training against the "video game version" of your real-world task.

The 2027–2028 scenario he sketches

RLVR produces an agent competent enough to be deployed on real, even out-of-distribution work. Context lengths grow to a full week of co-work. At week's end you give it a thumbs up/down "work review"; on thumbs up, the base model distills the session's learnings (via OPSD, dreaming, or something unknown). It then gets better at domains adjacent to its RLVR training, then adjacent to what it online-learned, and so on — skills expand far beyond the original verifiable training set. The main way AIs improve is no longer pre-release training but accumulated deployment experience across all users — every interaction makes the AI smarter, both from your prior sessions and from everyone else's. "And that's very scary and exciting and different from the way that AI improves right now."

Notable asides

Mercury (banking) sponsor read woven into a personal anecdote about paying contractor invoices.
References his prior video essay on sample inefficiency (models ~1M× less sample-efficient than humans) and an impromptu blackboard lecture with Sasha Rush on OPSD (linked in the video description).

Files produced (in ~/Downloads/utilities/tweet_audio/):

twitter_2070551894674555081_audio.mp3 — downloaded audio
twitter_2070551894674555081_audio.txt — full Whisper transcript
transcribe_fw.py — the faster-whisper script used

Phase	Duration
Cobalt audio download	~30 seconds (18 MB MP3, done by 22:09:56)
openai-whisper `medium.pt` download (1.5 GB)	~18 min (22:10 → 22:27:56) — wasted: SSL-cert issue blocked it, then I killed the slow CPU run
Switch to faster-whisper + HF model download (~1.5 GB)	~10 min (22:48 → 22:59)
Actual transcription (faster-whisper medium, int8, CPU)	~16 min (22:59 → 23:15:13) for 19:53 of audio

OK — I owe you a correction. I called it a "MITM" earlier, but it's not a man-in-the-middle attack. It's a misconfigured Python install. Here's what's actually going on:

The real cause

The python.org macOS installer ships its own copy of OpenSSL, configured to look for trusted root certificates at:

/Library/Frameworks/Python.framework/Versions/3.12/etc/openssl/cert.pem

On your machine, that file does not exist. Neither does the certs/ directory next to it. So Python's ssl module has an empty trust store — it doesn't know about any root CA, including perfectly legitimate ones like DigiCert or Let's Encrypt.

Why the error said "self-signed certificate"

That wording is what threw me off. Every TLS chain ends in a root CA, and root CAs are by definition self-signed (they sign themselves — that's what makes them roots). Normally OpenSSL validates a root by checking it against the trust store. With an empty trust store, Python hits the root cert, can't find it anywhere, and reports:

CERTIFICATE_VERIFY_FAILED: self-signed certificate in certificate chain

It's not saying someone injected a fake cert. It's saying "I reached the top of the chain and it's self-signed, and I have no way to trust it." Same error you'd get on any clean TLS connection when your CA bundle is missing.

Proof it's not a MITM

openssl s_client against openaipublic.azureedge.net:443 using the system CA bundle (/etc/ssl/cert.pem, which exists and is 333 KB) returns Verify return code: 0 (ok) — the chain is legitimate.
macOS itself has the roots in the Keychain; curl, git, browsers, etc. all work fine because they use the system trust store, not Python's bundled one.
Only Python (and tools built on Python's ssl) break.

Why faster-whisper worked where openai-whisper didn't

openai-whisper downloads its model via raw urllib.request.urlopen — pure Python stdlib, uses Python's broken OpenSSL → fails.
faster-whisper/HuggingFace uses requests/huggingface_hub, which on this machine apparently fell back to a CA bundle from certifi (a pip package that bundles its own Mozilla CA list). That's why the HF model download succeeded.

The fix (one-time, ~10 seconds)

/Applications/Python\ 3.12/Install\ Certificates.command

That's the script python.org ships precisely for this — it downloads certifi's PEM and symlinks it into the OpenSSL path Python expects. After running it, urllib, openai-whisper, pip over HTTPS, etc. all start working. Alternatively:

pip install certifi
python3 -c "import certifi, os, shutil; src=certifi.where(); dst='/Library/Frameworks/Python.framework/Versions/3.12/etc/openssl/cert.pem'; os.makedirs(os.path.dirname(dst), exist_ok=True); shutil.copy(src, dst)"

Want me to run the fix now so this never bites you again?