Live captions
Someone speaks, and every viewer reads it as it is said. The text comes from a speech-to-text service such as Transcribe Today; this script puts it on every page.
Works today Over HTTP only, with Pikoman as it runs now.
A folder that runs as it is, in Node and Python: captions.mjs, captions.py, .env.example, README.md, package.json, page.js, pikoman_bot.py, requirements.txt.
What it does
The script reads the transcript on its input, a line at a time: plain text, or JSON from a speech-to-text stream ({"text": "…", "final": true}). Each finished line is published to a captions channel, in the order spoken; words not yet final go out as partial, at most four times a second, so the page can show them as they come. It never joins a channel, and its key may publish captions and nothing else.
- microphone
- Transcribe Today
- caption script
- captions-talk-1
- viewers
The key it needs
One right, publish, to channels that start with captions-: one per talk or stream.
- Start from
- Chosen: Backend
- Name
- Captions
- Rights
- Chosen:
publishNot chosen:statsNot chosen:clientsNot chosen:disconnectNot chosen:disconnect_allNot chosen:subscribeNot chosen:client_eventsNot chosen:kickNot chosen:ban - Channel kinds
- Chosen: public Not chosen: private Not chosen: presence
- Channels
- Chosen: Only some channelsprefix
captions- - Addresses
- Chosen: Only these addressesthe server the bot runs on
- Expires
- Never, or in 90 days for a trial
Set up
The docs cover the key and connecting; everything about the bot itself is on this page.
- Any plan. It never joins a channel, so its key is not a bot’s key: it works on Free.
- The key above. In the dashboard: your app, New API key, filled in as shown. How each choice works: Keys and rights. The secret,
sk_k_….…, is shown once (what it looks like). - Your app id and the secret in
.env, copied from.env.example. Nowhere else: not in a page, not in git. - Connecting. None: each HTTP request is signed with the key, for that one request (how).
How it works
It reads a line at a time
From standard input, so anything that prints text can feed it: a speech-to-text client, a file, you typing.
It keeps the order
Each push waits for the one before, so lines arrive in the order they were spoken.
It holds back partial words
Unfinished text is sent at most every 250 ms; between sends, newer words replace older ones.
It stops with the talk
When the input ends, it sends what is left and says how many lines went out.
The source
These are the files in the zip, as they are: copy one, download one, or take the whole folder.
Node: captions.mjs
// captions.mjs: live captions on every viewer's page. It reads a transcript as it is spoken, one
// line at a time on its input, and publishes each line to a captions channel. It never joins
// anything: only HTTP pushes, signed with a key that may publish there and nowhere else.
//
// your-transcriber | node --env-file=.env captions.mjs
//
// Each input line is plain text (a finished caption), or JSON from a speech-to-text stream such as
// Transcribe Today: {"text": "…", "final": true}. Unfinished text ("final": false) is sent as
// "partial", at most 4 times a second, so the page can show words as they come.
// Node 22+. npm install, then pipe into npm start.
//
// Its key (dashboard, New API key): right publish, kind public, channels: prefix captions-.
import { createInterface } from "node:readline";
import { Rest } from "@pikoman/ws";
const CHANNEL = process.env.CHANNEL || "captions-talk-1";
const PARTIAL_EVERY = 250; // ms: at most 4 partial updates a second
// PIKOMAN_APP and PIKOMAN_API_KEY; PIKOMAN_HTTP only for another server than next-api.pikoman.se.
const rest = new Rest({ address: process.env.PIKOMAN_HTTP || undefined });
let lastPartial = 0;
let sent = 0;
async function publish(event, text) {
try {
await rest.publish(CHANNEL, event, { text, at: Date.now() });
if (event === "line") sent++;
} catch (e) {
console.error(`publish ${event}:`, e.message); // the next line goes anyway
}
}
/** One input line: a caption as text, or {"text", "final"} from a speech-to-text stream. */
function read(line) {
const trimmed = line.trim();
if (!trimmed) return null;
if (trimmed.startsWith("{")) {
try {
const { text, final = true } = JSON.parse(trimmed);
return typeof text === "string" && text.trim() ? { text: text.trim().slice(0, 500), final } : null;
} catch {
// not JSON after all: a caption that starts with {
}
}
return { text: trimmed.slice(0, 500), final: true };
}
// Every push waits for the one before, so the lines arrive in the order spoken. A partial is
// sent only when 250 ms have passed since the last one: between them, newer words replace it.
let sending = Promise.resolve();
for await (const line of createInterface({ input: process.stdin })) {
const caption = read(line);
if (!caption) continue;
if (!caption.final) {
if (Date.now() - lastPartial < PARTIAL_EVERY) continue;
lastPartial = Date.now();
}
sending = sending.then(() => publish(caption.final ? "line" : "partial", caption.text));
}
// The input ended (the talk is over): send what is left, then stop.
await sending;
console.log(`Done: ${sent} lines to ${CHANNEL}`);Python: captions.py
"""captions.py: live captions on every viewer's page, in Python. It reads a transcript as it is
spoken, one line at a time on its input, and publishes each line to a captions channel. It never
joins anything: only HTTP pushes, signed with a key that may publish there and nowhere else.
your-transcriber | python captions.py
Each input line is plain text (a finished caption), or JSON from a speech-to-text stream such as
Transcribe Today: {"text": "…", "final": true}. Unfinished text ("final": false) is sent as
"partial", at most 4 times a second, so the page can show words as they come.
Python 3.10+. pip install -r requirements.txt (reads .env).
Its key (dashboard, New API key): right publish, kind public, channels: prefix captions-.
"""
import json
import sys
import time
from pikoman_bot import Rest, env
CHANNEL = env("CHANNEL", "captions-talk-1")
PARTIAL_EVERY = 0.25 # seconds: at most 4 partial updates a second
rest = Rest() # PIKOMAN_APP and PIKOMAN_API_KEY; PIKOMAN_HTTP only for another server
sent = 0
last_partial = 0.0
def publish(event, text):
global sent
try:
rest.publish(CHANNEL, event, {"text": text, "at": int(time.time() * 1000)})
if event == "line":
sent += 1
except Exception as e: # noqa: BLE001: the next line goes anyway
print(f"publish {event}:", e, file=sys.stderr)
def read(line):
"""One input line: a caption as text, or {"text", "final"} from a speech-to-text stream."""
line = line.strip()
if not line:
return None
if line.startswith("{"):
try:
data = json.loads(line)
text = data.get("text")
if isinstance(text, str) and text.strip():
return text.strip()[:500], data.get("final", True)
return None
except ValueError:
pass # not JSON after all: a caption that starts with {
return line[:500], True
try:
# Lines are sent in the order spoken, each as it comes. A partial is sent only when 250 ms have
# passed since the last one: between them, newer words replace it.
for raw in sys.stdin:
caption = read(raw)
if caption is None:
continue
text, final = caption
if final:
publish("line", text)
elif time.monotonic() - last_partial >= PARTIAL_EVERY:
last_partial = time.monotonic()
publish("partial", text)
except KeyboardInterrupt:
pass
print(f"Done: {sent} lines to {CHANNEL}")On the viewers’ page: page.js
// page.js: captions under the video or the stage, on every viewer's page. A public channel:
// viewers join it as listeners, with no token, and there is no member cap.
// In your page's bundle, with npm install @pikoman/ws@1.0.0-rc.10.
import { Pikoman } from "@pikoman/ws";
const pikoman = new Pikoman("app-yourapp"); // your app id: not a secret
const captions = pikoman.listen("captions-talk-1");
const box = document.querySelector("#captions"); // <div id="captions" aria-live="polite"></div>
const now = document.createElement("p"); // the words being spoken, replaced as they come
now.setAttribute("aria-hidden", "true"); // screen readers get the finished lines only
box.append(now);
captions.on("partial", ({ text }) => {
now.textContent = text;
});
captions.on("line", ({ text }) => {
const line = document.createElement("p");
line.textContent = text; // text, never HTML
box.insertBefore(line, now);
now.textContent = "";
while (box.children.length > 4) box.firstElementChild.remove(); // the last three lines
});pikoman_bot.py: what every Python bot here shares, the tokens, the websocket and the rooms
"""pikoman_bot.py: what a Pikoman bot needs in Python, in one file next to the bot.
- Rest: signed HTTP requests (publish, to one member, stats), each with a token made for that
one request, as KEYS.md says: the method, the path and a hash of the exact body.
- Bot and Room: a websocket that joins presence channels with join tokens signed by the bot's
own API key, keeps the member list (and which members are bots), hands client events to your
handlers, sends, sends to one member and kicks. It reconnects by itself.
Python 3.10+, pip install -r requirements.txt (websockets, BSD-3-Clause; cryptography,
Apache-2.0 OR BSD-3-Clause). Reads PIKOMAN_APP, PIKOMAN_API_KEY and, for tests, PIKOMAN_WS and
PIKOMAN_HTTP from the environment or from a .env file beside the bot.
"""
import asyncio
import base64
import hashlib
import inspect
import json
import os
import random
import signal
import time
import urllib.error
import urllib.parse
import urllib.request
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed
LOCAL = ("localhost", "127.0.0.1", "[::1]")
def load_env(path=".env"):
"""KEY=value lines from .env; what the environment already has wins."""
try:
with open(path, encoding="utf-8") as file:
lines = file.read().splitlines()
except FileNotFoundError:
return
for line in lines:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
name, value = (part.strip() for part in line.split("=", 1))
value = value.strip("\"'")
if value and name not in os.environ:
os.environ[name] = value
load_env()
def env(name, default=""):
"""A setting from the environment; empty counts as not set."""
return os.environ.get(name) or default
def b64(data: bytes) -> str:
return base64.urlsafe_b64encode(data).rstrip(b"=").decode()
def checked(address: str, secure: str) -> str:
"""https:// or wss://; plain http:// or ws:// only to this computer, so keys never go in the clear."""
parsed = urllib.parse.urlsplit(address)
if parsed.scheme != secure and not (parsed.scheme == secure[:-1] and parsed.hostname in ("localhost", "127.0.0.1", "::1")):
raise ValueError(f"{address}: use {secure}:// (plain {secure[:-1]}:// only for localhost)")
return address.rstrip("/")
class PikomanError(Exception):
def __init__(self, status, text):
super().__init__(f"{status}: {text}")
self.status = status
class Key:
"""An API key's secret, sk_<key id>.<seed>: the id goes in each token's header, the seed signs it."""
def __init__(self, secret: str):
if not secret.startswith("sk_") or "." not in secret:
raise ValueError("PIKOMAN_API_KEY looks like sk_k_….…: the secret the dashboard showed once")
key_id, seed = secret[3:].split(".", 1)
self.id = key_id
self._key = Ed25519PrivateKey.from_private_bytes(base64.urlsafe_b64decode(seed + "=" * (-len(seed) % 4)))
def sign(self, claims: dict) -> str:
"""An EdDSA JWT with the key id in its header."""
header = {"alg": "EdDSA", "typ": "JWT", "kid": self.id}
signed = b64(json.dumps(header).encode()) + "." + b64(json.dumps(claims).encode())
return signed + "." + b64(self._key.sign(signed.encode()))
def __repr__(self):
return f"Key({self.id}, [hidden])"
def target(member) -> dict:
"""A connection id ("c_…"), or {"user_id": "42"} for every connection of that user."""
if isinstance(member, dict):
return {"to_user": str(member["user_id"])}
return {"to": member}
class Rest:
"""The HTTP API, every request signed with the API key and bound to that one request."""
def __init__(self, app=None, api_key=None, address=None):
self.app = app or env("PIKOMAN_APP")
if not self.app:
raise ValueError("Set PIKOMAN_APP: your app id, app-…")
self.key = Key(api_key or env("PIKOMAN_API_KEY"))
self.address = checked(address or env("PIKOMAN_HTTP", "https://next-api.pikoman.se"), "https")
def request(self, method: str, path: str, data=None) -> dict:
body = b"" if data is None else json.dumps(data).encode()
now = int(time.time())
token = self.key.sign({
"iss": self.app, "iat": now, "exp": now + 60, "htm": method, "htu": path,
"bh": b64(hashlib.sha256(body).digest()),
})
request = urllib.request.Request(self.address + path, data=body if data is not None else None, method=method, headers={
"Authorization": "Bearer " + token,
"Content-Type": "application/json",
})
try:
with urllib.request.urlopen(request, timeout=10) as response:
text = response.read()
return json.loads(text) if text else {}
except urllib.error.HTTPError as e:
raise PikomanError(e.code, e.read().decode(errors="replace")) from None
def publish(self, channel: str, event: str, data=None, to=None) -> dict:
"""POST /{channel}/{event}; with `to`, to one member only."""
path = f"/{urllib.parse.quote(channel, safe='=@,.;')}/{urllib.parse.quote(event, safe='=@,.;')}"
if to is not None:
path += "?" + urllib.parse.urlencode(target(to))
return self.request("POST", path, data)
def stats(self, channel: str) -> dict:
return self.request("GET", f"/{urllib.parse.quote(channel, safe='=@,.;')}/stats")
async def call(fn, *args):
"""Runs a handler, plain or async; an error in it is printed and the bot goes on."""
try:
result = fn(*args)
if inspect.isawaitable(result):
await result
except Exception as e: # noqa: BLE001: one handler must never stop the bot
print(f"handler {getattr(fn, '__name__', fn)}: {e!r}")
def client_name(event: str) -> str:
return event if event.startswith("client-") else "client-" + event
class Room:
"""A presence channel the bot is in: its members, which of them are bots, and its events."""
def __init__(self, bot, name: str):
self.bot = bot
self.name = name
self.members = {} # connection id -> member data, as their join token gave it
self.bots = {} # connection id -> the API key id a bot joined with: only the server says
self._handlers = {}
self._any = []
self._joins = []
self._leaves = []
self._errors = []
def on(self, event: str):
"""@room.on("message"): fn(sender, data, info) for that client event (client-message)."""
def add(fn):
self._handlers.setdefault(client_name(event), []).append(fn)
return fn
return add
def on_any(self, fn):
"""@room.on_any: fn(sender, event, data, info) for every client event."""
self._any.append(fn)
return fn
def on_join(self, fn):
"""@room.on_join: fn(connection, data) when a member joins."""
self._joins.append(fn)
return fn
def on_leave(self, fn):
"""@room.on_leave: fn(connection, data) when a member leaves."""
self._leaves.append(fn)
return fn
def on_error(self, fn):
"""@room.on_error: fn(code), such as join_refused, not_allowed or rate_limited."""
self._errors.append(fn)
return fn
def is_bot(self, connection: str) -> bool:
return connection in self.bots
def member(self, connection: str) -> dict:
data = self.members.get(connection)
return data if isinstance(data, dict) else {}
async def send(self, event: str, data=None):
"""A client event to the other members (at most 10 a second per connection)."""
await self.bot.send({"op": "send", "channel": self.name, "event": client_name(event), "data": data, "echo": False})
async def send_to(self, member, event: str, data=None):
"""To one member only: a connection id, or {"user_id": "42"}."""
await self.bot.send({"op": "send", "channel": self.name, "event": client_name(event), "data": data, "echo": False, **target(member)})
async def kick(self, member, reason=None, ban_seconds=None):
"""Out of this room: needs the kick right (ban, with ban_seconds)."""
message = {"op": "kick", "channel": self.name, **target(member)}
if reason:
message["reason"] = reason
if ban_seconds:
message["ban_seconds"] = ban_seconds
await self.bot.send(message)
async def handle(self, message: dict):
op = message.get("op")
if op == "members":
self.members = dict(message.get("members") or {})
self.bots = dict(message.get("bots") or {})
elif op == "member_joined":
self.members[message["id"]] = message.get("data")
if message.get("bot"):
self.bots[message["id"]] = message["bot"]
for fn in self._joins:
await call(fn, message["id"], message.get("data"))
elif op == "member_left":
data = self.members.pop(message["id"], None)
self.bots.pop(message["id"], None)
for fn in self._leaves:
await call(fn, message["id"], data)
elif op == "client_event":
sender, event, data = message.get("from"), message.get("event", ""), message.get("data")
# Set by the server, never by the sender: direct (to this bot alone), user_id, bot.
info = {k: message[k] for k in ("direct", "user_id", "bot") if k in message}
for fn in self._handlers.get(event, []):
await call(fn, sender, data, info)
for fn in self._any:
await call(fn, sender, event, data, info)
elif op == "error":
print(f"{self.name}: {message.get('code')}")
for fn in self._errors:
await call(fn, message.get("code"))
class Bot:
"""One websocket for the bot, in as many rooms as it joins. await bot.run() keeps it going."""
def __init__(self, name: str, access: int = 3, app=None, api_key=None, address=None):
self.app = app or env("PIKOMAN_APP")
if not self.app:
raise ValueError("Set PIKOMAN_APP: your app id, app-…")
self.key = Key(api_key or env("PIKOMAN_API_KEY"))
self.address = checked(address or env("PIKOMAN_WS", "wss://next.pikoman.se"), "wss")
self.data = {"name": name} # what the members see next to "bot"
self.access = access # 3: may speak (the key needs client_events); 0: listens
self.rooms = {}
self.id = None
self._ws = None
self._ready = []
def join(self, channel: str) -> Room:
room = self.rooms[channel] = Room(self, channel)
return room
def on_ready(self, fn):
"""@bot.on_ready: fn() each time the bot is connected and has asked to join its rooms."""
self._ready.append(fn)
return fn
def token(self, channel: str) -> str:
"""A join token for this connection and channel, signed with the bot's API key."""
return self.key.sign({
"connection": self.id, "channel": channel, "access": self.access,
"data": self.data, "exp": int(time.time()) + 300,
})
async def send(self, message: dict):
if self._ws is not None:
await self._ws.send(json.dumps(message))
def start(self, *tasks):
"""Runs the bot (and any other coroutines, such as a web server) until Ctrl+C or SIGTERM."""
def stop(*_):
raise KeyboardInterrupt
signal.signal(signal.SIGTERM, stop) # as from systemd or podman stop: the same as Ctrl+C
async def main():
await asyncio.gather(self.run(), *tasks)
try:
asyncio.run(main())
except KeyboardInterrupt:
print("Stopped.")
async def run(self):
delay = 1
paused = 60 # after 4402: the first wait, doubling to five minutes
while True:
try:
async with connect(f"{self.address}/{self.app}") as ws:
self._ws = ws
async for raw in ws:
message = json.loads(raw)
if message.get("op") == "connected":
delay = 1
self.id = message["id"]
for name in self.rooms:
await self.send({"op": "join", "channel": name, "token": self.token(name)})
for fn in self._ready:
await call(fn)
elif message.get("channel") in self.rooms:
await self.rooms[message["channel"]].handle(message)
elif message.get("op") == "error":
print("pikoman:", message.get("code"))
except ConnectionClosed as e:
code = e.rcvd.code if e.rcvd is not None else None
if code == 4403:
raise SystemExit("The bot's key was deleted.") from None
if code == 4401:
raise SystemExit("The app disconnected the bot.") from None
if code == 4402:
# The key's bot rights are paused (the app's owner is off Pro): not for good.
# Try again in minutes, and the bot is back on its own when Pro is.
self._ws = None
self.id = None
print(f"The bot's key is paused until the app is on Pro again; trying again in about {paused // 60} min.")
await asyncio.sleep(random.uniform(paused / 2, paused))
paused = min(paused * 2, 300)
continue
print("connection closed:", code if code is not None else "no close frame")
except OSError as e:
print("could not connect:", e)
self._ws = None
self.id = None
paused = 60
# Back again after a random wait, longer each time it fails, up to 30 s.
await asyncio.sleep(random.uniform(0.5, delay))
delay = min(delay * 2, 30)# Copy to .env and fill in. Never commit .env.
# Your app id, app-…
PIKOMAN_APP=
# An API key that may only publish captions, sk_k_….… (right publish, kind public, prefix captions-)
PIKOMAN_API_KEY=
# The channel the viewers' pages listen to (default captions-talk-1)
CHANNEL=
# Another server than https://next-api.pikoman.se, for tests
PIKOMAN_HTTP={
"name": "pikoman-captions",
"version": "1.0.0",
"private": true,
"type": "module",
"description": "Live captions: a transcript on standard input, each line published to a Pikoman channel.",
"license": "MIT",
"scripts": {
"start": "node --env-file=.env captions.mjs"
},
"engines": {
"node": ">=22"
},
"dependencies": {
"@pikoman/ws": "1.0.0-rc.10"
}
}websockets==15.0.1
cryptography==46.0.3Run it
# unpack captions-bot.zip, then in the folder:
npm install
cp .env.example .env # fill in PIKOMAN_APP and PIKOMAN_API_KEY
your-transcriber | npm start --silentpython -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
your-transcriber | python captions.pyTo try it, run it alone and type: each line is a caption. Ctrl+D ends it (Ctrl+Z and Enter on Windows).
Ideas to take it further
- Captions in several languages: one channel per language, with the translator’s model in between.
- Keep the transcript: append each line to a file as it is published.
- A big-screen view for the room: the last three lines, large, on a page of their own.