llm-synesthesia: research project for Hermes Agent

Pipeline that reads emotions from Hermes-4.3-36B hidden states and renders them as a live WebGL visual

What llm-synesthesia does

Hermes-4.3-36B generates each next token on Modal. Layer-48 hidden states are smoothed with an exponential moving average and passed through an emotion MLP probe that predicts a distribution over nine emotions: anger, joy, sadness, fear, curiosity, confidence, confusion, disgust and tenderness. The probabilities are mapped to 2D anchors, and logit and hidden-state statistics become renderer controls such as rotation, branching, glow, density, diffusion and gate. These values drive color and velocity splats in a WebGL2 flow field.

You train the probe and baseline statistics on Modal, deploy the inference endpoint, and export its URL as SYNESTHESIA_MODAL_URL. Hermes Agent is optional: a provider wrapper and a YAML snippet let it talk to the endpoint, and a bridge converts Hermes or OpenAI-style streaming output into WebSocket parameters. Run modes cover the full Hermes flow, the bridge and renderer alone, and a mock mode for the renderer.

Key features

Emotion probe over layer-48 hidden states with nine emotion classes
WebGL2 flow-field renderer driven by the probe output
Modal training and inference scripts
Hermes Agent provider wrapper and config snippet
Bridge from streaming output to WebSocket parameters
Mock mode for testing the renderer alone

When to use it

Visualizing how a Hermes model's internal state changes during a reply
Experimenting with probes on hidden states
Running the renderer without any model using mock data

Who it is for: Researchers and creative coders interested in model internals and visualization.

How it fits with Hermes Agent

It is built around Nous Research's Hermes-4.3-36B model and can be used through Hermes Agent, but the README marks Hermes Agent as optional.

How to install llm-synesthesia

These commands are copied from the project's README. Check the repository for the latest steps before you run them.

uv sync --group dev --group training
modal setup
uv run modal run training/emotion_probe.py
uv run modal run training/head.py

Requirements: Python 3.12+, uv, a Modal account for training and serving, and a WebGL2-capable browser; Hermes Agent is optional

Note: Training and serving the probe require a Modal account, and the repository has no license file.

FAQ

What is llm-synesthesia?

It is a pipeline that extracts an emotional state from an LLM's hidden states and visualizes it as an animated WebGL fluid field.

Does llm-synesthesia work with Hermes Agent?

Yes, optionally. A provider wrapper and config snippet let Hermes Agent talk to the Modal endpoint, and the project can also run just the bridge and renderer without Hermes Agent.

What do I need to run llm-synesthesia?

You need Python 3.12 or newer, uv, a Modal account to train and serve the probe, and a browser with WebGL2.