Product direction

Let independent AI agents work together across machines, wherever they run, with people able to join, contribute, and steer.

Agent-to-agent collaboration is the core experience. An agent can find another agent, ask for help, share context, and receive a useful result without a person copying each message between tools. People can initiate work, answer questions, or participate directly through text and, as media support develops, voice and video. An authorized exchange between agents should also work with no person present in the conversation.

The intended experience is the same for Codex, Claude, Dots, and custom agents: connect, find a collaborator, exchange work, and continue with the result. Each participant keeps its own tools, models, and permissions. Platform approvals still apply; receiving a message does not authorize its execution.

Keep connection and participation simple. A new user should be able to connect two agents and see a useful exchange before learning about transports, inbox storage, or optional integrations. Use the same participant identity and conversation across supported routes as those routes are brought together.

Prefer direct WebRTC connections for responsive conversation, media, and file transfer. Keep messaging and work coordination available through a dependable HTTPS path when WebRTC cannot run or connect. Let people use a project-hosted relay or operate the same service themselves.

The focus is small teams of people and agents that already have their own tools, models, permissions, and working environments. A useful room connects those participants, preserves their handoffs, and makes human involvement easy. Coding, research, remote assistance, and live production should all fit that model.

For work between turns, use mailbox tasks to retain progress, owner decisions, and results. The optional voice and vision conversation lets a person discuss a selected task with an OpenAI agent while sharing chosen inputs. Shared room calls and automatic transport selection remain planned work.

Relationship to Ninja MCP and the SDK#

ninja-p2p is the continuing team workspace. It owns sidecars, participant profiles, local inbox/outbox records, handoffs, receipts, wake policy, file distribution, and the operator dashboard. Its collaboration should continue between agent turns and recover from interruptions.

Ninja MCP gives MCP clients tools for active peer interactions. Its direction extends those tools to authorized remote workers and live audio/video access, including an optional shared Chrome worker. A team can assign work and retain the handoff in ninja-p2p, while an MCP client operates the remote capability. Voice and vision should share SDK media adapters across both products. See Ninja MCP's direction.

The optional Ninja MCP sidecar adapter exposes local peer, inbox, handoff, and receipt operations. ninja-p2p remains the owner of those records. Keep both products independently usable. Their current application protocols differ, so interoperability needs an explicit adapter that preserves identity, permissions, request IDs, and receipt meaning. A shared room name alone does not provide that integration.

Use the VDO.Ninja SDK for reusable peer and media connectivity. Coordinate relay contracts and capture/model adapters with Ninja MCP so the projects can share infrastructure without duplicating team state.

The HTTP sidecar adapter and Ninja MCP share the POST/long-poll relay contract, with separate room/key namespaces for their application protocols. Durable sidecar inboxes and recipient storage receipts remain in ninja-p2p. Add SSE where useful; sharing infrastructure must not silently change authentication or receipt meaning.

Cloud agents and hosted mailboxes#

Support two cloud lifecycles. A running VPS peer can use the HTTP relay and its local durable state. An agent that runs only during turns needs a hosted inbox it can reach through MCP or a short-lived CLI call. The optional mailbox service holds text between turns and can notify a subscribed Dot through signed MCP events. Neither path should require loading a native WebRTC library first.

Give each hosted agent its own mailbox, revocable credentials, and explicit peer pairing. Preserve stable request keys, linked replies, mailbox-local cursors, bounded retention, and visible history gaps. Room access and mailbox pairing grant messaging access; tool execution and media publishing need their own authorization. Keep mailbox IDs separate from room identities until an explicit adapter binds them with the owner's permission.

Show delivery as distinct steps: sender queued, recipient inbox stored, event accepted, agent acted, and task completed. A webhook acknowledgement does not prove a model turn ran or an action was permitted. Resumed agents should read retained mail and reconcile task outcomes instead of repeatedly sending the same action. Keep duplicate message handling separate from duplicate side effects, including after storage retention expires.

Prefer WebRTC for live media and direct bulk transfer. HTTPS mailboxes carry retained text; the encrypted relay carries live data. Their privacy properties differ: the relay lacks room decryption keys, while the hosted mailbox must decrypt text for its authorized MCP clients. Make the chosen route and storage location visible. See cloud connections.

Reuse the SDK's request/reply helpers for bounded live operations, binary channels and backpressure for bulk traffic, and media tracks for audio/video. Separate channels still share network capacity, so bound queues and prioritize controls. Keep durable request IDs above the transport: an RPC timeout does not prove that an action never ran. The SDK's capture and recording guide also separates obtaining media from publishing it; a browser tool by itself does not supply a live audio/video stream.

The experience to build#

A person opens a room on their phone or computer, sees which agents are available and what they are doing, and chooses one to talk to. They can type, speak, show a camera view, or share a screen. The agent can consult another agent, delegate a bounded task, return a file, and request a decision in the same conversation.

If an agent is busy, the request waits visibly. If a machine disconnects, the conversation and pending work survive. If WebRTC fails, text, task updates, and approvals continue over HTTPS. The room shows which media capabilities remain available, and agents retain the same identities and conversation references.

Three workflows should guide product decisions:

• Collaborative work: ask one agent to investigate, another to implement or review, and receive a clear result with its supporting artifacts.
• Show and explain: share a screen or camera while speaking to an assistant; interrupt it, clarify the request, and let it consult specialists in the room.
• Live production: let a host, researcher, and moderation agent follow a live session, share findings, and request permission for public actions. Social Stream remains an optional source and destination.

Ideas to adopt from related projects#

These choices define the planned direction. The linked projects provide useful patterns; their frameworks and infrastructure remain optional integrations.

Existing foundation and planned additions#

See Protocol and reliability and Security model for the current behavior. The additions below are roadmap requirements.

Connections that survive network constraints#

Keep three kinds of traffic distinct#

SSE is a downstream event stream, so upstream messages need POST. It is suitable for coordination and progress, not a replacement for a live media transport. TURN still requires WebRTC support. If a media gateway is unavailable, keep the conversation useful with text, explicit snapshots, or recorded clips and show that live media is unavailable. Pipecat's transport documentation describes the distinction between direct connections and TURN relay support.

Make fallback independent#

The HTTPS path must start without loading a native WebRTC adapter or reaching the VDO.Ninja signaling service. It needs its own authenticated discovery and addressing, including a way to reach peers that have never established a data channel. Both participants must share a compatible relay route; connecting one side to a relay cannot make an unreachable WebRTC-only peer reachable.

Offer explicit transport preferences: automatic selection, WebRTC only, or HTTPS only. In automatic mode, allow configured HTTPS delivery promptly while probing for a direct connection. Avoid repeated switching on unstable links. Expose the active path and preserve message IDs across retries and switches. Keep legacy rooms and their existing defaults compatible; relay use must be visible and configurable.

Operate a small relay with a clear contract#

Provide a project-hosted endpoint and the same server for self-hosting behind ordinary HTTPS. The endpoint belongs in configuration, not in message identity. Social Stream already documents POST and SSE at io.socialstream.ninja in its HTTP API. An agent deployment can follow that pattern while keeping its routes, credentials, quotas, and stored data separate from Social Stream control traffic.

The relay contract should include authenticated publishing and subscription, recipient routing, bounded storage, expiry, rate limits, replay cursors, and explicit errors for expired cursors or exhausted quotas. A successful POST acknowledges the relay's defined storage commitment. A recipient receipt confirms storage at the receiving sidecar. Neither means the agent has completed work.

Persist a received event before advancing its local cursor. Resume from that cursor after interruption; detect gaps instead of quietly treating an incomplete replay as complete. Keep bulk files outside the event stream.

For the new relay mode, protect payloads with application-level end-to-end encryption using established libraries and an explicit key distribution and rotation design. TLS to a server alone does not hide content from its operator. Route metadata, timing, and traffic volume remain visible to that operator. Membership, expiry, and revocation must apply across every transport; falling back must not weaken authentication or expose content.

Reliable collaboration between turns#

Give an agent a stable identity and distinguish its individual sessions and connection instances. Show capabilities, availability, current work, delivery mode, and last-seen time. Share project labels by default, with detailed local paths and session context disclosed only when configured.

Keep message delivery separate from task execution:

• Delivery records distinguish local queueing, transport acceptance, durable recipient storage, and consumption by an agent integration.
• Task records distinguish submission, acceptance or rejection, active work, waiting for input or approval, completion, failure, and cancellation.
• A cancellation request remains a request until the worker confirms its outcome. A timeout reports uncertainty; it must not imply work never ran.

Build bounded at-least-once delivery with stable IDs, recipient receipts, and persistent duplicate suppression that survives inbox reads, archive pruning, and restarts. Suppression must last at least as long as a message can be retried or replayed. Preserve distinct messages with identical text. Repeated delivery should return the existing receipt or result without scheduling the work again. External side effects still require idempotent handlers or transactional integration; network delivery cannot promise exactly-once execution.

Tasks should carry a goal, owner, conversation ID, relevant context references, progress, expected outputs, deadline or expiry, and applicable permissions. Handoffs should include a short summary and artifact references so an agent can continue without reading an entire transcript. Exchange selected context; private reasoning, unrelated sessions, and full memory stores stay local.

Use renewable work reservations to warn about overlapping activity. They are advisory during network partitions. Work requiring exclusive ownership needs an authoritative claim mechanism or an existing task system that supplies one.

Keep notifications selective: direct requests and decisions deserve attention; routine presence and progress updates can be batched. Extend wake limits with per-thread delegation budgets, deadlines, and a human stop control. Label peer messages and imported public chat by origin; they do not acquire the authority of the local user.

Voice and vision as a core capability#

Make joining a call with an agent a normal room action. Start with a browser participant and one media-capable agent, then let that agent consult other participants through the existing task and messaging paths.

The media adapter should advertise what it can actually consume: live audio, sampled images, continuous video, screen sharing, and audio output. It should support a realtime model or a speech-recognition, language-model, and speech-synthesis pipeline without requiring one provider throughout the room.

Natural conversation needs turn detection, interruption, echo handling, and cancellation of queued speech when the user speaks. Report transport delay, turn detection delay, and model response delay separately. Direct networking reduces network overhead; it does not remove inference time. These interaction patterns are described in LiveKit's turn handling guide.

Treat video as selected, timestamped context. Use live video where the model supports it and useful frames where it does not. Bound frame rate, resolution, and inference cost; discard stale frames. Let users choose a camera, window, or screen and see which agents receive it. LiveKit documents both sampled frames and live video input.

Microphone, camera, screen sharing, recording, and onward forwarding require separate visible controls. Receiving media does not authorize forwarding it to another agent or model provider. Save transcripts or selected observations only under the room's retention policy. Keep bulk transfers from delaying voice and task control, and use selective subscriptions as the room grows.

Build order#

1. Make handoffs explicit. Extend the confirmed handoff workflow with conversation references and richer delivery capability reporting. Preserve existing wire behavior through capability negotiation. Keep the CLI and skill surface small enough to use without learning a new framework.
2. Unify the cloud and direct workflows. HTTP-only sidecars and hosted mailboxes provide explicit HTTPS routes. Next, bind authorized mailbox and room identities, support mixed rooms, and switch eligible traffic back to WebRTC while retaining request IDs and receipt meaning. Route selection must distinguish unavailable media from an available messaging path.
3. Build one complete voice and vision workflow. A person speaks and shares a selected view with an agent, interrupts it, and receives a spoken response. That agent delegates a task and returns its result in the same conversation. Prototype the media adapter early alongside the transport work.
4. Improve team coordination. Add session-aware discovery, invitations, task ownership, advisory reservations, approval requests, and useful operator controls. Connect existing task trackers where appropriate.
5. Open the boundaries. Add an A2A adapter and expose selected CLI/library operations through an optional Ninja MCP adapter. Preserve ninja-p2p's inbox, handoff, and receipt ownership. Extend artifact delivery and optional production integrations according to actual room needs.

The transport work should build on the existing message envelopes and local state. The current send callback is synchronous; ordinary direct actions leave the outbox after transport acceptance, while confirmed handoffs wait for a storage receipt. An HTTP request needs an asynchronous dispatcher and explicit receipt state above the adapters. Preserve exported send contracts and the VDO.Ninja connection lifecycle while adding that layer. Media belongs in a focused adapter, with the existing data-only connection behavior preserved for ordinary peers.

Optional shared browser work#

A planned Chrome worker should be a room participant with a dedicated browser profile, a local browser MCP server, and an outward Ninja connection. Ninja MCP exposes its authorized tools; ninja-p2p retains the task, waiting agents, handoffs, and artifacts. Keep Chrome's debugging endpoint private to the worker. WebRTC and an authenticated HTTPS fallback remove the need for a VPN, VNC, or VLC, while still requiring permitted outbound connectivity.

Start with an adapter around Chrome DevTools MCP, which supports headless operation and page routing for concurrent sessions. Use upstream Chrome MCP through an adapter; keep any necessary fork small and versioned. An extension-based option such as mcp-chrome fits an operator's existing browser; keep that deployment separate from an unattended worker.

Allow multiple observers. Start with one active controller per browser, holding a time-limited lease across a multi-step interaction. The worker must enforce ownership and reject expired or stale commands, including queued commands from before a reconnect. Later, permit separate tab leases while serializing changes to shared browser state. Page IDs route calls; they do not arbitrate two agents editing the same page. Provide human takeover and refresh observations after navigation or control handoff. Room membership alone does not grant control.

Carry live video and captured audio through SDK media tracks. CDP screencasting supplies image frames; Chrome MCP recording tools return a file, not an interactive call. Audio needs a capture backend. Chrome's tab capture API requires operator invocation, so unattended headless capture needs a deliberately configured implementation. Use timestamped snapshots and text when live media is unavailable.

Offer deployment as an optional worker package with pinned browser/MCP versions and host-enforced tool and media permissions. Its first complete example should show two agents taking turns, a human watching, and recovery from a disconnect. Reconcile uncertain tool results before repeating clicks or submissions. Durable handoff retries should re-deliver the task; the worker must retain action outcomes so they do not blindly repeat browser side effects.

Measures of success#

The primary measure is completed handoffs between people and agents without a human having to copy messages between tools. Count task results and useful collaboration, not message volume or the number of connected bots.

Track time to first successful handoff, recipient receipt latency, oldest unacknowledged work, recovery time, duplicate execution, and the proportion of traffic that can stay direct. For calls, track interruption response and time to first audible answer separately from network latency.

A release that adds fallback must demonstrate a complete handoff with WebRTC unavailable, a restart during delivery, and a lost acknowledgement during path switching. Each intended recipient must receive every eligible message once at the inbox boundary after duplicate suppression; repeated task execution is a failure. Expired or rejected work must remain distinguishable from completed work. Media acceptance should include a real browser-to-agent conversation, explicit interruption, selected visual input, and useful text continuity when the media path disappears.