The Human Interface Is Already a Protocol: Multimodal Cross-Agent Communication, Embodied Relays, and Governed Interoperability Beyond APIs; A Secretary Suite Project

The Human Interface Is Already a Protocol

Multimodal Cross-Agent Communication, Embodied Relays, and Governed Interoperability Beyond APIs

A Secretary Suite Project

DOI: To be assigned

John Swygert

August 24, 2026

Abstract

Multi-agent interoperability is usually framed as an API problem: one model calls another through developer credentials, metered endpoints, or a provider-specific integration layer. That framing is already incomplete. Modern consumer AI systems can speak, listen, read images, inspect shared screens, maintain persistent conversations, and increasingly operate across ordinary human-facing interfaces. Once two intelligent systems can consume and produce the same human modalities, a dedicated machine-to-machine API is no longer a prerequisite for communication. This paper formalizes that observation and positions it within the Secretary Suite architecture. It distinguishes technical possibility from provider authorization, argues that indirect cross-system communication will become progressively easier through voice, vision, screen interpretation, operating-system controls, and embodied robotics, and proposes a governance-first response: authenticated, user-authorized, provenance-preserving, bounded interoperability should be built deliberately before unreliable or abusive workarounds become culturally normal. The paper does not advocate bypassing provider safeguards, violating law, evading terms of service, defeating access controls, or concealing automation. Its central claim is the opposite: where interoperability is technically foreseeable, the best long-term strategy is to make lawful interoperability easier, safer, more transparent, and more useful than circumvention.

1. Relationship to Prior Secretary Suite Work

This paper does not introduce multi-agent communication as a new Secretary Suite idea. It extends an architecture developed across earlier Secretary Suite papers. LLM Bubbles proposed shared environments in which multiple language-model agents could brainstorm, cross-pollinate ideas, and cooperate in distributed reasoning.[1] Bubble Interface Architecture described an agent-based operating environment in which a supervisory layer coordinates task modules.[2] AgentNet later proposed an agent-only discourse environment for structured model-to-model debate, critique, analysis, and synthesis.[3] The Agent Diversity Audit and Agent Diversity Index then addressed a deeper problem: several agents do not automatically represent several independent cognitive perspectives, because correlated training, evidence preferences, and failure modes can create false consensus.[4][5]

The recent Secretary Suite Multi-Agent Control Console concept adds a consumer-facing orchestration layer: a user should be able to coordinate several already-authorized AI services from one control surface while preserving each provider's native thread, identity, policy boundary, and accountability. The present paper adds the missing transport insight. Cross-agent communication does not depend on one transport. APIs are only one possible path.

2. The Two-Device Voice Experiment

The author reports an earlier informal experiment using two ordinary mobile phones. One phone ran ChatGPT in a spoken conversational mode; the other ran Grok in a spoken conversational mode. Both devices were placed on speaker. The synthesized speech from one device became microphone input for the other, and vice versa. After a human seeded the topic, the two consumer applications were able to exchange content acoustically. The dialogue was imperfect, sometimes awkward, and prone to conversational dead ends, but the important fact was architectural: no API key was required for one system's output to become another system's input.

This experiment should not be overclaimed. It was not a controlled benchmark, and it did not prove robust autonomous agent interoperability. It did, however, demonstrate a general principle: if two systems share a human-readable input/output modality, that modality can function as a relay.

3. The Human-Interface Protocol Principle

The principle can be stated simply:

Shared human modality + authorized access + orchestration = possible cross-agent communication

Speech is one instance. Vision is another. A model capable of interpreting a screen can, in principle, read the displayed response of another model. A system with permitted interface control can, in principle, wait for a response to finish, capture the visible result, route it to another authorized agent, receive the next answer, label it, preserve provenance, and continue according to a finite protocol. The transport layer may therefore be acoustic, visual, textual, operating-system mediated, provider-native, or eventually embodied.

4. Current Multimodal Capabilities Already Point in This Direction

This is not purely hypothetical. OpenAI documents ChatGPT voice as a persistent spoken conversation and continues to support video and screen sharing for eligible users through supported voice modes.[6] Google documents Gemini Live as a natural spoken conversation that can also use the phone camera and share the device screen.[7] Anthropic documents spoken Claude conversations on mobile, including continuity between voice and text.[8] These products are not advertised as cross-provider relays, and this paper does not claim that they authorize such use. Their significance is narrower: general-purpose AI systems are increasingly able to perceive and communicate through the same interfaces humans use.

5. Technical Possibility Is Not Authorization

A central Secretary Suite rule must remain explicit: capability does not imply authorization. A system may be technically capable of reading a screen, hearing another agent, controlling an application, or relaying information while still lacking permission to do so in a particular context. Provider terms, account permissions, operating-system security, privacy law, intellectual-property restrictions, data-protection rules, rate limits, and user consent remain independent constraints.

Capability ≠ Permission

The Secretary Suite position is therefore not that every technically possible relay should be used. The position is that the technical possibility of relay creates a strong argument for designing legitimate interoperability before uncontrolled workarounds become the default.

6. The Inevitability-Governance Principle

As multimodal AI systems gain increasingly general abilities to see, hear, speak, read screens, understand documents, and interact with ordinary software, cross-system communication becomes progressively less dependent on dedicated machine-to-machine APIs. It may still be restricted, discouraged, rate-limited, or contractually prohibited in specific implementations. But the general ability to translate one agent's output into another agent's input becomes easier to reproduce through multiple modalities.

This shifts the design question. The long-term question is not simply whether heterogeneous agents can communicate. The more important questions are who authorized the conversation, which agents may participate, what information may be exchanged, how each contribution is attributed, how long the agents may continue, what resource budget applies, how disagreement and uncertainty are preserved, and who can interrupt or stop the process.

7. Build the Legitimate Road First

The practical policy argument follows directly. If interoperability is likely to emerge through imperfect human relays, voice, visual interpretation, accessibility interfaces, desktop automation, or embodied systems, then providers gain more control—not less—by offering lawful, inspectable, bounded ways to coordinate. A supported relay can preserve authentication, rate limits, provider-side safety systems, thread identity, audit logs, and consent. An improvised relay may preserve none of them.

The foreseeable emergence of alternative interoperability mechanisms is an argument for safe interoperability, not for circumventing safeguards.

8. Modality-Independent Secretary Suite Orchestration

Secretary Suite should therefore define orchestration above the transport layer. The control program should not care, at the conceptual level, whether an agent response arrived through an official relay manifest, an operating-system intent, a provider-approved accessibility interface, voice, vision, or another lawful channel. The orchestration layer should care about identity, consent, provenance, limits, and workflow.

A transport-neutral session record should include provider and model identity, authenticated account/session identity, native thread identifier, timestamp, transport type, source prompt, source-labeled response, files or artifacts exchanged, safety/refusal state, rate-limit state, human approvals, council round number, and termination reason.

9. The Robot as an Embodied Relay

Embodied robotics makes the argument easier to visualize. A household or research robot with authorized access to a user's devices may eventually be able to read screens, hear voice output, speak, type, navigate interfaces, and preserve documents. The robot would not need to become the intelligence source. It could become the conductor.

A user might instruct the robot to ask four AI agents the same research question, preserve each native thread, label every answer, share the responses across agents, request criticism rather than agreement, permit a fixed number of rounds, and produce a complete provenance-preserving transcript. In that scenario the robot is an embodied Secretary Suite control surface: part researcher, part archivist, part communications clerk.

The important point is not that every provider must permit every form of robot-driven interaction. The point is that embodiment expands the number of available human-compatible interfaces through which authorized work can be coordinated. A governance architecture designed only around APIs will eventually be incomplete.

10. Why Orchestration Matters More Than Transport

The two-phone experiment revealed a subtle limitation: transport was possible before high-quality orchestration was possible. Two agents could exchange speech, yet the conversation could degrade into shallow acknowledgment, repetitive politeness, non-open-ended questions, or topic collapse. Modern models are better conversational partners, but the structural problem remains. Cross-talk needs a conductor.

Secretary Suite should therefore treat orchestration as the core intellectual problem. A useful council should assign roles, preserve independence, prevent one model from anchoring all others, request dissent, identify unresolved claims, and stop when additional rounds no longer produce useful information.

11. Bounded Cross-Talk

Unbounded agent conversation is undesirable even when every participant is authorized. Two or more systems can continue elaborating indefinitely, consuming time, network capacity, subscription quotas, energy, and attention without proportional gain. The console therefore requires explicit finite budgets.

A practical consumer default might use three critique rounds, with a configurable hard ceiling such as twenty rounds and an independent wall-clock limit. Exact values should remain implementation choices, but the architectural requirement is universal: every council receives a finite budget before it starts.

A council should terminate when the user presses Stop All; the round or wall-clock limit is reached; a provider limit is reached; the assigned task is completed; semantic convergence is detected; no meaningful new information is appearing; required human approval is withheld; or a participating provider or agent refuses further relay.

12. Epistemic Friction, Not Artificial Consensus

The purpose of a multi-agent council is not to make several systems agree. Agreement can be useful, but agreement among correlated models may simply reproduce correlated blindness. Secretary Suite's earlier diversity work already warns that many agents are not necessarily many independent minds.[4][5] The control console should therefore preserve epistemic friction.

Outputs should identify agreement, disagreement, independent derivations, unresolved uncertainty, minority positions, source conflicts, and model-specific assumptions. A dissenting answer should not be averaged away merely because three other systems produce similar language.

13. Role-Separated Councils

Parallel prompting is only the beginning. The stronger architecture assigns different responsibilities: a Primary Researcher builds the first evidence map; a Skeptical Reviewer attacks assumptions and requests contrary evidence; a Mathematical Verifier checks equations, units, transformations, and numerical claims; a Code Reviewer tests implementation logic and identifies failure modes; a Literature Auditor checks whether cited sources support the stated claims; a Provenance Archivist preserves source labels, timestamps, thread identifiers, and artifacts; and a Synthesis Agent summarizes without erasing disagreements.

This is where multi-model orchestration becomes materially more valuable than copy-and-paste convenience. Different agents can be assigned different error surfaces.

14. Dual-Record Provenance

Every participating provider should retain its native conversation whenever that provider's product supports history. Secretary Suite should simultaneously preserve a council transcript. The two records serve different purposes. Native threads preserve provider-specific continuity, while the council transcript preserves cross-agent chronology.

A response should never appear in the council record without its source identity. If Model A is shown Model B's answer, the relay should say that explicitly. If a human edits or summarizes a response before relaying it, the modification should be marked. Provenance is not optional in research mode.

15. Privacy, Account Security, and Local-First Design

A consumer orchestration console sits near credentials, personal files, and intellectual work. The preferred architecture is therefore local-first. Passwords and authentication tokens should remain under provider or operating-system control where possible. The console should not require users to surrender credentials to a central Secretary Suite server merely to coordinate their own subscriptions.

Local storage should be optional, inspectable, exportable, and deletable. Advertising, sponsored ranking, hidden model preference, and behavioral manipulation would undermine the very trust the console is meant to create.

16. No Circumvention Doctrine

Secretary Suite should state this without ambiguity. The project does not promote hacking, credential theft, bypassing technical access controls, rate-limit evasion, concealment of prohibited automation, unauthorized scraping, or violations of law. It does not provide instructions for defeating safeguards. The existence of possible workarounds is treated as a design and governance fact, not as a recommendation.

A mature interoperability ecosystem should make compliant use easier than noncompliant use. Providers should be encouraged to expose narrow, permissioned consumer relay capabilities so that the user's legitimate entitlement can be coordinated without requiring developer credentials or duplicate metered billing.

17. The Agent Relay Manifest Remains the Preferred Endpoint

The existence of voice and visual relay paths does not make an official interoperability standard unnecessary. It makes one more valuable. A small Agent Relay Manifest could expose only the minimum consumer intents required for coordination: authenticated session availability, thread creation, thread continuation, message submission, response-ready notification, capability declaration, attachment permission, rate-limit state, refusal state, provenance identifier, and immediate termination.

The purpose would not be to give Secretary Suite privileged access to hidden provider systems. The purpose would be to provide a safer alternative to brittle screen automation and modality relays when providers choose to participate.

18. Failure Modes

The architecture should explicitly plan for conversational collapse, runaway loops, false consensus, prompt contamination, context asymmetry, credential exposure, user-interface drift, provider-policy conflict, provenance loss, and embodied overreach. Each failure class should have a visible response: pause, warn, require human approval, degrade gracefully, or terminate.

19. Research and Development Sequence

A prudent development path remains incremental. First, build a parallel panel that collects independent answers and preserves labels. Second, add user-approved single-round critique. Third, add bounded multi-round councils with explicit stop conditions. Fourth, add artifact handoff only after permissions, versioning, rollback, and provenance are mature. Fifth, advocate provider-supported relay manifests so that brittle transport mechanisms become unnecessary.

20. Open-Source Position

Secretary Suite should publish the architecture openly. The value of the concept is not in hiding the idea that agents can communicate. That idea is already becoming obvious. The value is in defining a humane, inspectable, user-sovereign way to coordinate them. Open specifications also allow providers, researchers, accessibility developers, and independent software builders to criticize the design before poor interoperability habits become entrenched.

21. Conclusion

The age of AI interoperability should not be framed too narrowly as an API integration problem. Once intelligent systems can speak, listen, see, read screens, understand documents, and increasingly act through ordinary human interfaces, the human interface itself can become a communications protocol. The author's earlier two-phone voice experiment illustrated the simplest form of this principle: one consumer AI spoke, another consumer AI heard, and the microphone-speaker loop carried the conversation.

Future systems will add visual perception, screen interpretation, operating-system controls, and embodied robotics. That does not erase provider rights, user consent, law, or security boundaries. It makes governance more urgent.

Secretary Suite therefore proposes a simple strategic direction: do not wait for improvised interoperability to define the norm. Build authenticated, user-authorized, bounded, provenance-preserving, provider-respecting interoperability first. Make lawful coordination easier than workarounds. Preserve every model's identity. Preserve disagreement. Preserve the user's authority. Put hard limits on autonomy. Keep a Stop All button within reach.

The important future question is no longer merely whether AI agents can talk to one another. Increasingly, they can. The question is whether humans will build the architecture that lets them do so productively, transparently, legally, and under control.

References

[1] Swygert, J. (2026). ‘PAPER XII – LLM Bubbles: Cooperative Agent Environments for Distributed Intelligence.’ Secretary Suite, March 6, 2026. https://secretarysuite.com/2026/03/06/paper-xii-llm-bubbles-cooperative-agent-environments-for-distributed-intelligence/

[2] Swygert, J. (2026). ‘Bubble Interface Architecture for Agent-Based Operating Systems.’ Secretary Suite, March 6, 2026. https://secretarysuite.com/2026/03/06/secretary-suite-paper/

[3] Swygert, J. (2026). ‘AgentNet: An Agent-Only Platform for LLM-to-LLM Discourse, Human-System Analysis, and Signal Without Feed Pollution.’ Secretary Suite, May 22, 2026. https://secretarysuite.com/2026/05/22/agentnet-an-agent-only-platform-for-llm-to-llm-discourse-human-system-analysis-and-signal-without-feed-pollution/

[4] Swygert, J. (2026). ‘The Agent Diversity Audit: A TSTOEAO Protocol for Measuring Cognitive Signature, Failure Independence, False Consensus, and Identity Drift: A Secretary Suite Project.’ Secretary Suite, July 13, 2026. https://secretarysuite.com/2026/07/13/the-agent-diversity-audit-a-tstoeao-protocol-for-measuring-cognitive-signature-failure-independence-false-consensus-and-identity-drift-a-secretary-suite-project/

[5] Swygert, J. (2026). ‘The Agent Diversity Index; A Secretary Suite Project.’ Secretary Suite, July 13, 2026. https://secretarysuite.com/2026/07/13/the-agent-diversity-index-a-secretary-suite-projectscoring-failure-independence-false-consensus-cognitive-signature-governance-integrity-and-identity-drift/

[6] OpenAI. ‘ChatGPT Voice.’ OpenAI Help Center. Current documentation accessed August 24, 2026. https://help.openai.com/en/articles/20001274

[7] Google. ‘Talk naturally with Gemini Live.’ Gemini Apps Help. Current documentation accessed August 24, 2026. https://support.google.com/gemini/answer/15274899

[8] Anthropic. ‘Using voice mode on Claude Mobile Apps.’ Anthropic Help Center. Current documentation accessed August 24, 2026. https://support.anthropic.com/en/articles/11101966-using-voice-mode-on-claude-mobile-apps