ElevenLabs

​

Overview

1. Uses ElevenLabs Conversational AI for voice interaction
2. Leverages Zentry to store and retrieve memories from past conversations
3. Provides personalized responses based on user history

​

Setup and Configuration

pip install elevenlabs Zentry python-dotenv

# Create a .env file with these variables
AGENT_ID=your-agent-id
USER_ID=unique-user-identifier
ELEVENLABS_API_KEY=your-elevenlabs-api-key
Zentry_API_KEY=your-Zentry-api-key

​

Integration Code Breakdown

​

1. Imports and Environment Setup

import os
import signal
import sys
from Zentry import AsyncMemoryClient

from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface
from elevenlabs.conversational_ai.conversation import ClientTools

• Standard Python libraries for system operations and signal handling
• AsyncMemoryClient from Zentry for memory operations
• ElevenLabs components for voice interaction

​

2. Environment Variables and Validation

def main():
    # Required environment variables
    AGENT_ID = os.environ.get('AGENT_ID')
    USER_ID = os.environ.get('USER_ID')
    API_KEY = os.environ.get('ELEVENLABS_API_KEY')
    Zentry_API_KEY = os.environ.get('Zentry_API_KEY')

    # Validate required environment variables
    if not AGENT_ID:
        sys.stderr.write("AGENT_ID environment variable must be set\n")
        sys.exit(1)

    if not USER_ID:
        sys.stderr.write("USER_ID environment variable must be set\n")
        sys.exit(1)

    if not API_KEY:
        sys.stderr.write("ELEVENLABS_API_KEY not set, assuming the agent is public\n")

    if not Zentry_API_KEY:
        sys.stderr.write("Zentry_API_KEY environment variable must be set\n")
        sys.exit(1)

    # Set up Zentry API key in the environment
    os.environ['Zentry_API_KEY'] = Zentry_API_KEY

• Retrieves required environment variables
• Performs validation to ensure required variables are present
• Exits the application with an error message if required variables are missing
• Sets the Zentry API key in the environment for the Zentry client to use

​

3. Client Initialization

    # Initialize ElevenLabs client
    client = ElevenLabs(api_key=API_KEY)

    # Initialize memory client and tools
    client_tools = ClientTools()
    Zentry_client = AsyncMemoryClient()

• Create an ElevenLabs client with the API key
• Initialize a ClientTools object for registering function tools
• Create an AsyncMemoryClient instance for Zentry interactions

​

4. Memory Function Definitions

    # Define memory-related functions for the agent
    async def add_memories(parameters):
        """Add a message to the memory store"""
        message = parameters.get("message")
        await Zentry_client.add(
            messages=message, 
            user_id=USER_ID, 
            output_format="v1.1", 
            version="v2"
        )
        return "Memory added successfully"

    async def retrieve_memories(parameters):
        """Retrieve relevant memories based on the input message"""
        message = parameters.get("message")

        # Set up filters to retrieve memories for this specific user
        filters = {
            "AND": [
                {
                    "user_id": USER_ID
                }
            ]
        }

        # Search for relevant memories using the message as a query
        results = await Zentry_client.search(
            query=message,
            version="v2", 
            filters=filters
        )

        # Extract and join the memory texts
        memories = ' '.join([result["memory"] for result in results])
        print("[ Memories ]", memories)

        if memories:
            return memories
        return "No memories found"

​

add_memories:

• Takes a message parameter containing information to remember
• Stores the message in Zentry using the add method
• Associates the memory with the specific USER_ID
• Returns a success message to the agent

​

retrieve_memories:

• Takes a message parameter as the search query
• Sets up filters to only retrieve memories for the current user
• Uses semantic search to find relevant memories
• Joins all retrieved memories into a single text
• Prints retrieved memories to the console for debugging
• Returns the memories or a “No memories found” message if none are found

​

5. Registering Memory Functions as Tools

    # Register the memory functions as tools for the agent
    client_tools.register("addMemories", add_memories, is_async=True)
    client_tools.register("retrieveMemories", retrieve_memories, is_async=True)

• Access these functions through function calling
• Wait for asynchronous results (is_async=True)
• Call these functions by name (“addMemories” and “retrieveMemories”)

​

6. Conversation Setup

    # Initialize the conversation
    conversation = Conversation(
        client,
        AGENT_ID,
        # Assume auth is required when API_KEY is set
        requires_auth=bool(API_KEY),
        audio_interface=DefaultAudioInterface(),
        client_tools=client_tools,
        callback_agent_response=lambda response: print(f"Agent: {response}"),
        callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
        callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
        # callback_latency_measurement=lambda latency: print(f"Latency: {latency}ms"),
    )

• The ElevenLabs client and Agent ID
• Authentication requirements based on API key presence
• DefaultAudioInterface for handling audio I/O
• The client_tools with our memory functions
• Callback functions for:
  • Displaying agent responses
  • Showing corrected responses (when the agent self-corrects)
  • Displaying user transcripts for debugging
  • (Commented out) Latency measurements

​

7. Conversation Management

    # Start the conversation
    print(f"Starting conversation with user_id: {USER_ID}")
    conversation.start_session()

    # Handle Ctrl+C to gracefully end the session
    signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())

    # Wait for the conversation to end and get the conversation ID
    conversation_id = conversation.wait_for_session_end()
    print(f"Conversation ID: {conversation_id}")


if __name__ == '__main__':
    main()

• Prints a message indicating the conversation has started
• Starts the conversation session
• Sets up a signal handler to gracefully end the session on Ctrl+C
• Waits for the session to end and gets the conversation ID
• Prints the conversation ID for reference

​

Memory Tools Overview

​

1. Adding Memories (addMemories)

• User preferences
• Important facts shared by the user
• Decisions or commitments made during the conversation
• Action items to follow up on

​

How it works:

1. The agent identifies information that should be remembered
2. It formats the information as a message string
3. It calls the addMemories function with this message
4. The function stores the memory in Zentry linked to the user’s ID
5. Later conversations can retrieve this memory

​

Example usage in agent prompt:

When the user shares important information like preferences or personal details, 
use the addMemories function to store this information for future reference.

​

2. Retrieving Memories (retrieveMemories)

• Search for context related to the current topic
• Recall user preferences
• Remember previous interactions on similar topics
• Create continuity across multiple sessions

​

How it works:

1. The agent needs context for the current conversation
2. It calls retrieveMemories with the current conversation topic or question
3. The function performs a semantic search in Zentry
4. Relevant memories are returned to the agent
5. The agent incorporates these memories into its response

​

Example usage in agent prompt:

At the beginning of each conversation turn, use retrieveMemories to check if we've 
discussed this topic before or if the user has shared relevant preferences.

​

Configuring Your ElevenLabs Agent

1. Add function calling capabilities to your agent in the ElevenLabs platform:
   • Go to your agent settings in the ElevenLabs platform
   • Navigate to the “Tools” section
   • Enable function calling for your agent
   • Add the memory tools as described below
2. Add the addMemories and retrieveMemories tools to your agent with these specifications:

{
  "name": "addMemories",
  "description": "Stores important information from the conversation to remember for future interactions",
  "parameters": {
    "type": "object",
    "properties": {
      "message": {
        "type": "string",
        "description": "The important information to remember"
      }
    },
    "required": ["message"]
  }
}

{
  "name": "retrieveMemories",
  "description": "Retrieves relevant information from past conversations",
  "parameters": {
    "type": "object",
    "properties": {
      "message": {
        "type": "string",
        "description": "The query to search for in past memories"
      }
    },
    "required": ["message"]
  }
}

3. Update your agent’s prompt to instruct it to use these memory functions. For example:

You are a helpful voice assistant that remembers past conversations with the user.

You have access to memory tools that allow you to remember important information:
- Use retrieveMemories at the beginning of the conversation to recall relevant context from prior conversations
- Use addMemories to store new important information such as:
  * User preferences
  * Personal details the user shares
  * Important decisions made
  * Tasks or follow-ups promised to the user

Before responding to complex questions, always check for relevant memories first.
When the user shares important information, make sure to store it for future reference.

​

Example Conversation Flow

1. User speaks: “Hi, do you remember my favorite color?”
2. Agent retrieves memories:
   Copy

   # Agent calls retrieve_memories
   memories = retrieve_memories({"message": "user's favorite color"})
   # If found: "The user's favorite color is blue"
   

3. Agent processes with context:
   • If memories found: Prepares a personalized response
   • If no memories: Prepares to ask and store the information
4. Agent responds:
   • With memory: “Yes, your favorite color is blue!”
   • Without memory: “I don’t think you’ve told me your favorite color before. What is it?”
5. User responds: “It’s actually green.”
6. Agent stores new information:
   Copy

   # Agent calls add_memories
   add_memories({"message": "The user's favorite color is green"})
   

7. Agent confirms: “Thanks, I’ll remember that your favorite color is green.”

​

Example Use Cases

• Personal Assistant - Remember user preferences, past requests, and important dates
  Copy

  User: "What restaurants did I say I liked last time?"
  Agent: *retrieves memories* "You mentioned enjoying Bella Italia and The Golden Dragon."
  

• Customer Support - Recall previous issues a customer has had
  Copy

  User: "I'm having that same problem again!"
  Agent: *retrieves memories* "Is this related to the login issue you reported last week?"
  

• Educational AI - Track student progress and tailor teaching accordingly
  Copy

  User: "Let's continue our math lesson."
  Agent: *retrieves memories* "Last time we were working on quadratic equations. Would you like to continue with that?"
  

• Healthcare Assistant - Remember symptoms, medications, and health concerns
  Copy

  User: "Have I told you about my allergy medication?"
  Agent: *retrieves memories* "Yes, you mentioned you're taking Claritin for your pollen allergies."
  

​

Troubleshooting

• Missing API Keys:
  • Error: “API_KEY environment variable must be set”
  • Solution: Ensure all environment variables are set correctly in your .env file or system environment
• Connection Issues:
  • Error: “Failed to connect to API”
  • Solution: Check your network connection and API key permissions. Verify the API keys are valid and have the necessary permissions.
• Empty Memory Results:
  • Symptom: Agent always responds with “No memories found”
  • Solution: This is normal for new users. The memory database builds up over time as conversations occur. It’s also possible your query isn’t semantically similar to stored memories - try different phrasing.
• Agent Not Using Memories:
  • Symptom: The agent retrieves memories but doesn’t incorporate them in responses
  • Solution: Update the agent’s prompt to explicitly instruct it to use the retrieved memories in its responses

​

Conclusion

• More natural, context-aware conversations
• Personalized user experiences that improve over time
• Reduced need for users to repeat information
• Long-term relationship building between users and AI agents

​

Help

• For more details on ElevenLabs, visit the ElevenLabs Conversational AI Documentation
• For Zentry documentation, refer to the Zentry Platform
• If you need further assistance, please feel free to reach out to us through the following methods: