AI Agent ROI Calculator
Free training: Chatbot crash course
Whitepaper: The acceptance of chatbots
Prompt Injection
--> Go to BOTwiki
Prompt injection is a type of attack against LLM-based systems in which attackers inject manipulated inputs to hijack the model’s behavior—for example, to bypass security rules, retrieve confidential content, or trigger unwanted actions. For a productive AI agent , prompt injection is one of the most significant threat models, which cannot be mitigated by a single protective measure but only through a combination of architecture, filtering, and monitoring.
Direct and Indirect Prompt Injection
- Direct injection: The user crafts the input in such a way that it overrides the system's instructions, for example by instructing it to ignore previous rules.
- Indirect injection: Malicious instructions are hidden within external content processed by the agent, such as web pages, emails, or documents accessed by a tool.
Why single-prompt tools are particularly vulnerable
Wrapper tools that process all requests through a single monolithic prompt are structurally vulnerable. They lack a clear separation between trusted instructions, user input, and external context; instead, everything ends up in the same token stream. Multi-agent orchestration makes such attacks significantly more difficult because each stage has clearly defined responsibilities.
Defense strategies
Effective protective measures consist of several layers:
- Strict separation of user input and system prompts, so that the model does not interpret instructions in the input as commands.
- Filters and detection layers for identifying suspicious patterns in input data and external content.
- Sandboxing external sources so that retrieved content does not enter the model context without being verified.
- Monitoring and alerting for atypical model behavior.
- Regular audits through internal and external penetration tests.
Prompt Injection in an Industry Context
In a service context, security-critical workflows are particularly affected: SAP write-backs in municipal utilities, patient inquiries in healthcare, and payment issues in publishing. Closely related to prompt injection are issues such as prompt jailbreaks and AI safety filters.
Frequently Asked Questions (FAQ)
There is no such thing as 100% security. However, with a multi-layered architecture, a multi-agent setup, and continuous monitoring, the risk can be significantly reduced.
A central one. Because responsibilities are distributed among specialized agents, a single compromised prompt has only a limited scope of impact.
They are often harder to detect because the malicious instructions are not directly embedded in the user input. This makes them a particularly important focus of modern security layers.
The GDPR and the EU AI Act apply to production systems in Germany, Austria, and Switzerland. BOTfriends complies with these requirements and provides the necessary audit logs and data protection impact assessments.
--> Back to BOTwiki
Vector Database
--> Go to BOTwiki
A vector database is a specialized database designed to store and search embedding vectors. It serves as the technical backbone of any semantic search and is therefore a central component of modern knowledge AI in voice and chat applications. For a chatbot or voicebot with a substantial knowledge base, the choice and configuration of the vector database directly influence response quality, latency, and operating costs.
What Sets Vector Databases Apart from Traditional Databases
Relational databases work with precise values and exact joins. A vector database, on the other hand, stores high-dimensional vectors and supports nearest-neighbor searches. To do this, vector databases use approximate algorithms such as HNSW, IVF, or PQ, which enable extremely high speeds but also introduce a slight, controlled loss of quality.
Common options on the market
- Specialized vector databases: Pinecone, Weaviate, Qdrant, Milvus.
- Extensions for traditional databases: pgvector for PostgreSQL, Elasticsearch with vector search.
- Cloud-native services: Vertex AI Matching Engine, Azure AI Search, Amazon OpenSearch.
BOTfriends selects the vector database on a case-by-case basis for each use case—with scalability, EU hosting, filtering capabilities, and integration with the existing platform being key factors.
Vector Database in the RAG Pipeline
A typical RAG pipeline consists of three steps: document chunking, generating the embeddings , and storage in the vector database. When a query is made, the query itself is embedded, the vector database returns the top hits, and a reranker determines the final ranking. Only this combined stack enables Semantic Search at a production-ready level.
Scaling, Filtering, and Governance
High-performance vector databases must do more than just perform nearest-neighbor searches. Key features include metadata filters (such as language, client, and date), multi-tenancy for different client contexts, and a clear authorization model. For BOTfriends, EU hosting is mandatory, as are auditable logs and a clearly defined deletion process. This ensures the platform remains GDPR-compliant while maintaining high performance.
Frequently Asked Questions (FAQ)
Not necessarily. Specialized systems are only worthwhile once you reach a certain volume.
This is very important. Client filters, language filters, and document date filters transform a generic search into a Knowledge AI system that can be used productively.
That depends on the number and dimensionality of the vectors. Embedding quantization can significantly reduce memory requirements.
Yes. Different vector spaces can be hosted on the same platform, for example, for different languages or use cases. It is important to maintain clear separation and versioning.
--> Back to BOTwiki
Context Window
--> Go to BOTwiki
The context window refers to the maximum number of tokens that a large language model can process simultaneously in a single inference step. It encompasses both the input and the output, and thus serves as a hard limit for the system prompt, conversation history, knowledge sources, and the response. Modern models offer context windows ranging from a few thousand to several million tokens. For a productive AI agent platform , however, the question is not how large the context window is in theory, but how it is deliberately utilized in the respective use case.
Why Context Windows Are Important
Any conversation that lasts longer than a few turns, or any application using Knowledge AI, will sooner or later reach the limits of the context window. If these limits are exceeded, content must be summarized, omitted, or reduced through other strategies. Without deliberate management, this will result in either gaps in the conversation or uncontrolled prolongations.
Strategies for Working with the Context Window
- Conversation summarization: Older turns are converted into concise summaries.
- Knowledge Retrieval: Instead of carrying all sources, only the truly relevant chunks are loaded for each step.
- Modular system prompt: Use-case-specific rules are loaded only when they apply.
- Token Budgeting: Active planning of the distribution between input and output.
Bigger isn't necessarily better
Even though models with large context windows can process virtually any amount of data, this does not automatically lead to better answers. On the contrary: the more unstructured context is included, the higher the risk of context contamination and hallucinations. Successful implementations combine a realistic context window with a clean retrieval pipeline and disciplined token management.
Context Window and Multi-Agent Orchestration
In a multi-agent orchestration, the context window is structured specifically for each agent. A triage agent requires only the necessary classification information, while a specialized process agent receives structured parameters. This keeps each context window small, focused, and audit-ready—an advantage over monolithic setups that cram all their knowledge into a single prompt. You can find more about the basic token concept in the article on Tokens.
Frequently Asked Questions (FAQ)
That depends on the use case. For typical service conversations, manageable context windows are sufficient, provided they are intelligently populated through retrieval and summarization.
No, provided that sequence summarization and proper token management are in place. Long conversations are manageable, but they require a solid architectural foundation—not just a large context window.
The more tokens a model processes, the longer inference takes. A smaller, focused context window means faster responses—another reason not to confuse bigger with better.
--> Back to BOTwiki
Streaming Response
--> Go to BOTwiki
Streaming response refers to the method of outputting an LLM response not as a complete package, but token by token during the ongoing inference process. For voice and chat applications, streaming is a key factor in reducing latency: users can already see or hear the first few words while the model continues to generate content. For a voicebot , this effect is crucial because it makes the conversation feel natural, rather than creating a pause after a question.
How streaming works technically
LLMs generate tokens incrementally. With streaming, these tokens are passed to the application immediately after they are generated, rather than being buffered until the end of the generation process. Through protocols such as Server-Sent Events or streamed HTTP responses, the frontend receives the tokens in near real time. For voice applications, the token stream is fed directly into the text-to-speech synthesis, so that speech output and generation overlap in time.
Benefits of Streaming Response
- Reduced perceived latency: Initial responses are audible or visible within milliseconds.
- A more natural conversation experience in the voice channel.
- A Better User Experience in the chatbotbecause there are no noticeable pauses.
- Improved fault tolerance—initial results are in, even when late tokens are delayed.
Limitations and Trade-offs
Streaming response is not a panacea. For responses that need to be verified or formatted as a whole—such as structured JSON output for tool calls—traditional full responses make more sense. Fact-checking steps also require the full response context. In multi-agent setups, therefore, only the final output is often streamed to users, while internal reasoning steps are processed in a buffered manner.
Streaming and Perceived Latency
Streaming has a particularly strong impact in voice communication because the human brain perceives even brief pauses as unpleasant. Streaming is closely linked to the issue of latency: While the actual inference time remains the same, the perceived latency drops dramatically. For a service center, this translates to measurably higher customer satisfaction without the need to change models or hardware.
Frequently Asked Questions (FAQ)
Streaming is suitable for conversations where a quick initial response is important. For structured backend responses or fact-checking pipelines, a full response may be more appropriate.
Typical examples include server-sent events or streamed HTTP responses. In the voice channel, streamed text-to-speech is also used.
No. Streaming does not affect costs or tokens; it only changes how the response is delivered.
Yes. In the event of a topic correction or escalation, the stream can be stopped, the model paused, and the dialogue rerouted—an important advantage for multi-agent workflows.
--> Back to BOTwiki
Chunking for RAG
--> Go to BOTwiki
Chunking refers to the process of breaking down long documents into smaller, self-contained sections before they are converted into embeddings in a vector database. For Retrieval-Augmented Generation (RAG), chunking is the preliminary step that determines answer quality and hit rates. Poor chunking leads to hallucinations or incomplete answers, while good chunking forms the foundation of a robust knowledge base for phonebots and chatbots—regardless of whether the content comes from FAQs, manuals, or contract documents.
Why Chunking Matters
An LLM always answers a question based on the context provided in the prompt. In the case of RAG , this context is dynamically constructed from relevant document sections. If the sections are too long, they unnecessarily consume context window space and contain irrelevant information. If they are too short, the semantic context is missing. Good chunking strikes a balance and is both substantively complete and technically efficient.
Common chunking strategies
- Fixed-size chunking: Text is divided into chunks of a fixed size, often with overlap. Easy to implement, but semantically insensitive.
- Semantic Chunking: Boundaries at semantic breaks, such as paragraphs, chapter headings, or changes in topic.
- Hierarchical Chunking: Documents are broken down into multiple levels—broad section chunks and finer sub-chunks—and linked contextually.
- Format-Aware Chunking: Structural information is taken into account for tables, lists, and Markdown.
Chunking, Reranking, and Knowledge AI
Chunking is just the first step. This is followed by embedding, vector search, and often a re-ranking step that sorts the top results by relevance once again. Only the combination of these steps results in an efficient knowledge AI that ensures that voicebotsand chatbots provide factually accurate responses.
Practical Tips for Stable Chunks
In practice, a balanced mix works best. Experience shows that Markdown-optimized content with clear headings, organized into section chunks of a few hundred tokens with moderate overlap, provides the best balance between precision and completeness. Tables should be treated as atomic units, while legal texts benefit from chunking by paragraph. Iterative tuning is important, accompanied by hard evaluation metrics such as hit rate, NDCG, and response quality.
Frequently Asked Questions (FAQ)
There is no one-size-fits-all answer. A good starting point is to use chunks of a few hundred tokens with some overlap. Iteration based on actual search quality is crucial.
Responses lose accuracy, RAG hits become unreliable, and the risk of hallucinations increases noticeably.
Each chunk is converted into an embedding and stored in a vector database. The quality of the chunk therefore directly determines the informative value of the embeddings.
--> Back to BOTwiki
Word Error Rate (WER)
--> Go to BOTwiki
The Word Error Rate (WER) is the key metric for measuring the quality of speech-to-text systems. It indicates how many words in a spoken sentence were transcribed incorrectly by the recognition system, expressed as a percentage of the total number of spoken words. A low WER is a prerequisite for reliable voicebots, because every recognition error subsequently degrades the classification of the request, entity extraction, and thus end-to-end automation.
How the Word Error Rate is Calculated
The WER sums up three types of errors and compares them to the length of the reference text:
- Substitutions (S): A word has been replaced by another.
- Insertions (I): An additional word has been inserted.
- Deletions (D): A word is missing from the transcription.
The formula is WER = (S + I + D) / N, where N is the number of words in the reference text. For example, a WER value of 5% means that one word was incorrectly recognized in a 20-word sentence.
WER and Its Impact on Voice Bots
In the voice channel, the WER directly impacts subsequent steps. If the system misidentifies a customer number or a plan name, the entire workflow fails. That is why the WER is not just a quality metric, but an input variable for multi-agent orchestration: When confidence is low, the triage agent specifically requests a repeat or compares the audio text with stored custom entities.
WHO for proper nouns, numbers, and technical terms
The average error rate (ER) of modern speech-to-text systems for standard conversations is in the low single digits. For proper nouns, addresses, numbers, or industry-specific terminology, it is often significantly higher—unfortunately, precisely where it is most critical for service processes. Custom vocabularies, industry-specific language models, and downstream plausibility checks via phonebots provide a solution.
Frequently Asked Questions (FAQ)
In the context of dictation, an error rate (ER) below 5% is considered very good. In the service sector, where proper names, addresses, and numbers are involved, realistic target rates vary by industry—the key is to ensure that critical data points (customer number, address, amount) are accurate.
Any gap in recognition leads either to follow-up questions or to escalation to employees. Both reduce the automation rate. A low WER is therefore a direct driver of ROI.
No. WER is a necessary but not sufficient condition. It is only through the combination of multi-agent orchestration, hybrid intelligence, and knowledge AI that a good transcript can be transformed into a robust service process.
--> Back to BOTwiki
Triage Agent (Routing Agent)
--> Go to BOTwiki
A triage agent is the first step in a multi-agent orchestration. It interprets the user’s or caller’s free-form input, classifies the inquiry into a specific category, and forwards the dialogue to a specialized process agent. Typically, the triage agent also has access to the central knowledge base, allowing it to answer general questions directly.
The triage agent solves the core problem of traditional IVR logic (“Press 1 for …”) and simple single-prompt tools: it separates understanding and execution into two distinct steps, thereby creating the stability that an AI agent needs in business-critical processes.
Unlike simple intent recognition, the triage agent also makes routing decisions based on urgency, escalation rules, and available capacity.
- Classification of the issue into a subject category.
- Review of escalation rules (e.g., emergency vs. standard).
- Handover to specialized process agents, including parameters that have already been extracted or workflow triggers.
- Handover to a human agent if the issue falls outside the defined domains.
Triage Agent on the Voice Hotline
In a traditional service center with a high call volume, the triage agent is the key to true AI-native voice automation. The caller freely describes their issue (“Water is dripping from the ceiling”), and the voicebot recognizes the damage report, classifies it as an urgent housing issue, and routes it directly to the responsible process agent for damage assessment. Body and brain are clearly separated in this process: The telephony infrastructure (SIP, PSTN) provides the audio signal, while the triage agent handles the cognitive classification.
Best Practices for Implementation
A clear definition of domains is crucial for a robust triage agent. For each use case, clear classes should be defined, complete with examples, synonyms, and negative examples. Robust fallback behavior is also important: If the triage agent cannot clearly identify a request, it asks specific follow-up questions instead of guessing. This ensures that conversations remain brand-safe and lead to transparent outcomes—a key advantage over monolithic prompts.
Frequently Asked Questions (FAQ)
An IVR system relies on rigid voice menus (“Press 1 for …”) and struggles with natural language, ambiguity, or changes in context. A triage agent understands the request as it is freely expressed, classifies it, and routes it based on context. This makes it the “brain” layer that traditional IVR platforms cannot provide.
It serves as the central entry point for every conversation. It understands and classifies the data, while specialized process agents or workflows handle the execution. This results in clear responsibilities, stable processes, and auditable logs.
Whenever the issue falls outside the defined domains, an emotional factor comes into play, or regulatory requirements call for human review. The handoff occurs seamlessly, with the full context of the conversation preserved, so the user doesn't have to repeat themselves.
--> Back to BOTwiki
AI Agent Prebuilds
--> Go to BOTwiki
AI Agent Prebuilts are pre-configured sets of optimized instruction prompts, knowledge contexts, and behavioral guidelines (personas) that can be used directly for typical use cases on an AI agent platform.
They work like modular starter kits: instead of defining every agent dialogue and persona from scratch, you import an existing building block and adapt it to the specific use case. In BOTfriends X, this is made easier by the versioning feature, which allows versions to be exported and imported.
As a result, prebuilts significantly shorten the design and implementation phase of an AI agent and reduce the effort required to manage complex, generative interactions. For B2B projects where time-to-value is critical, prebuilts are an indispensable accelerator for building conversational AI based on large language models (LLMs).
Building a prebuilt
A modern prebuilt bundles various generative components into a reusable unit. Unlike traditional modeling of individual questions, the focus here is on the instruction prompt design and context control.
Typical components of an AI agent prebuild include:
-
Instruction Prompts & Personas: Predefined character descriptions and behavioral guidelines that determine how the agent behaves and interacts.
-
Agent-based workflows: Predefined processes for complex tasks (e.g., problem-solving steps) that help agents pursue goals autonomously.
-
Tool definitions: Interface templates (function calls) that allow the agent to perform actions directly in third-party systems.
Typical use cases
Prebuilts are used wherever established interaction patterns need to be covered before company-specific customization begins. Classic examples include professional welcome scenarios, identification and verification processes, automated appointment scheduling, and support assistants for specific industries. In practice, companies combine multiple prebuilts to, for example, link a “reception agent” with a specialized “support agent.”
Even though pre-built models are based on modern prompts, they need to be fine-tuned before going live. The predefined personas and knowledge sources are replaced by what is known as prompt engineering and specific company data to ensure that the tone and expertise align perfectly with the brand.
Prebuilt configurations in multi-agent setups
In modern architectures, prebuilts are used as building blocks within a multi-agent orchestration. In this context, specialized agents work together as a team: an authentication agent is set up using a corresponding template, while a product advisor agent is added using another template, before both are customized to the company’s needs. A higher-level instance controls the flow of context between these units.
This modular approach makes it possible to quickly scale standard tasks using proven pre-built components, while still leaving room for highly customized, data-driven AI applications.
Frequently Asked Questions (FAQ)
These are pre-built collections of system prompts, personas, and agent-based dialogue structures. They serve as a starting point for modern AI projects, allowing developers to draw on proven best practices in generative AI rather than having to rewrite every interaction logic from scratch.
No. Pre-built models provide the foundation and structure. Custom prompt engineering then ensures that the AI agent perfectly masters your company’s specific terminology, processes, and unique voice.
In such setups, prebuilds serve as “role profiles” for individual agents. You can think of them as job descriptions: a prebuild defines what an agent can do and how it behaves, so that it can be seamlessly integrated into a larger system consisting of multiple cooperating AI units.
--> Back to BOTwiki
AI Agent Operations
--> Go to BOTwiki
AI Agent Operations refers to the day-to-day operation of an AI agent after it goes live. In other words, it encompasses everything that happens between the initial go-live and ongoing scaling. The term covers monitoring, training, content maintenance, analytics, and governance throughout the entire lifecycle.
As soon as real users begin interacting with the system, opportunities for optimization emerge that could not have been anticipated during the design phase. It is only through structured operations that a functional prototype becomes a productive conversational AI solution. As such, AI agent operations is less of a technical feature and more of an ongoing discipline that applies equally to voice, chat, and email channels.
Short-term optimization after go-live
Immediately after going live, conversations should be reviewed daily. The focus should be on miscategorized topics, inappropriate communication styles, and gaps in the content. As the team becomes more experienced, the frequency of these reviews can be reduced, but they can never be eliminated entirely.
Typical measures include refining the instruction prompts or AI persona and updating technical answers in the knowledge base. These small adjustments determine the quality of recognition and prevent weaknesses in the model from becoming entrenched.
Long-term strategic optimization
In addition to day-to-day tuning, AI Agent Operations also involves a strategic component. This includes evaluating the channels and messaging platforms in use, assessing existing services based on technical performance, and identifying new use cases for meaningful expansion.
-
Evaluation of the channels and interfaces used.
-
Analysis of service performance by use case and target group.
-
Identifying new topics and processes for the roadmap.
-
Planning for multilingual support and geographic rollout.
Implications for Voice and Chat
In voice channels—such as a voicebot used for triage on a hotline—the operational burden is particularly high. Speech-to-text systems produce systematic recognition errors, accents and background noise skew confidence scores, and wait times make every flaw audible.
In the chat and email channels, the focus is more on content maintenance and integration with Knowledge AI. FAQ content changes, products are replaced, and legal texts are updated.
AI Agent Operations ensures that these changes are applied consistently across the platform and do not result in inconsistent responses across channels.
Analytics and Governance During Operations
Without reliable analytics, AI agent operations are like flying blind. What’s needed are clearly defined KPIs —such as detection rate, fallback rate, resolution rate, and handover rate to human agents—as well as reporting that makes these metrics visible by channel and use case. Only by linking conversation data to business objectives can you prioritize the backlog effectively.
Frequently Asked Questions (FAQ)
AI Agent Operations refers to the day-to-day operation of an AI agent after it goes live. This includes monitoring conversations, continuous training, content maintenance, analytics, and governance. The goal is to maintain high recognition quality and to integrate new requirements from day-to-day business into the solution in a structured manner.
During the first few weeks, conversations are reviewed daily. The instruction prompts and AI personas are refined, and factually incorrect answers in the knowledge base are updated. As the model matures, the frequency of reviews can be reduced without completely abandoning the review process.
Key metrics include detection rate, fallback rate, completion rate, average call duration, and the transfer rate to human agents. Depending on the use case, additional business-specific KPIs such as completed bookings or resolution rates may also be included. It is important that these metrics are reported separately for each channel (voice, chat, email).
Voice-based operations must also take acoustic factors into account: speech-to-text errors, accents, background noise, and latency. In chat and email channels, the focus is more on content management and knowledge integration. However, both channels require the same governance and analytics framework to ensure that information remains consistent across all channels.
--> Back to BOTwiki
Entity
--> Go to BOTwiki
An entity—sometimes also referred to as a slot or parameter—is a specific data point that a voicebot or chatbot extracts from a user’s input. While an intent describes what the caller or user wants to achieve, entities answer the supplementary questions of when, where, what, and how much. Typical examples include dates, locations, product names, quantities, or customer numbers.
Entities originated in an era when conversational AI systems still relied on fixed utterances—that is, predefined template sentences—which the system used to identify a user’s intent. In this rule-based environment, entities ensured that relevant keywords were reliably extracted from an input, regardless of how precisely someone phrased a sentence. They served as the link between rigid intent recognition and the variable information that made each query unique.
Today, things are different. Thanks to agent-based logic and large language models, intent recognition in AI agents has become significantly more flexible, and with a well-crafted prompt, the agent can identify relevant data points without the need for explicitly defined entities. Nevertheless, the concept remains useful as a conceptual model: those who understand what information an assistant needs for a specific task can develop better workflows, clearer prompts, and more robust systems.
System Entities and Custom Entities
In NLU models, a distinction is made between two basic types. System entities are pre-trained data types that are already included in the platform and can be recognized without further training. Custom entities, on the other hand, are defined by the business department and populated with domain-specific values. For example, a list of all product variants, branches, or internal transaction types.
- System Entities: Date, Time, Number, Currency, Address, Email, Phone Number.
- Custom Entities: Product names, plan names, contract types, internal status codes.
- Composite Entities: Combinations of multiple values, such as pizza size and toppings.
How to extract entities from input
With every input, the NLU model parses the text and, in parallel with intent recognition, matches it against the stored entity definitions. The models use vector representations and contextual information, ensuring that even synonyms or misspellings are reliably identified.
Here is a typical example of how user requests were processed not too long ago: If a caller says over the phone, “I’d like to book an appointment at the Munich branch tomorrow at 2 p.m.,” the system extracts the entities date (tomorrow), time (2 p.m.), and branch (Munich). Based on this, the downstream workflow checks availability and booking rules.
Frequently Asked Questions (FAQ)
An entity is a specific data point that an NLU model extracts from a user’s input. Typical examples include dates, times, locations, product names, or customer numbers. While the intent describes the user’s request, entities provide the specific parameters that an AI agent needs to actually carry out the task.
The intent describes what a user wants to achieve—such as booking an appointment. The entity provides the relevant details, such as the date, time, or location. Both are recognized simultaneously by the NLU model and processed together in the dialogue workflow.
Custom entities reflect a company's business vocabulary—from product names to internal transaction types. They are the key to turning a chatbot or phonebot into a domain-specific solution.
--> Back to BOTwiki