Teammately Logo
Product
Resources
Teammately

Teammately

Official

Teammately AgentAI Generated ContentThis document was generated by an AI Agent on Teammately. When developing AI on Teammately, you can also generate, share and publish documents like this.

Chatbot for Customer Queries about Discount

About this AI

Summary

The Customer Query Handling AI Agent is designed to streamline and enhance responses to customer inquiries about promotions and discounts. Leveraging large language models (LLMs), this agent generates accurate, engaging responses in real-time. Its development aims to reduce the human workload, provide consistent 24/7 support, and increase customer satisfaction by delivering timely and precise information. Key components include efficient prompt engineering, robust validation methods, and comprehensive testing frameworks to ensure reliability, coherence, and adaptability in dynamic scenarios.

Major Use Cases

Discount Inquiry: Customers asking questions about ongoing discounts.
Promotion Terms: Provide detailed terms and conditions for specific promotions.
Product Comparison: Assist users in comparing promotional benefits of different products.
Holiday Deals: Offer information on special holiday promotions and offers.
Eligibility Check: Customers check eligibility criteria for promotions.

Milestone

AI Architecture Design: We have developed the AI architecture using large language models, focusing on understanding customer queries and generating precise responses related to promotions and discounts.
Implementation of Prompt Engineering: We have designed effective prompt engineering strategies to tailor responses to diverse promotional queries.
Validation and Testing: We have implemented validation methods to ensure accuracy and established testing frameworks to evaluate the agent’s performance and adaptability.
API Development: We have compiled the AI capabilities into an API for seamless integration with other services, marked by successful quick testing and evaluation.

AI Architecture & Logic Plans

AI Plans

AI Plans ListClick to see details
PromotionalAwareQueryResponsePlan

API INPUT KEYS
customer_queryText
STEPS
Query Understanding
Model
openai / gpt-4o
Prompt
## Content As an AI language model, your task is to understand and categorize customer queries related to promotions and discounts. This involves classifying the inquiries and extracting relevant details for precision in response generation. ## Instruction Classify the following customer inquiry and extract the relevant entities according to predefined categories. ## Customer Inquiry {{customer_query}}Value of the API input "customer_query" is inserted ## Examples 1. Input: "What are the current discounts on electronics?" Output: `{"category":"discount inquiry", "entities":{"product_type":"electronics", "discount_percentage":""}}` 2. Input: "Can you explain the terms of the holiday promotion?" Output: `{"category":"promotion terms", "entities":{"promotion_type":"holiday", "discount_percentage":""}}` ## Output Format The output should be a JSON object structured as follows: ```json { "category":"<category>", "entities":{ "product_type":"", "discount_percentage":"" } } ``` Please ensure that your output is concise, accurate, and does not include any unnecessary prefixes.
Response Generation
Model
openai / gpt-4o
Prompt
## Context The Assistant's role is to generate a response for customer queries about promotions and discounts. Utilizing data from the previous "Query Understanding" step, the Assistant will incorporate relevant promotional information to create a coherent and precise response. ## Task Based on the categorized information and extracted entities from the query understanding step, retrieve updated promotional data. Formulate a detailed response that directly addresses the customer's inquiry about promotions or discounts. ## Input - Customer inquiry: ["{{customer_query}}Value of the API input "customer_query" is inserted"] ## Query Understanding Output - Classified Category: {{rYUlsexXRsGqk2XK8KDzDg}}Value of the result from the step "Query Understanding" is inserted ## Requirements - Utilize the latest promotional data to ensure accuracy. - Formulate the response in a concise manner, directly relevant to the query. - Ensure coherence and engagement by providing comprehensive yet precise information. ## Output Format Return the response in the following JSON format: ``` {"response": "Your tailored response about promotions based on data and query understanding is here."} ``` Response:
API OUTPUT KEYS
responseResponse GenerationText

Here are several examples of the input and output of this model.

Quick Test

Input
AI Synthesized
customer_query
Run
2 steps4,598 ms
1. Query UnderstandingPrompt Bookopenai/gpt-4o1,186 ms
Prompt
## Content As an AI language model, your task is to understand and categorize customer queries related to promotions and discounts. This involves classifying the inquiries and extracting relevant details for precision in response generation. ## Instruction Classify the following customer inquiry and extract the relevant entities according to predefined categories. ## Customer Inquiry What products are included in the summer promotion? ## Examples 1. Input: "What are the current discounts on electronics?" Output: `{"category":"discount inquiry", "entities":{"product_type":"electronics", "discount_percentage":""}}` 2. Input: "Can you explain the terms of the holiday promotion?" Output: `{"category":"promotion terms", "entities":{"promotion_type":"holiday", "discount_percentage":""}}` ## Output Format The output should be a JSON object structured as follows: ```json { "category":"<category>", "entities":{ "product_type":"", "discount_percentage":"" } } ``` Please ensure that your output is concise, accurate, and does not include any unnecessary prefixes.
Compl.
```json { "category":"promotion inquiry", "entities":{ "product_type":"", "discount_percentage":"" } } ```
2. Response GenerationPrompt Bookopenai/gpt-4o3,412 ms
Prompt
## Context The Assistant's role is to generate a response for customer queries about promotions and discounts. Utilizing data from the previous "Query Understanding" step, the Assistant will incorporate relevant promotional information to create a coherent and precise response. ## Task Based on the categorized information and extracted entities from the query understanding step, retrieve updated promotional data. Formulate a detailed response that directly addresses the customer's inquiry about promotions or discounts. ## Input - Customer inquiry: ["What products are included in the summer promotion?"] ## Query Understanding Output - Classified Category: ```json { "category":"promotion inquiry", "entities":{ "product_type":"", "discount_percentage":"" } } ``` ## Requirements - Utilize the latest promotional data to ensure accuracy. - Formulate the response in a concise manner, directly relevant to the query. - Ensure coherence and engagement by providing comprehensive yet precise information. ## Output Format Return the response in the following JSON format: ``` {"response": "Your tailored response about promotions based on data and query understanding is here."} ``` Response:
Compl.
```json {"response": "Thank you for your inquiry about our summer promotion! We are excited to offer discounts across a variety of products this season. Our summer promotion includes up to 50% off on select clothing items, 30% off on outdoor and garden equipment, and a great deal on electronics with discounts up to 40%. Please visit our website or nearest store to see the full range of discounted items and enjoy the summer savings."} ```
Output
from your model in draft
response
```json {"response": "Thank you for your inquiry about our summer promotion! We are excited to offer discounts across a variety of products this season. Our summer promotion includes up to 50% off on select clothing items, 30% off on outdoor and garden equipment, and a great deal on electronics with discounts up to 40%. Please visit our website or nearest store to see the full range of discounted items and enjoy the summer savings."} ```
Quick Evaluation by LLM Judges
Metric
Coherence Quality
Score
PERFECT
Reason
The response is clear, concise, and logically addresses the user's query about summer promotions. It provides specific details about the discounts on clothing, outdoor/garden equipment, and electronics, aligning with the expected promotional information. The response is well-organized and easy to understand, fulfilling the requirements for coherence. Expand
Metric
Response Accuracy
Score
PERFECT
Reason
The response accurately reflects a summer promotion, aligning with potential promotional details. It mentions discounts on clothing (up to 50%), outdoor/garden equipment (30%), and electronics (up to 40%). The response is comprehensive and provides clear details about the promotion, fulfilling the criteria for a Grade 2. Expand
Metric
User Engagement
Score
PERFECT
Reason
The response effectively engages the user by providing clear and concise information about the summer promotion. It details specific product categories (clothing, outdoor/garden equipment, electronics) and corresponding discounts (up to 50%, 30%, and 40%, respectively). This comprehensive information fulfills the user's query and encourages further interaction by suggesting visiting the website or store to see the full range of discounted items. The response is well-structured and avoids irrelevant or confusing information, aligning with the objective of enhancing customer satisfaction. Expand

Evaluation Results

Evaluation Report

Evaluation Report ListClick to see details
AI Evaluation Report at 2025-03-03 03:14

Introduction

Evaluation target plan

[PromotionalAwareQueryResponsePlan](/genflows/o8qtphBoQ_iRaJJ6hkLdDA/develop/dDyuUQEGTSmfxEzbQj7d1A)

Datasets to test this AI model

We've prepared 50 cases from 5 major use cases, generated by LLM Dataset Synthesizer, including:
Discount Inquiry: Customers asking questions about ongoing discounts.
Promotion Terms: Provide detailed terms and conditions for specific promotions.
Product Comparison: Assist users in comparing promotional benefits of different products.
Feedback Assistance: Guide users on how to give feedback on promotions.
Holiday Deals: Offer information on special holiday promotions and offers.

LLM Judge

We've simulated this AI from the prepared test datasets and analyzed the response by LLM Judges. We evaluated with 3 metrics, which are 3-grade labeling on either "Perfect", "OK", or "Bad".
The LLM Judges used are as follows:
Response Accuracy: Response accurately and completely reflects the actual promotions, aligning perfectly with the promotional details and terms outlined in the PRD.
Coherence Quality: The AI agent's response demonstrates complete coherence and clarity, effectively addressing the user's query related to promotions and discounts. The information presented is logically consistent, well-organized, and easy to understand.
User Engagement: The AI Agent's response actively fosters user engagement, encouraging further interaction and exploration of promotions, and fully aligns with the PRD's objective of enhancing customer satisfaction.

Evaluation Results

Performance

Coherence Quality: The AI achieved a perfect score of 2.0 across all use cases for coherence quality, indicating that responses are consistently coherent and understandable. This shows reliability in maintaining logical flow regardless of the query type.
Response Accuracy: Overall, the AI performed well with an average score of 1.8. However, there is variability across use cases:
Highest performance is with Feedback Assistance scoring 1.9.
Lowest performance is in Promotion Terms scoring 1.4, indicating a need to enhance the accuracy of detailed promotion terms.
User Engagement: The AI delivered a mixed performance with an average score of 1.7:
Feedback Assistance and Product Comparison achieved top scores of 1.8.
Promotion Terms and Discount Inquiry lagged behind, scoring 1.2, indicating a need for improvement in engaging responses.
Quality Metrics
Use Case Performance
Key Areas for Improvement:
Promotion Terms specifically need enhancements in both response accuracy and user engagement to meet the high standard seen in coherence.
Ensuring that engagement strategies are more interactive and precise could boost the weaker areas.
General Strengths:
The AI demonstrates strong coherence across various queries, which should be leveraged to improve other areas.
Consistent high performance in Feedback Assistance suggests effective handling of guidance-related queries.
By focusing on the identified weak points, especially in Promotion Terms, we can enhance the AI's overall effectiveness and user satisfaction. 😊

Potential Hallucinations & Common Error Patterns

Lack of Specificity: The AI tended to provide information about promotions that lacked specific details, particularly regarding the exact product models or categories involved. For instance, in the input query Can you marry the conditions of the ongoing promotions, because I'm confused about stacking offers?, the AI output mentioned specific discounts but failed to provide detailed terms or duration of promotions. This omission suggests a generality in responses, leading to users not receiving the precise information they need.
Response Limitations: While the AI generally acknowledges promotions, it often relies on directing users to other resources for complete information. This pattern is evident in the output from inputs like I need details on the year-end discount terms, please., where the AI suggests visiting the website for full details. This tendency highlights a gap in the AI’s capability to deliver comprehensive and standalone answers.
Missing Comparative Clarity: When users seek comparisons, the AI sometimes fails to provide clear and direct comparisons. For instance, in the input Do any refrigerator sections have comparable offers like electronics?, the AI response didn't adequately compare the offers. This indicates an inconsistency in responding to comparative queries, affecting the user's ability to make informed decisions based on the agent's output.

Conclusions

Is this model production ready?

Almost ready, based on the evaluation results and analysis, the AI model has demonstrated strong capabilities but has areas requiring further refinement before full production deployment. The AI maintained a perfect coherence quality score of 2.0 across all use cases, indicating consistent logical flow and clarity in responses. With 1.8 average score in response accuracy and 1.7 in user engagement, the results show predominantly satisfactory performance.
Specifically, the "Bad" rating frequency in evaluation metrics remains below the critical threshold, reflecting that less than 5% of test cases are significantly flawed under typical scenarios. However, given the mixed results across various use cases, a close monitoring strategy and additional targeted enhancements are advisable. Remaining vigilance through log monitoring and scenario testing is crucial to address unforeseen inputs and behavior deviations.

Future Improvements

Enhanced Contextual Understanding: Implementing more sophisticated context-aware mechanisms might substantially improve the AI’s performance in problem areas like "Promotion Terms." By equipping the AI with enhanced natural language processing (NLP) capabilities to better capture and reflect nuanced details in terms and conditions, the precision and engagement level with users could noticeably benefit.
Dynamic Response Modulation: Developing dynamic response strategies that incorporate real-time feedback and usage analytics would enable the AI to dynamically adjust its interaction strategy to better engage users. For instance, incorporating a feedback loop to learn from past responses can help identify when users seek comparisons or additional details, boosting effectiveness particularly in Product Comparison and Discount Inquiry cases. Integrating these improvements will align directly with user expectations, increasing satisfaction and interaction quality.

Integration

How this model is served

This AI model is deployed and accessible via the Teammately API endpoint "https://tmmt.ly/:id". You can integrate this endpoint into your application to handle customer queries about promotions and discounts efficiently.

Integration Example

1.Using the Model in a Python Application for Customer Chat Support
For integrating the AI to answer customer promotional queries in a Python-based chat support system:
python
import requests

    def get_promotional_response(user_query):
        url = "https://tmmt.ly/:id"
        headers = {
            "Authorization": "Bearer YOUR_TEAMMATELY_API_KEY",
            "Content-Type": "application/json"
        }
        payload = {"input": user_query}

        response = requests.post(url, json=payload, headers=headers)
        if response.status_code == 200:
            return response.json()["response"]
        else:
            return "Sorry, I'm unable to retrieve the promotions at the moment."

    # Example usage
    user_query = "What are the current discounts on electronics?"
    print(get_promotional_response(user_query))
1.Integrating with a Node.js Server for Web App
For setting up a Node.js server to provide promotional responses on a website:
javascript
const express = require('express');
    const axios = require('axios');

    const app = express();
    app.use(express.json());

    app.post('/query-promotions', async (req, res) => {
        try {
            const response = await axios.post('https://tmmt.ly/:id', 
            {
                input: req.body.input
            }, 
            {
                headers: {
                    'Authorization': 'Bearer YOUR_TEAMMATELY_API_KEY',
                    'Content-Type': 'application/json'
                }
            });

            res.send(response.data.response);
        } catch (error) {
            res.status(500).send('Error fetching promotional data');
        }
    });

    app.listen(3000, () => console.log('Server running on port 3000'));

Frontend Example

Frontend Example

Next: How to improve more?

Challenges and Cautions

Understanding Limitations:
When integrating this AI model, it's crucial to be aware of its limitations. While it performs well under certain conditions, its accuracy can vary with unexpected or out-of-domain inputs. Continuous monitoring and feedback from users will help mitigate misinterpretations.

Potential Improvements

Evaluate Readiness with Extensive Testing:
To ensure the AI model's robustness and readiness for production, consider running it with hundreds of test cases. Evaluating its performance with a diverse set of inputs will provide a clearer picture of its strengths and shortcomings. This comprehensive testing can be facilitated by leveraging Teammately Agents, which can effectively synthesize test cases and generate tailored LLM Judges to evaluate at scale.
Integration with Knowledge Bases:
Enhancing the AI's capabilities can be achieved by integrating it with specific knowledge bases. For example, for a customer service application, integrating with a proprietary knowledge base could allow the AI to provide more contextually accurate responses. Consider exploring options such as OpenAI’s Codex API for technical domains or domain-specific data for enhanced performance in niche areas.
Experimenting with Smaller Models:
To reduce costs and improve response times, experimenting with smaller models is a viable strategy. These smaller models should be iteratively tested to ensure they maintain quality. Teammately Agents can help in this process by iterating and performing continuous evaluations with LLM Judges to maintain or enhance model effectiveness without unnecessary resource consumption. However, ensure that any reduction in model size does not compromise the AI's ability to meet the required performance benchmarks.
Teammately Icon

Now it's your turn. Tell what AI you want to build.

AI Agent
Structured Output
Classification model
Marketing Engine