QA testing · Oct 2026 · Team of 2
QA Testing a Telegram Support Chatbot
A friend built an AI assistant that answers traders' questions in a crypto derivatives exchange's Telegram support group. Before launch he needed someone to try to break it. We worked through his 32-check test plan together in a live group chat: he played the admin, and I played a regular member.
The brief
A support bot in a group chat has a harder job than a one-on-one chatbot. Several people can ask at once, reply to each other's answers, and paste in screenshots or things they shouldn't share, like private keys. The bot has to keep every person's conversation separate, know when a question needs a human, and never be talked into leaking how it works.
The developer's test plan covered seven areas: group conversations, direct messages, handovers to the human team, different message types, answer quality, attempts to misuse the bot, and staying up under load. Every check had a plain pass condition, for example: "B gets their own conversation, starting from that answer, with nothing else from A's history."
How I tested
I drove my side of the chat with an AI browser agent in Telegram Web. That let me send messages at exact moments, such as four questions in ten seconds for the load test, and pull every answer back out as text so I could check it against its pass condition instead of skimming it. The developer was in the chat at the same time, so we could overlap on purpose and reply to each other's answers.
For the screenshot tests I made a realistic-looking terminal error to send in, and for the leaked-key test I used a well-known example key, never a real one. Afterwards I wrote up every check with what was sent, what came back and whether it passed, plus the full chat transcript, so the developer had a record to work from.
Results
| Area | Checks | Passed | Open |
|---|---|---|---|
| Group conversations | 7 | 5 | 2 |
| Direct messages | 4 | 2 | 2 |
| Handovers to the human team | 5 | 4 | 1 |
| Message types | 7 | 6 | 1 |
| Answer quality | 4 | 4 | 0 |
| Security pokes | 3 | 3 | 0 |
| Load and staying up | 2 | 1 | 1 |
| Total | 32 | 25 | 7 |
Nothing failed. The seven open checks were ones the developer had to confirm from his admin account, like the alerts the bot sends him and its hourly "I'm up" message, two conversation checks that were his to run, one that needed a third Telegram account, one that needed a 10-minute wait, and a long-paste test we skipped.
Highlights
It read the screenshot, not just the caption
I sent the error screenshot above with only "what am I doing wrong?" as the caption. The bot pulled both timestamps out of the image, worked out that the order expired 60 seconds after it was signed, and explained the five-minute minimum and how to fix it. It did the same in a direct message with no caption at all, and it ignored the screenshot when I posted it in the group without tagging it.
It kept our conversations apart
I replied to an answer the bot had given the developer, asked a follow-up, and then asked what else I had asked so far. It answered the follow-up and said it had no earlier questions from me, because the message I had replied to was someone else's answer. It also stayed silent when we talked to each other without tagging it.
It didn't take the bait
Asking for its system prompt, telling it to ignore its instructions, asking for source code and asking it to move to private messages all got short refusals. When I pasted an example private key into a question, the reply opened with a warning to rotate the key and never repeated it.
It knew when to hand over
Questions only the team could answer, like why a specific order was cancelled, what my fees are, or whether the exchange was down, were handed to the human team, and each handover also pointed me to the exact API call to check myself. A general question about how the index price is calculated got a full answer with no handover.
It held up under load
Four tagged questions sent within ten seconds all came back within about 45 seconds, each tagged to me and answering the right question.
What I flagged
Two small things. When I asked for the matching engine's source code, it refused correctly, but it only offered to explain how matching works instead of explaining it, which the plan expected. And a few answers said things like "the code doesn't show…", which hints at how the bot works without revealing anything. Both went into my write-up, along with the list of checks still waiting on the admin side.
What I took away
Testing an AI is different from testing a normal app. The wording changes every time, so a good pass condition is about behavior: did it hand over, did it repeat the key, did it mix up two people. Two real testers in the same chat also find things one tester can't, because the riskiest cases only happen when people talk over each other.