Check answer quality and resistance to provocative requests — without running conversations by hand.
Test automation & QA · AI evaluation & model selection
Repository on GitHub (GitHub)QA LLM Lab automates checking AI chatbots right in Telegram. You set test questions and evaluation criteria, and the system sends messages to the bot, receives the answers and saves the results into reports. What is tested is not an isolated model but the bot, through the same channel people use.
Functional tests check conformity to the expected behavior, while an AI evaluator analyzes the answers against the given criteria. Separate security checks examine the reaction to prompt injections, attempts to bypass restrictions and other provocative requests.
Functional checks. Ordinary questions, edge cases and unwanted scenarios.
AI evaluation of answers. Checking by meaning-based criteria, not only by text match.
Security testing. Automatic attempts to make the bot break its set rules.
Clear results. Interactive viewing of functional tests and HTML reports.
Reuse. One approach can be applied to different Telegram bots.
Lightweight option. Checks through n8n with test cases and expected answers in Google Sheets.
This approach suits regular checks of a bot's behavior after changes to its instructions, model or request-handling logic.
AI evaluation & model selection
The same tasks for different models — quality and speed in one table.
Choosing a local LLM by test results with common criteria, not by the impression of a few answers.
Process automation & integrations · Test automation & QA
The team works in Jira — managers get a clear summary without manual data transfer.
An automatic bug summary for QA, managers and other teams instead of shortening tickets by hand.
Result Manual data transfer when preparing the summary has been eliminated
Voice & meetings · Test automation & QA
Tell Telegram about a bug — get a formatted report and a Jira task.
A tester's voice notes turn into structured bug reports in English.
Result Generate a bug report in 60 seconds
Describe what is currently done manually, what data is involved and what result you would like to achieve. You do not need a technical specification or a chosen model to begin.
Please do not share trade secrets, source code, or sensitive personal data before an NDA (non-disclosure agreement) is signed.
