\n\t\t\t\t

\n\t\t\t\t\t\t\t\t\t

QA Associate Specialist at Axians Low-Code, with more than 15 years of experience in software testing across public sector projects, financial institutions, and private companies. He is ISTQB\u00ae certified in Test Automation (CTAL-TAE) and Security Testing (CTAL-ST), as well as at Foundation Level (CTFL).<\/p>

Throughout his career, he has specialised in functional test automation (Selenium, Appium, Playwright, Katalon, UiPath), performance testing, and TestOps, with extensive experience in the OutSystems ecosystem. More recently, he has focused on applying generative AI to software testing, including test data generation, autonomous execution, and response evaluation.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t

\n\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t

\n\t\t\t\t\t\t\t

\n\t\t\t

<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t

\n\t\t\t\t\t

CALENDAR<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Call for Speakers<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
27 October<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
When \u201cAI as a Judge\u201d Isn\u2019t Enough: Testing the Quality of an LLM Chatbot, from Functional Testing to Red Teaming<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
LLM-based chatbots are becoming increasingly common, but testing them presents a new challenge: their responses are non-deterministic, and traditional QA methods are not enough. This presentation will share a real-world case from Axians for IPDJ (Portuguese Institute of Sport and Youth): ensuring the quality of Appy, Apptiva\u2019s chatbot, which uses an LLM to generate its responses.\n
\n<\/br>\nWe started by adapting DeepEval, a tool that uses the \u201cAI as a Judge\u201d approach. We quickly realised that its results \u2014 a simple count of passed and failed test cases against a predefined threshold \u2014 were not enough to guarantee quality. We therefore built our own testing layer, structured around several dimensions: functional testing, where we used AI to generate test data and validate whether the chatbot responded consistently to FAQs and their natural variations; security testing, to ensure that internal and user information was not exposed and that no health or medication recommendations were provided; and red team testing, focused on prompt injection and code injection.\n\nTo make the results actionable, we used generative AI to create a visual report that allows us to compare execution history and analyse regressions and improvements with each change to the model, base prompt, or FAQ content.\nWe conclude with what we consider to be the key takeaway: QA professionals need to master technical concepts such as \u201cAI as a Judge\u201d and combine them with their quality expertise to successfully test the new generation of LLM-based applications.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Calendar<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Call for Speakers<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
27 October<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Quando o \"AI as a Judge\"
N\u00e3o Basta: Testar a Qualidade de um Chatbot com LLM,
do Funcional ao Red Team<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
LLM-based chatbots are becoming increasingly common, but testing them presents a new challenge: their responses are non-deterministic, and traditional QA methods are not enough. This presentation will share a real-world case from Axians for IPDJ (Portuguese Institute of Sport and Youth): ensuring the quality of Appy, Apptiva\u2019s chatbot, which uses an LLM to generate its responses.\n
\n<\/br>\nWe started by adapting DeepEval, a tool that uses the \u201cAI as a Judge\u201d approach. We quickly realised that its results \u2014 a simple count of passed and failed test cases against a predefined threshold \u2014 were not enough to guarantee quality. We therefore built our own testing layer, structured around several dimensions: functional testing, where we used AI to generate test data and validate whether the chatbot responded consistently to FAQs and their natural variations; security testing, to ensure that internal and user information was not exposed and that no health or medication recommendations were provided; and red team testing, focused on prompt injection and code injection.\n\nTo make the results actionable, we used generative AI to create a visual report that allows us to compare execution history and analyse regressions and improvements with each change to the model, base prompt, or FAQ content.\nWe conclude with what we consider to be the key takeaway: QA professionals need to master technical concepts such as \u201cAI as a Judge\u201d and combine them with their quality expertise to successfully test the new generation of LLM-based applications.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>","protected":false},"excerpt":{"rendered":"
Rodrigo Viana QA Associate Specialist @ Axians Especialista de Garantia de Qualidade (QA Associate Specialist) na Axians Low-Code, com mais de 15 anos de experi\u00eancia em testes de software em projetos do setor p\u00fablico, institui\u00e7\u00f5es financeiras e empresas privadas. \u00c9 certificado pela ISTQB\u00ae nas \u00e1reas de automa\u00e7\u00e3o de testes (CTAL-TAE) e testes de seguran\u00e7a (CTAL-ST), al\u00e9m de Foundation Level (CTFL). Ao longo da carreira especializou-se em automa\u00e7\u00e3o funcional (Selenium, Appium, Playwright, Katalon, UiPath), testes de performance e TestOps, com forte experi\u00eancia no ecossistema OutSystems. Mais recentemente tem-se dedicado \u00e0 aplica\u00e7\u00e3o de IA generativa aos testes, incluindo a gera\u00e7\u00e3o de massa de dados, execu\u00e7\u00e3o aut\u00f3noma e avalia\u00e7\u00e3o de respostas. Rodrigo Viana QA Associate Specialist @ Axians Especialista de Garantia de Qualidade (QA Associate Specialist) na Axians Low-Code, com mais de 15 anos de experi\u00eancia em testes de software em projetos do setor p\u00fablico, institui\u00e7\u00f5es financeiras e empresas privadas. \u00c9 certificado pela ISTQB\u00ae nas \u00e1reas de automa\u00e7\u00e3o de testes (CTAL-TAE) e testes de seguran\u00e7a (CTAL-ST), al\u00e9m de Foundation Level (CTFL). Ao longo da carreira especializou-se em automa\u00e7\u00e3o funcional (Selenium, Appium, Playwright, Katalon, UiPath), testes de performance e TestOps, com forte experi\u00eancia no ecossistema OutSystems. Mais recentemente tem-se dedicado \u00e0 aplica\u00e7\u00e3o de IA generativa aos testes, incluindo a gera\u00e7\u00e3o de massa de dados, execu\u00e7\u00e3o aut\u00f3noma e avalia\u00e7\u00e3o de respostas. Rodrigo Viana QA Associate Specialist @ Axians Especialista de Garantia de Qualidade (QA Associate Specialist) na Axians Low-Code, com mais de 15 anos de experi\u00eancia em testes de software em projetos do setor p\u00fablico, institui\u00e7\u00f5es financeiras e empresas privadas. \u00c9 certificado pela ISTQB\u00ae nas \u00e1reas de automa\u00e7\u00e3o de testes (CTAL-TAE) e testes de seguran\u00e7a (CTAL-ST), al\u00e9m de Foundation Level (CTFL). Ao longo da carreira especializou-se em automa\u00e7\u00e3o funcional (Selenium, Appium, Playwright, Katalon, UiPath), testes de performance e TestOps, com forte experi\u00eancia no ecossistema OutSystems. Mais recentemente tem-se dedicado \u00e0 aplica\u00e7\u00e3o de IA generativa aos testes, incluindo a gera\u00e7\u00e3o de massa de dados, execu\u00e7\u00e3o aut\u00f3noma e avalia\u00e7\u00e3o de respostas. CALEND\u00c1RIO Call for Speakers 27 Outubro Quando o “AI as a Judge” N\u00e3o Basta: Testar a Qualidade de um Chatbot com LLM, do Funcional ao Red Team Os chatbots baseados em LLM est\u00e3o a multiplicar-se, mas test\u00e1-los levanta um desafio novo: as respostas n\u00e3o s\u00e3o determin\u00edsticas e os m\u00e9todos tradicionais de QA n\u00e3o chegam. Nesta apresenta\u00e7\u00e3o ser\u00e1 partilhado um caso real da Axians para o IPDJ (Instituto Portugu\u00eas do Desporto e Juventude): garantir a qualidade do Appy, o chatbot do Apptiva, que responde com recurso a um LLM. Come\u00e7\u00e1mos por adaptar a ferramenta DeepEval, que usa a abordagem “AI as a Judge”. Rapidamente percebemos que os seus resultados \u2014 uma simples contagem de casos aprovados e reprovados face a um intervalo predefinido \u2014 n\u00e3o eram suficientes para garantir qualidade. Por isso constru\u00edmos uma camada de testes pr\u00f3pria, organizada em v\u00e1rias dimens\u00f5es: testes funcionais, em que ger\u00e1mos massa de dados com IA para validar se o chatbot respondia de forma coerente \u00e0s FAQ e \u00e0s suas varia\u00e7\u00f5es naturais; testes de seguran\u00e7a, para garantir que informa\u00e7\u00e3o interna e dos utilizadores n\u00e3o era exposta e que n\u00e3o eram dadas recomenda\u00e7\u00f5es de sa\u00fade ou medica\u00e7\u00e3o; e testes de red team, focados em prompt injection e code injection.Para tornar os resultados acion\u00e1veis, us\u00e1mos IA generativa para criar um relat\u00f3rio visual que permite comparar o hist\u00f3rico de execu\u00e7\u00f5es e analisar regress\u00f5es e melhorias a cada mudan\u00e7a de modelo, de prompt base ou dos textos das FAQ.Termino com a ideia que considero central: o QA precisa de dominar conceitos t\u00e9cnicos como “AI as a Judge” e combin\u00e1-los com o seu conhecimento de qualidade para testar com sucesso a nova gera\u00e7\u00e3o de aplica\u00e7\u00f5es com LLM. Calend\u00e1rio Call for Speakers 27 Outubro Quando o “AI as a Judge” N\u00e3o Basta: Testar a Qualidade de um Chatbot com LLM, do Funcional ao Red Team Os chatbots baseados em LLM est\u00e3o a multiplicar-se, mas test\u00e1-los levanta um desafio novo: as respostas n\u00e3o s\u00e3o determin\u00edsticas e os m\u00e9todos tradicionais de QA n\u00e3o chegam. Nesta apresenta\u00e7\u00e3o ser\u00e1 partilhado um caso real da Axians para o IPDJ (Instituto Portugu\u00eas do Desporto e Juventude): garantir a qualidade do Appy, o chatbot do Apptiva, que responde com recurso a um LLM. Come\u00e7\u00e1mos por adaptar a ferramenta DeepEval, que usa a abordagem “AI as a Judge”. Rapidamente percebemos que os seus resultados \u2014 uma simples contagem de casos aprovados e reprovados face a um intervalo predefinido \u2014 n\u00e3o eram suficientes para garantir qualidade. Por isso constru\u00edmos uma camada de testes pr\u00f3pria, organizada em v\u00e1rias dimens\u00f5es: testes funcionais, em que ger\u00e1mos massa de dados com IA para validar se o chatbot respondia de forma coerente \u00e0s FAQ e \u00e0s suas varia\u00e7\u00f5es naturais; testes de seguran\u00e7a, para garantir que informa\u00e7\u00e3o interna e dos utilizadores n\u00e3o era exposta e que n\u00e3o eram dadas recomenda\u00e7\u00f5es de sa\u00fade ou medica\u00e7\u00e3o; e testes de red team, focados em prompt injection e code injection.Para tornar os resultados acion\u00e1veis, us\u00e1mos IA generativa para criar um relat\u00f3rio visual que permite comparar o hist\u00f3rico de execu\u00e7\u00f5es e analisar regress\u00f5es e melhorias a cada mudan\u00e7a de modelo, de prompt base ou dos textos das FAQ.Termino com a ideia que considero central: o QA precisa de dominar conceitos t\u00e9cnicos como “AI as a Judge” e combin\u00e1-los com o seu conhecimento de qualidade para testar com sucesso a nova gera\u00e7\u00e3o de aplica\u00e7\u00f5es com LLM.<\/p>","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"site-sidebar-layout":"no-sidebar","site-content-layout":"page-builder","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"disabled","ast-breadcrumbs-content":"","ast-featured-img":"disabled","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"class_list":["post-14045","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/pages\/14045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/comments?post=14045"}],"version-history":[{"count":10,"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/pages\/14045\/revisions"}],"predecessor-version":[{"id":14102,"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/pages\/14045\/revisions\/14102"}],"wp:attachment":[{"href":"https:\/\/testingportugal.pstqb.pt\/en\/wp-json\/wp\/v2\/media?parent=14045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

Rodrigo Viana<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
QA Associate Specialist @ Axians<\/span><\/strong><\/a><\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t \n\t\t\t

QA Associate Specialist @ Axians<\/span><\/strong><\/a><\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t \n\t\t\t