Today, I'm exploring how ChatGPT can be utilized in software testing.
If you're a new tester or it will be your first time to create this and there's no existing template in your team, this is a good starting point. ChatGPT gave me the list of topics and what kind of information I should provide on each of them. You can combine this + other knowledge sources you find to create the document.
It covers positive and negative cases and even web responsiveness. I think this is a good basic coverage of a Login test. It's still the human tester's responsibility to review this and check what else is missing or if there's anything that needs an update.
I actually expected it to return it in a table format so that I can copy-paste it in an Excel sheet and then upload it to Jira. This is a fail.
With a few tweaks to the message, it was able to provide me the test case in a table format. Yay!
I'm actually impressed that they're able to identify the positive and negative tests.
To further check its capabilities in identifying the test case attributes, I added a few more requests and frankly, I'm quite satisfied.
it said that it's not a good candidate for a regression test because it might not be stable to test repeatedly.
It said that the criticality of a feature does not determine whether a test case is a good candidate for regression testing.
Then I asked why do they recommend it for automation when it's not a good regression test? Generally, the best test cases to automate are regression test cases. And then it corrected itself by saying that it's probably correct that it shouldn't be an automation candidate as well.
What is ChatGPT? ChatGPT is an app built by OpenAI. You can ask questions, ask them to provide you documentation, or hold a conversation. It's been talked about how AI will soon replace the manual work that we can do and take over our jobs. Do I believe that's possible? Probably, but that will probably take several years.
I explored a bit how ChatGPT can be used for software testing and if it can really replace the majority of the manual tasks such as creating test plans and test cases. I think it did fairly well. I'm not sure yet if it can be trained when you provide it some detailed requirements so that it can provide you custom test cases based on your needs, but I don't recommend this since this means you'll feed confidential information about your company's products.
You can check below how I used ChatGPT. It was fun to see the answers it gave me. We had a bit of back and forth at the end when discussed a specific test case's attribute. Let me know what you think about how it does!
Create a test plan and strategy document
If you're a new tester or it will be your first time to create this and there's no existing template in your team, this is a good starting point. ChatGPT gave me the list of topics and what kind of information I should provide on each of them. You can combine this + other knowledge sources you find to create the document.
Create login test cases for https://twitter.com/
It covers positive and negative cases and even web responsiveness. I think this is a good basic coverage of a Login test. It's still the human tester's responsibility to review this and check what else is missing or if there's anything that needs an update.
Format these test cases so I can upload them to Jira
I actually expected it to return it in a table format so that I can copy-paste it in an Excel sheet and then upload it to Jira. This is a fail.
Prepare the test cases in a table that I can upload to Jira
With a few tweaks to the message, it was able to provide me the test case in a table format. Yay!
Prepare the test cases in a table that I can upload in Jira and add a column to identify if they're a positive or negative test
I'm actually impressed that they're able to identify the positive and negative tests.
Prepare the test cases in a table that I can upload in Jira and add the ff. columns:
1. Identify if they're a positive or negative test
2. Identify their test classification
3. Identify their test type
4. Identify if they're a good regression testing candidate
5. Identify if they're a good candidate for test automation
To further check its capabilities in identifying the test case attributes, I added a few more requests and frankly, I'm quite satisfied.
However, we had a few back and forths when I asked about a specific test case where I'm not quite sure how it decided that it's not suitable as a regression test. This is TC05:
Why is TC05 not a good regression candidate?
it said that it's not a good candidate for a regression test because it might not be stable to test repeatedly.
Wouldn't you consider TC05 as a critical feature?
It said that the criticality of a feature does not determine whether a test case is a good candidate for regression testing.
Then how come you considered TC05 as a good automation candidate when it's not good for regression?
Then I asked why do they recommend it for automation when it's not a good regression test? Generally, the best test cases to automate are regression test cases. And then it corrected itself by saying that it's probably correct that it shouldn't be an automation candidate as well.
My current conclusion:
ChatGPT can be used in a way where it can help you get started on what to test or what information you have to provide. It only returns you generic solutions, which are not bad because there's like a set of universal test cases anyway for a common feature like `Login`. It saves you time because instead of putting your effort into remembering and writing these universal test cases, you can focus more on custom tests that are applicable to your company. It also helps start a conversation regarding how we identify the test cases we have.
Another takeaway is that its response also depends on how well you ask the question.
I recommend using ChatGPT especially if you're still exploring software testing. In terms of real-world application, At the end of the day, the final decision still falls on you and your team.