This blog was written by beta specialist, @AndySQ
We beta test new features and experiences to catch and resolve unexpected problems — or bugs in tech lingo — before they go out to everybody. We do extensive internal testing to identify and prioritize fixes, but there’s nothing like the real world to introduce novel scenarios and test from a variety of angles. Beta testers are an essential part of confirming a feature is stable and complete while it’s still in development. The beta team uses bug tickets to document requested changes and to surface improvements, feedback trends, and disruptive bugs.
Using an internal tool called Atlassian Jira, we document, prioritize, troubleshoot, and resolve issues as they arise in our beta programs. Our customer support team also uses JIRA to report and fix issues that occur outside of beta.
Journey of a Bug
1. Report: The bug is discovered
Bugs come to us from many different sources:
- Your posts or replies in the Beta Community
- Your emails to betafeedback@squareup.com
- Your survey responses that call out issues in the comments
- Insight calls or site visits with beta testers
- Your accounts hat show incorrect behavior
- Testing on our demo account
The beta team will record these reports and get a description of the error. At this point it may be a known limitation of the beta feature, an already reported bug that’s being worked on, or an issue unrelated to the beta itself. The beta team will work with our engineers and product team to attribute the issue to the right engineering subteam within Square. If it’s a new bug, we identify the potential impact of the error by outlining when and how it occurs. We call this the reproduction step.
Example Beta Community report:
2. Reproduction: The bug is described in detail and isolated
When we reproduce a bug, we outline the exact order of operations and circumstances that make it occur. First, we try to replicate the same bug on our demo accounts to confirm what the tester has described, and we test a variety of actions to see which one causes an error. At this time we look for workarounds to keep sellers in business while we investigate the issue. If we can’t reproduce the error while testing, we’ll often ask for video, screenshots, or a step-by-step description from the tester who reported the bug. This allows us to get a clear picture as quickly as possible so we can get our engineers working on a solution.
We try to isolate the issue by identifying the exact applications, devices, operating systems, account locations, networks, employee logins, readers, and other hardware on which it occurs. This helps us identify the scale of the impact and prioritize our fix.
Example steps to reproduce:
- Open Square Point of Sale Application (app version 6.56, iOS 17, iPad Air 2022)
- Add “12oz Weissbier” to tab; save open ticket
- Reopen saved ticket later the same day
- Add additional drink item
- Advance to checkout
- Insert or tap card into contactless reader to complete payment (affects all card brands)
- Error “Unable to Complete Payment” appears on screen
- Return to checkout
- Attempt payment again, and it completes successfully
- Customer sees two pending charges on card statements — one pending charge clears after 3–4 business days
All of these details go into a detailed JIRA ticket, are labeled as part of the relevant beta test, and get escalated to the current engineer on duty for the team that owns the feature. As you can see, narrowing down the issue and routing it to the right place can be complex. That’s why your detailed reports are so critical. They cut down the time to resolve dramatically.
Example JIRA ticket:
3. Resolution: Identify the source of the error, plan and implement a fix
The engineer on call is pinged that they have a new ticket to review. The beta team also shares the reported issue in a dedicated channel within Slack, our internal communication tool, so that our product and engineering teams are aware.
We also log the beta ticket itself so that we can track how many bugs were reported and resolved as part of the test. Finding bugs early and solving them quickly is one of the ways we measure how successful our test was.
The beta team shares details in Slack and in Jira. It follows up with the reporting testers to gather details the engineers may need to confirm the bug, to review the code, and to find the best fix. Some fixes can be pushed out immediately, and others require code changes that need to be rolled into an upcoming app update.
The beta team will stay in touch with affected testers to share a timeline for the fix and any recommended workarounds that can help avoid disruption while a fix is on the way. Once the bug is fixed, the engineer will update the ticket status to indicate either that it’s now resolved, that it’s resolved in the next app version, or that it’s lined up for another future release.
Example resolution post:
In the end, the quality of our bug tickets and the speed of our fixes rely heavily on the details that our testers and their employees share with our beta team. It’s never a good time when you run into a bug that disrupts your operations, but smoking these out and squashing them while we’re still in testing makes a really huge difference to the thousands of businesses that will use these beta features later in the test and after it’s been rolled out fully.
A big thank you to all of our beta sellers who take the time to document and let us know about issues as they come up. We couldn’t do it without you. 👍
Have questions? Let us know in the comments and we’ll be happy to squash them.
Andy Rea is a longtime Square (9 years!), backpacker, girl-dad, and bug squisher. Andy worked in support, advanced support, and the bugs triage team before joining the beta team as a specialist.
