Requesting post-mortem for 2019-10-25 Point of Sale incident

[aeronaut]

Hi, we were one of the affected customers during the downtime experienced yesterday evening, and we'd like a little more information about what happened then. Here is the incident from our perspective:

- Prior to 6:30PM EST, all of our iPad devices simultaneously logged out, and we were unable to log back in on any of them. The app would appear to crash and return to the login screen immediately after signing in and selecting our location. We tried using device codes, and also tried email/password login, both failed.

- We immediately called the 24/7 support line. To our surprise, the message told us that your phone infrastructure was down, suggested emailing or checking the status page, and then hung up. This phone error did not recover during the entire incident.

- We checked the status page and called repeatedly for 1/2 hour, after which the first incident status appeared.

- I noticed many posts on the forums here, none of which have been replied to even now.

- More than an hour after the incident started, logins started working again and no longer crashed the app back to the login page.

Status incident page: https://www.issquareup.com/incidents/rcqx7mzn0c5l

I know this type of thing can rarely happen and I appreciate the apologies that went out on the status page, but I think it's reasonable to ask for some more information on what went wrong here. We were totally unable to take iPad payments for over an hour, and during this time there was also no way to reach a single person working at Square, so this was certainly alarming for all of your customers who were affected by this and it had a material effect on our sales.

Can someone from Square within the coming days please post some type of report or statement with some more detail about what exactly happened to cause this disruption?

Thanks!

[ggilmore]

I agree.  We were also affected and because we are a cash free business we were effectively shut down for an hour. To do the right thing for our customer, we simply gave away product (we are a brewery, so that meant free beer).  It was INCREDIBLY stressful during that time because it was unclear how long the outage might last when we were going into our busiest night of the week and had special events booked so we couldn't simply close the door.  

We do hundreds of transactions an hour...  I think Square owes us an answer and some kind of fee discount to make up for what happened.

[nika]

Thank you for your posts @aeronaut and @ggilmore. And I'm very sorry for the disruption last week. I'm not sure what exactly caused the issues that were occurring. But what I do know is that our engineers are working through some post-mortem syncs to pin point what happened on our end.

I'm happy to share some information when more becomes available. 

In the mean time, we really appreciate our sellers being patient and understanding that these things can unfortunately happen every once in a blue moon. We know it's SUPER stressful to try and run a business when your POS systems are down. So thank you again for hanging in there with us last week.