Performance Issue: Square Services Disruption 7 Sept, 2023

[urs]

[The title of this post was updated by a moderator from "IS SQUARE DOWN, EVERYWHERE?"]

The Square Support Line says..."you have reached us during an unexpected closure..."  Anyone have an idea what is going on?

[anchohoney]

Remember how Square went completely belly-up for 12 hours yesterday? In our case, the “disruption” caused thousands of dollars in lost sales, as well as completely disrupted our payroll run, leaving all of our employees in the lurch. Contrary to reports from Square, it was not possible to run transactions in “offline” mode (which, from the problems I am reading from others, turns out to be a blessing).

The question is, how will Square make this right; and preserve the loyalty of their user base? Waived processing fees? Some other recompense?

[PartyManiaMD]

While it really sucks that the system went down yesterday this is a learning lesson for everyone involved. Always have a backup method of taking transactions such as a crash kit for CC processing. We physically wrote down card info/customer info and saved their transactions. When the system came up I manually entered each transaction and then shredded their info. Am I upset about paying extra to process it manually after the fact? Of course not, but I've learned the hard way to have a backup system or other payment source just in case. Much like I have my phone as a mobile hot spot when comcast goes out for 12+ hours, at most comcast will give a fractional amount based on my overall bill.

I think it's too early to blame Square completely given I've heard rumors it was an orchestrated attack against the larger system.

[anchohoney]

Hard disagree. Square wants us to bake their offerings into every aspect of our business, from taking payments to marketing to payroll. That’s all well and good, and when it works, it’s wonderful. But a backup payment processor wouldn’t have gotten my staff paid on time. The fact is, when companies are this dependent on a suite of services, outages like this simply can’t happen, and they certainly can’t last this long with basically no explanation. 

[Donnie-M]

@anchohoney I have to also disagree with the expectation that nothing like this can ever happen.  If you look back, there have been a bunch of companies that had things just like this happen that caused the services to go down and impacted people.  And in reality, Square could have done more to communicate, but they went far beyond some of the other companies and even explained the issue in follow up communication.

Amazon Web Services, December 2021

Facebook went down for 6 hours in 2021 completely how much did that affect business that was advertising

Fastly a Web Services company went down in June 2021 affecting even Target

British Airways in 2017..engineer unplugged a data center and power surge afterwards...672 flights cancelled

Google has had an outage 2020 even though it was only 45 mins it was everything.

Dyn in 2016 hada ddos attack that took down netflix, twitter, and some others..dyn is a internet backbone

Microsoft December 2021

Comcast November 2021

Akami Edge 2021 was another DNS issue that affected a bunch of third party business's like Steam

Heck, the whole country of Armenia (3 million) lost all internet in 2011 due to a lady digging a hole.

I am not saying things shouldn't be done to prevent, but it is unrealistic to say you are any better off going somewhere else or that it should never happen.  That isn't fair or realistic.

Payroll is a big deal, but there can be any number of reasons why they could have not got paid...if your office manager drops dead in a car accident...payroll might be late..it stinks..but it could happen.  We all do our best and look at what went wrong and try to do better.

[BenchmarkPub]

I am not optimistic we will see any form of compensation, after a global outage like this, the numbers would be staggering in trying to compensate for, plus management would have to answer to the board and shareholders on why their earnings took such a hit, who's to blame, etc.  Management gets less bonus and options...so operators, owners, and employees, are expected to pay the price.  Part of the business many will say.  We either have to feel good about their remedy to prevent future events like this or look at the marketplace.  I am a long time Square customer, but for me, this has motivated me to look at the marketplace and see if there is a better fit. I'd still like to feel valued with a personal touch from them, maybe it will come, maybe not. The grass may not be greener elsewhere with competitors, but I would not be a good steward for my business if I didn't evaluate and try to reduce future risks.  As customers we always retain our rights to vote on their lack of a response, with our dollars, and they might feel how we felt on Sept. 7th in lost revenue, or tips, or wages.  Just food for thought my fellow forum members.

[DaveH5050]

We run a nail salon and after the outage started, my wife asked for cash payment only. A few elderly customers didn't want to go use the ATM at the nearby Walmart so they left with promising to return and pay when Square was able to process again. My wife tried and failed to turn on the offline mode on the Square Terminal. 

We were down $456 Thursday, but over by $88 Friday and $185 Saturday. Hopefully the rest of the customers return and make good on their promise to pay. 

I do think Square should make some effort to reimburse customers, even if only by waiving processing fees for Thursday. 

[Cll5]

I also would like to know this because my customers sale was declined because of the outage yet there is NO record of declined transaction and my clients pending charge is still holding up the funds.  support just keeps telling me “the funds should be returned within a few days” nope!! Lies. This is beyond disgusting as it has affected me and my clients money.  

[DaveH5050]

This has happened a few times to us. Outage either with Comcast (our ISP) or once with a faulty Square terminal (I suspect a firmware update went screwy, although Square support denied it was anything to do with them), and the last 2 times because of a Square system outage. My solution is to buy a Paypal POS terminal as a backup. $199 for the terminal only (no receipt printer) but it is standalone - no need to plug a card reader into your phone and launch the "Zettle" Paypal app. My wife is completely non-technical so this is the only solution for her at the salon we own. 

After the Sept. 2023 outage, we lost around $180 net so having a backup system would pay for itself in one outage it seems. 

[Mumpower00]

Has anyone had issues with the offline mode yesterday when square was down?

[TheRealChipA]

Hi @Mumpower00.  I lost one — the very last offline transaction that I entered at the end of my day.  Thankfully, my backup card processor (Stripe) had a log of it.  I’ve heard of other sellers who lost some transactions, as well.

You didn’t say, so I’ll ask.  Have you looked at your POS transaction list to see if your missing offline transactions just haven’t been processed yet?  Or is that list of unprocessed transactions empty now?  

[nvalentin]

Yes, I left a tablet open with over $300 of offline transactions when I left for the evening. This morning they hadn't processed and were just missing.

[TheRealChipA]

That is odd.  They are either processing or gone.  Unfortunately, all I can suggest is that you start a chat or call with support.  It would help a lot if you had a printed receipt with the receipt numbers.  I’m sorry to hear this and I hope it resolves in your favor.

[TheRealChipA]

https://squareup.com/help/us/en/contact?panel=FAB456B04971&skip_unit_select=true

[nvalentin]

I started a chat. The only answer I got was "The engineers expect for this data to catch up in the next 24 hours."

[KrisTM]

Same issue here. I had over a dozen offline card transactions yesterday (9/7) and this morning when Square came back up only four were available to push through. Of those four, none are "itemized" with items listed, just an amount, and tips are not broken out so I have no way to figure out how much of that is from sales and how much is for staff.

[coho24]

I have offline transactions that still haven't uploaded yet.  Is anyone else still having this issue?

[Roostynv]

During the square downage yesterday we had over $1k in offline credit card sales and we are back and connected to the internet but those sales are not showing anywhere. What can we do??

[perkits]

I had several hundred $ in offline and over the last few hours they have now been processed. Data appears to be trickling in slowly. Keep checking your sales and transaction data for yesterday, my data has been changing by the hour. Hopefully by the end of the day everything will be back.

[gld3]

Took offline sales during the outage last night, but they are not on any of my reports now? My understanding is they populate when we get back online.  Have they disapperared?  Will there be any recourse?

[MichiganFarmsta]

I have noticed that I have significantly less sales this morning in offline sales. They are not showing up in the reports.

[MichiganFarmsta]

It appears all the cash transactions uploaded. Now we are waiting on the CC transactions.  I would guess (hope) these will upload shortly as square gets things up and running smoothly!