While we have been unable to definitively isolate the underlying issue we did take steps to address some system deficiencies. We currently believe that normal system behavior taken under certain circumstances can cause issues that take hours to resolve. Yesterday we addressed two minor system issues and then overnight we applied a method for modulating load on a specific troubled system. These efforts to have addressed the issue. We will continue to monitor the systems an be responsive to a recurrence.
Posted Jun 18, 2019 - 11:23 PDT
Update
We are actively working to implement code and structure changes to ease the load on the troubled system. In the interim we continue to monitor the system to ensure acceptable system performance.
Posted Jun 17, 2019 - 15:25 PDT
Update
We have identified the issue as one of our database nodes is being affected by an increased workload. We continue to investigate the root cause. In the interim we have found a stable balance of the overall workload and the site is behaving within normal bounds. All inventory, order, and price data is syncing properly.
Posted Jun 17, 2019 - 11:44 PDT
Investigating
Starting approximately on Thursday morning we started to experience some database load issues of unknown origin. Our operational team took steps in an attempt to alleviate the load by moving resources around. Currently it seems those efforts were unsuccessful as we continue to see the app be unresponsive to many merchants.
We have currently paused all background processes and will resume them over the next few hours as we work to identify and resolve the root cause of this performance issue. We have also reached out to our database provider for support and are actively collaborating with them on a fix.
Fixing this issue is our number one concern and we are actively working toward a fix. Once we identify a satisfactory solution everything should be back to normal within a few hours. We will do our best to resolve this issue today.