Resolved -
As of 12:30am PT, 7:30am GMT today, all historic webhooks have been cleared from the queues.
Oct 21, 04:13 PDT
Update -
At 15:01 PT, AWS confirmed that service restoration from their side after network connectivity issues they had across US-EAST-1 Regions was resolved. However, some AWS services are still degraded due to AWS's backlog of messages that will take a few hours to be processed, and an RCA will be shared from AWS side.
All of Marqeta’s impacted services have been restored. For Webhooks service, processing was disrupted starting at approximately 12:00 AM PT. We have since restored service as of 12:30 PM PT, and we began flushing the queued webhooks starting at 6:15 PM PT, with processing expected to complete within 12 hours.
We are actively monitoring systems closely to ensure continued stability.
Oct 20, 18:22 PDT
Update -
Numerous AWS services are degraded. This directly impacts several Marqeta services, including:
Impacted services :
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded 3DS service;
Degraded Sandbox service;
Delay in webhooks;
Recovered services :
Network Updates service
Tokenization services
Core API response
AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.
Oct 20, 14:34 PDT
Update -
Numerous AWS services are degraded. This directly impacts several Marqeta services, including:
Impacted services :
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded 3DS service;
Degraded performance in tokenization functions.
Degraded Sandbox service;
Recovered /improved services :
Network Updates service (Recovered).
AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.
Oct 20, 13:35 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded Network Updates service;
Degraded performance in tokenization functions.
AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.
Oct 20, 12:34 PDT
Update -
We are continuing to monitor for any further issues.
Oct 20, 11:41 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded Network Updates service;
Degraded performance in tokenization functions.
AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.
Oct 20, 11:35 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded Network Updates service;
Degraded performance in tokenization functions.
AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.
Oct 20, 10:35 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded Network Updates service;
Degraded performance in tokenization functions.
AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.
Oct 20, 09:19 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Delay in fulfillment files;
Partial outage to KYC Services;
Degraded Network Updates service;
Degraded performance in tokenization functions.
AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.
Oct 20, 08:22 PDT
Update -
Numerous AWS services are still degraded. This directly impacts several Marqeta services, including:
Delayed or unsent webhooks;
Delay in all bank files and all DIVA MQD reports;
Delay in recon and settlement reports;
Delay in API responses;
Delay in VAU and ABU reports;
Delay in some direct deposits;
Partial outage to KYC Services;
Degraded performance in tokenization functions.
AWS reports they are still experiencing elevated errors for new EC2 instance launches. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.
Oct 20, 07:10 PDT
Update -
Numerous AWS services are still affected, which directly impacts all webhooks. Marqeta engineers have also confirmed this will delay all bank files and all DIVA MQD reports. Additionally, recon and settlement reports are delayed.
AWS reports they are still experiencing elevated errors for new EC2 instance launches. Engineers are continuing to monitor and will be able to slowly resume webhooks services and reporting once AWS has recovered.
Oct 20, 06:51 PDT
Update -
AWS Fargate is still affected, which directly impacts all webhooks. AWS reports they are continuing to work through processing the backlog of SQS messages in Lambda queues. Engineers are continuing to monitor and will slowly resume webhooks services once AWS has recovered.
Oct 20, 06:29 PDT
Update -
AWS Fargate is still affected, which directly impacts all webhooks. AWS reports working through processing the backlog of SQS messages in Lambda queues. Engineers are continuing to monitor and will slowly resume webhooks services once AWS has recovered.
Oct 20, 05:21 PDT
Update -
AWS reports they are seeing significant signs of recovery. AWS Fargate is still affected, which directly impacts all webhooks. Engineers are continuing to remediate and will slowly resume webhooks services once AWS has recovered.
Oct 20, 04:10 PDT
Update -
AWS reports they are seeing significant signs of recovery. We are monitoring this recovery and the recovery of Marqeta services.
Oct 20, 03:53 PDT
Monitoring -
AWS reports they are seeing significant signs of recovery. We are monitoring this recovery and the recovery of Marqeta services.
Oct 20, 02:52 PDT
Identified -
Marqeta engineers are tracking a confirmed AWS us-east-1 outage that is affecting multiple systems and other services. This can include SMS, Tokenization, KYC, APIs, Marqeta Dashboard, and more. Investigation is continuing to understand the scope of impacted systems. We are monitoring AWS communication channels for updates.
Oct 20, 02:06 PDT
Update -
Marqeta engineers are tracking a confirmed aws-us-east-1 outage that is affecting multiple systems and other services. This can include SMS, Tokenization, KYC, APIs, Marqeta Dashboard, and more. Investigation is underway and more information will be available soon.
Oct 20, 01:12 PDT
Investigating -
Marqeta engineers are tracking a possible aws-us-east-1 outage that is affecting multiple systems and other services. This can include SMS, tokenization, and more. Investigation is underway and more information will be available soon.
Oct 20, 01:02 PDT