Application Outages
Resolved
Jun 10 at 04:59pm MDT
All services have been restored and applications are now accessible.
Our infrastructure provider has managed to bring all services back online and are continuing validation efforts. While primary functionality has returned, they are still conducting internal testing and making incremental improvements.
We continue to actively monitor our systems. Brief service interruptions may still occur as stabilization efforts continue. We will maintain heightened monitoring throughout this period.
Thank you for your patience during this extended disruption.
Affected services
Updated
Jun 10 at 02:04pm MDT
Update
Our infrastructure provider is testing fixes in pre-production environments with positive initial results. They are continuing validation before deploying to production systems.
A resolution timeline will be provided once testing is complete. We continue to monitor the situation and will update you as information becomes available.
Thank you for your patience.
Affected services
Updated
Jun 10 at 12:28pm MDT
Our infrastructure provider has identified the root cause as an automated update that disrupted network connectivity. They are implementing fixes and developing a permanent solution.
No immediate timeline for restoration was provided, but we will update once we hear an estimate recovery timeframe. We continue to monitor the situation and will provide updates as they become available.
Affected services
Updated
Jun 10 at 10:42am MDT
Our services remain unavailable due to an ongoing issue with our infrastructure provider.
Our provider has informed us that initial restart attempts have been unsuccessful, with errors continuing to occur. They have identified a potential configuration-related cause and are actively working on remediation efforts. Their team is pursuing multiple approaches, including reverting recent changes.
We continue to closely monitor the situation and our upstream provider as they work toward resolution. Unfortunately, no estimated time of restoration is available at this moment.
We understand the impact this extended outage is having and sincerely apologize for the inconvenience. We will provide updates as soon as more information becomes available.
Thank you for your patience and understanding.
Affected services
Updated
Jun 10 at 09:28am MDT
Update:
We have identified that the service disruption affecting our Portal infrastructure is part of a widespread outage impacting multiple major infrastructure providers globally. This is affecting numerous services across the internet.
Current Situation:
- Portal services remain unavailable
- The issue has been traced to upstream infrastructure providers experiencing a major incident
- This is impacting multiple cloud and hosting providers simultaneously
- Resolution timeline depends on third-party provider recovery
Actions Taken:
- Confirmed the root cause is external to our systems
- Our engineering team continues to monitor upstream services and connectivity
- All internal systems are ready to restore service once upstream connectivity is restored
Next Steps:
We are actively monitoring our providers' status pages and will begin service restoration immediately once the upstream issues are resolved. We will continue to provide updates every 1 hour or when significant changes occur
We appreciate your patience during this unprecedented disruption and remain committed to restoring service as quickly as possible.
Affected services
Created
Jun 10 at 02:33am MDT
We are currently experiencing an outage that as impacting the Portal, Kaiya and Proxima. This is causing a heightened number of login issues. We are actively investigating the issue and will report back with updates.
Update: Incident Identified
We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Prosci), which is causing issues across all of our applications. We will continue to work on the situation and update here.
Affected services