tag:status.bonsai.io,2005:/historyBonsai Status - Incident History2024-03-19T02:30:32ZBonsaitag:status.bonsai.io,2005:Incident/162700172023-02-24T18:35:09Z2023-02-24T18:35:09ZHeroku Bonsai Elasticsearch add-on users experiencing provisioning failures<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>18:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. Heroku users will need to resend their provisioning requests. Heroku will also remove failed provisioning requests after 24 hours, so users will have to resend their provisioning requests.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>18:10</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:25</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bonsai.io,2005:Incident/161384882023-02-15T01:37:18Z2023-02-15T01:37:18ZPerforming maintenance to multitenant platform<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>01:37</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing maintenance and expect no customer impact during this time.</p>tag:status.bonsai.io,2005:Incident/112090182022-09-21T17:05:40Z2022-09-21T17:05:40ZScheduled maintenance in GCP US-East-1 Region<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>17:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:35</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Scheduled</strong> - The Bonsai Platform team is in the process of rolling out an upgrade to GCP resources. This will not affect Elasticsearch, but may temporarily impact performance for some GCP resources in the GCP US-East-1 Region.</p>tag:status.bonsai.io,2005:Incident/103261222022-06-17T08:25:46Z2022-06-17T15:41:05ZInterruption to Metrics service<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>08:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>00:30</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>20:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>20:15</var> UTC</small><br><strong>Investigating</strong> - We are investigating interruptions to the Metrics service. This does not affect Elasticsearch, but may temporarily impact metrics collection for users.</p>tag:status.bonsai.io,2005:Incident/103238922022-06-17T08:25:19Z2022-06-17T15:37:42ZInterruption in Grafana reporting<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>08:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>23:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and our team is working to restore the Grafana dashboards.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>16:55</var> UTC</small><br><strong>Investigating</strong> - The metrics database used by Grafana is currently experiencing an interruption. Elasticsearch clusters are operating normally.<br /><br />View an alternate source of metrics data on the "Metrics" tab in your Bonsai dashboard: https://docs.bonsai.io/article/328-metrics.</p>tag:status.bonsai.io,2005:Incident/98652562022-04-27T16:16:23Z2022-04-27T16:16:23ZScheduled maintenance in AP and EU regions<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>16:16</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>15:17</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>15:15</var> UTC</small><br><strong>Scheduled</strong> - The Bonsai Platform team is in the process of rolling out an upgrade to our metrics service. This will not affect Elasticsearch, but may temporarily impact metrics collection for some users in the US-East-1 region.</p>tag:status.bonsai.io,2005:Incident/98574622022-04-26T16:30:23Z2022-04-26T16:30:23ZPlanned maintenance of metrics service<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>13:09</var> UTC</small><br><strong>Scheduled</strong> - The Bonsai Platform team is in the process of rolling out an upgrade to our metrics service. This will not affect Elasticsearch, but may temporarily impact metrics collection for some users in the US-East-1 region.</p>tag:status.bonsai.io,2005:Incident/98518682022-04-25T20:20:37Z2022-04-25T20:20:37ZScheduled Upgrade in Progress<p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>18:21</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>18:17</var> UTC</small><br><strong>Scheduled</strong> - The Bonsai Platform team is in the process of rolling out an upgrade to our metrics service. This will not affect Elasticsearch, but may temporarily impact metrics collection for some users.</p>tag:status.bonsai.io,2005:Incident/96524822022-03-24T04:30:00Z2022-03-28T21:51:35ZMissing Cluster Metrics<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>04:30</var> UTC</small><br><strong>Resolved</strong> - At 05:30 UTC on 24 March 2022, Bonsai’s metrics database experienced node loss, which impacted metrics collection for a small percentage of clusters. Normally node loss is tolerable, and nodes are periodically replaced without incident. In this case, the node loss event affected replicated data as well. Affected users may see a gap in their metrics for the duration of the event.<br /><br />This event only affected metrics collection; customers’ Elasticsearch clusters were not impacted and continued to function normally.</p>tag:status.bonsai.io,2005:Incident/95118522022-03-10T20:56:10Z2022-03-10T20:56:10ZElevated 404 errors<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>20:56</var> UTC</small><br><strong>Resolved</strong> - The incident is resolved. These 404 errors were attributed to a stale SSL certificate, which has been updated.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>20:30</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bonsai.io,2005:Incident/94006492022-02-24T17:47:15Z2022-02-24T17:50:24ZBonsai.io intermittent unresponsiveness<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring Heroku: https://status.heroku.com/</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Identified</strong> - The issue has been identified. Bonsai.io is dependent on Heroku, and there is a incident for Heroku: https://status.heroku.com/</p>tag:status.bonsai.io,2005:Incident/88556502021-12-15T16:31:23Z2021-12-15T16:31:24ZConnectivity problems in US-West-2<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:31</var> UTC</small><br><strong>Resolved</strong> - The vendor has resolved the issue causing intermittent timeouts.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>Update</strong> - The problem appears to be due to intermittent third party infrastructure issues. We are evaluating mitigation options.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>15:58</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bonsai.io,2005:Incident/88123472021-12-11T05:37:37Z2021-12-11T05:37:38ZLog4j Zero-Day RCE<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>05:37</var> UTC</small><br><strong>Resolved</strong> - We have received independent confirmation via Elastic, Inc. that Elasticsearch is not vulnerable to RCE due to its use of the Java Security Manager. Our team will finish rolling out mitigations, but otherwise are standing down on updates here, pending any new developments.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>00:59</var> UTC</small><br><strong>Update</strong> - All relevant versions for new cluster deployments have been updated, and we have re-enabled Sandbox cluster creation. We appreciate the patience from everyone who was stuck at the last step of new account creation this afternoon!</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>00:34</var> UTC</small><br><strong>Update</strong> - We are continuing to make steady progress in rolling out updates, with all of ES 5.x clusters updated, approximately 80% of ES 6.x, and over 50% of ES 7.x clusters updated.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Update</strong> - Our team is continuing to roll out updates and making steady progress.<br /><br />We’ve determined that a configuration based mitigation is not available in some early versions of Elasticsearch 5.x. Some customer clusters running on early versions of Elasticsearch 5.x have been upgraded to Elasticsearch 5.6.16.<br /><br />Updates to Elasticsearch 6.x, 7.x, and OpenSearch 1.x are still under way.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>18:17</var> UTC</small><br><strong>Update</strong> - At this time we're reasonably confident that Bonsai is not susceptible to the Remote Code Execution in this vulnerability.<br /><br />However, we believe certain combinations of Java, Elasticsearch, and log4j can plausibly execute a remote ping. Out of an abundance of caution, we’re moving forward with a rollout of configuration mitigations.<br /><br />For those following along and interested in the details of this incident, there are different combinations of the JDK version alongside the version of Log4j that are relevant to reproducibility. Per the security update from Apache (https://logging.apache.org/log4j/2.x/security.html)<br /><br />>>><br />Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against RCE by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".<br /><<<<br /><br />Java 8u121 was released in January 2017, and Bonsai is running with newer versions of Java than that across the board. We believe this default has made our systems safe by default from this particular vulnerability.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>17:40</var> UTC</small><br><strong>Update</strong> - We have temporarily disabled creation of Sandbox clusters pending updates to the underlying services.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>17:18</var> UTC</small><br><strong>Identified</strong> - Our engineers have identified the services within our platform which may be affected, however have not been able to reproduce the vulnerability. Out of an abundance of caution we are proceeding to roll out additional safeguards in the underlying service configurations.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>17:18</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>17:11</var> UTC</small><br><strong>Investigating</strong> - The team is currently investigating the issue.</p>tag:status.bonsai.io,2005:Incident/80661912021-09-23T19:57:24Z2021-09-23T20:06:41ZBonsai Website is down<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>19:47</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>19:11</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bonsai.io,2005:Incident/79578012021-09-09T21:05:06Z2021-09-09T21:07:17ZBonsai Website is down<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>21:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>19:45</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bonsai.io,2005:Incident/73738472021-06-30T21:16:46Z2021-06-30T21:16:46ZScheduled Maintenance<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>21:16</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>21:13</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>20:44</var> UTC</small><br><strong>Scheduled</strong> - We're updating the Bonsai app. This will not impact any services.</p>tag:status.bonsai.io,2005:Incident/71896872021-06-07T19:42:59Z2021-06-07T19:42:59ZDelayed metrics<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>19:42</var> UTC</small><br><strong>Resolved</strong> - Metrics are now emitting to the Bonsai Dashboard "Metrics" tab. We will continue to monitor the situation.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>18:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. We will update periodically.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>17:48</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that is causing delayed metrics emitted to the Bonsai Dashboard "Metrics" tab.</p>tag:status.bonsai.io,2005:Incident/66438312021-03-31T17:33:32Z2021-03-31T17:33:32ZPlanned Maintenance<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>17:33</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance is complete.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>20:22</var> UTC</small><br><strong>Scheduled</strong> - We are rolling out a minor upgrade to our platform. This operation is expected to be zero downtime with no customer impact.</p>tag:status.bonsai.io,2005:Incident/66423662021-03-30T17:15:30Z2021-03-30T17:15:30ZPlanned maintenance<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>17:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>16:15</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Scheduled</strong> - We are rolling out a minor upgrade to our platform. This operation is expected to be zero downtime with no customer impact.</p>tag:status.bonsai.io,2005:Incident/66084842021-03-25T16:54:47Z2021-03-25T17:51:36ZBonsai Front Page Unresponsive<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>14:39</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue. Users can directly access the Bonsai app through: https://app.bonsai.io/.</p>tag:status.bonsai.io,2005:Incident/65872422021-03-23T19:28:09Z2021-03-23T19:28:09ZPlanned Service Upgrade<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>19:28</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>15:07</var> UTC</small><br><strong>Scheduled</strong> - We are rolling out a minor upgrade to our platform. This operation is expected to be zero downtime with no customer impact. However we will be monitoring the deploy closely and performing the rollout slowly throughout the day.</p>tag:status.bonsai.io,2005:Incident/64457702021-03-04T23:00:30Z2021-03-04T23:00:30ZPlanned Service Upgrade<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>22:04</var> UTC</small><br><strong>Update</strong> - We are continuing to verify the maintenance items.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>22:04</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Scheduled</strong> - We are rolling out a minor upgrade to our platform. This operation is expected to be zero downtime with no customer impact. However we will be monitoring the deploy closely and performing the rollout slowly throughout the day.</p>tag:status.bonsai.io,2005:Incident/60886722021-01-22T21:40:43Z2021-01-22T22:50:10ZElevated timeouts in US-East region<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>21:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring the results of a fix.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Investigating</strong> - We have detected an increase in timeout errors affecting multitenant subscriptions in the US-East region.</p>tag:status.bonsai.io,2005:Incident/60799462021-01-21T22:15:53Z2021-01-21T22:19:32ZElevated timeouts in US-East<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>22:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Identified</strong> - We’ve identified the cause for the elevated timeouts. We’re currently taking the necessary steps for recovery.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>18:45</var> UTC</small><br><strong>Investigating</strong> - We have detected an increase in timeout errors affecting multitenant subscriptions in the US-East region.</p>tag:status.bonsai.io,2005:Incident/56454142020-11-24T20:38:07Z2020-11-24T20:38:07ZPlanned Network Upgrade<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>20:38</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>Scheduled</strong> - We will be deploying an update to improve a reliability of our network infrastructure. Minimal to no customer impact is expected.</p>