Posts in 2023
Decaf Ko-Fi: Launching GitHub Sponsors et al
Tuesday, February 28, 2023 in Posts
Since our massive growth at the end of last year, many of you have asked about ways to donate beyond Nóva’s Ko-Fi. There were a few limitations there, notably the need to create an account in order to donate. There were a few milestones we …
TLS Expires: media.hachyderm.io
Monday, February 27, 2023 in Incidents
On February 28th, 2023 at approximately 01:55 UTC Hachyderm experienced a service degradation in which images failed to load in production. We were able to quickly identify the root cause as expired TLS certificates in production for …
Fritz Timeouts
Saturday, January 07, 2023 in Incidents
On January 7th, 2023 at approximately 22:26 UTC Hachyderm experienced a spike in HTTP response times as well as a spike in 504 Timeouts across the CDN. Working backwards from the CDN to fritz we discovered another cascading failure. Context There is …
Fritz on the fritz
Tuesday, January 03, 2023 in Incidents
On January 3th, 2023 at approximately 12:30 UTC Hachyderm experienced a spike in response times. This appeared to be due to a certificate that had not been renewed on fritz, which runs the Mastodon Puma and Streaming services. The service appeared to …
Posts in 2022
The Queues ☃️ down in Queueville
Tuesday, December 20, 2022 in Incidents
Every Queue down in Queueville liked ActivityPub a lot. But John Mastodon who lived just north of Queuville, did not! John Mastodon hated ActivityPub, the whole Activity season! Now please don’t ask why. No one quite knows the reason. It could …
Degraded Service: Media Caching and Queue Latency
Sunday, December 18, 2022 in Incidents
On Saturday, December 17th, 2022 at roughly 12:43 UTC Hachyderm received our first report of media failures which started a 2-day-long investigation of our systems by @hazelweakly, @quintessence, @dma, and @nova. The investigation coincidentally …
Global Outage: 504 Timeouts
Tuesday, December 13, 2022 in Incidents
On Tuesday, December 13th, 2022 at roughly 18:52 UTC Hachyderm experienced a 7 minute cascading failure that has impacted our users around the globe resulting in unresponsive HTTP(s) requests and 5XX level requests. The service has not experienced …
Updating Domain Blocks
Monday, December 05, 2022 in Announcements
Today we are unblocking x0f.org from our list of suspended instances to federate with. Hachyderm will begin federating with x0f.org immediately. Reason for suspending We believe the original suspension was related to early moderation actions taken …
Opening Hachyderm Registrations
Sunday, December 04, 2022 in Announcements
Yesterday I made the decision to temporarily close user registrations for the main site: hachyderm.io. Today I am making the decision to re-open user registrations again for Hachyderm. Reason for Closing The primary reason for closing user …
Growth and Sustainability
Sunday, December 04, 2022 in Posts
Thank you to everyone who has been patient with Hachyderm as we have had to make some adjustments to how we do things. Finding ourselves launched into scale has impacted our people more than it has impacted our systems. I wanted to provide some …