Posts in 2023
  • A Minute from the Moderators

    Monday, May 08, 2023 in Posts

    Hello Hachydermia! It’s time for, you guessed it, the monthly Moderator Minute! Recently, our founder and former admin stepped down. As sad as we are to see her go, this does provide an excellent segue into the topics that we wanted to cover in …

    Read more

  • Moderation Postmortem

    Tuesday, May 02, 2023 in Incidents

    Hello Hachydermians! There has been a lot of confusion this week, so we’re writing up this blog post to be both a postmortem of sorts and a single source of truth. This is partly to combat some of the problems generated by hearsay: hearsay generates …

    Read more

  • Stepping Down From Hachyderm

    Sunday, April 30, 2023 in Posts

    Stepping Down From Hachyderm Recently I abruptly removed myself from the “Admin” position of Hachyderm. This has surfaced a number of threads about me, Hachyderm, and the broader Fediverse. Today I would like to offer an apology as well …

    Read more

  • A Minute from the Moderators

    Wednesday, April 05, 2023 in Posts

    Hello and welcome to April! This month we’ll be reviewing the account verification process we rolled out as well as two more classic moderation topics: how to file a report and what to do if you’re moderated. Account Verification Mastodon …

    Read more

  • A Minute from the Moderators

    Thursday, March 16, 2023 in Posts

    We’ve been working hard to build out more of the Community Documentation to help everyone to create a wonderful experience on Hachyderm. For the past month, we’ve focused most heavily on our new How to Hachyderm section. The docs in this …

    Read more

  • Decaf Ko-Fi: Launching GitHub Sponsors et al

    Tuesday, February 28, 2023 in Posts

    Since our massive growth at the end of last year, many of you have asked about ways to donate beyond Nóva’s Ko-Fi. There were a few limitations there, notably the need to create an account in order to donate. There were a few milestones we …

    Read more

  • TLS Expires: media.hachyderm.io

    Monday, February 27, 2023 in Incidents

    On February 28th, 2023 at approximately 01:55 UTC Hachyderm experienced a service degradation in which images failed to load in production. We were able to quickly identify the root cause as expired TLS certificates in production for …

    Read more

  • Fritz Timeouts

    Saturday, January 07, 2023 in Incidents

    On January 7th, 2023 at approximately 22:26 UTC Hachyderm experienced a spike in HTTP response times as well as a spike in 504 Timeouts across the CDN. Working backwards from the CDN to fritz we discovered another cascading failure. Context There is …

    Read more

  • Fritz on the fritz

    Tuesday, January 03, 2023 in Incidents

    On January 3th, 2023 at approximately 12:30 UTC Hachyderm experienced a spike in response times. This appeared to be due to a certificate that had not been renewed on fritz, which runs the Mastodon Puma and Streaming services. The service appeared to …

    Read more

Posts in 2022
  • The Queues ☃️ down in Queueville

    Tuesday, December 20, 2022 in Incidents

    Every Queue down in Queueville liked ActivityPub a lot. But John Mastodon who lived just north of Queuville, did not! John Mastodon hated ActivityPub, the whole Activity season! Now please don’t ask why. No one quite knows the reason. It could …

    Read more