10:18 UTC: The Beeminder main site seems to be down at the moment, but we’re on it!
1 Like
10:33 UTC: looks like our hosting provider is experiencing an issue, affecting all of their services in US-EAST (which is where we are hosted). They are also investigating!
1 Like
Still unavailable…
If you want to follow along, we’ll keep this page updated!
The usual rules apply, of course: if this causes you to derail, that’s on us, and you should just reply to the legit-check email and we’ll cancel any charges.
You can also keep an eye on status.beeminder.com , which is independently hosted, and shows what we know so far (which is that our hosting provider, Linode, has a problem affecting a bunch of their services out of one location).
3 Likes
No real news, we’re afraid! For the past two hours, all our hosting provider has said is that they’re investigating.
There are some signs of life from our servers, and we did manage to get our “Bee Right Back” page up, but our database server is still down.
We do have a database backup, though it’s a little stale, so almost all data is safe in the event that it doesn’t come back up.
As ever, if you derail because of this, just reply to the legit check and let us know, once we’re back up!
2 Likes
Oh, and from the support team: it’s technically my day off, but I’ve been checking in with the weekend workerbees in case there’s anything I can do, and someone’s had their eye on it all the time so far. We’ll keep up with any inbox queries as best as we can, though we typically check the inbox in timed slots rather than all day, so there may be some gaps.
1 Like
Yeah, an “emerging service issue affecting All Services in US-EAST” is not a lot to go on. Newark, New Jersey has not been hit by a meteor, per Google news. No response to our support tickets yet.
We do know it’s not just us at least. (“Mal de muchos, consuela de tontos”)
Still no updates, I’m afraid! Or rather, this still-uninformative update from Linode is all we’ve got:
Our team is investigating an issue affecting connectivity in our US-EAST (Newark) data center. During this time, users may experience unavailability issues for all services deployed in this data center. We will share additional updates as we have more information.
Posted 42 minutes ago. Jul 27, 2025 - 12:34 UTC
The hope is that as soon as we get back up, we’ll run a delay on all pending charges. That way, we hope fewer will go through before we get a chance to cancel them. The timing of that can never be perfect (there’ll be people who’ll derail because of the downtime whose charge won’t have been queued up yet to delay), so do reply to any derailment that was caused by the downtime ASAP and don’t count on it being delayed… but nonetheless, it’ll help.
1 Like
Maybe-phew, from Linode:
The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online.
We’re seeing some signs of life, but not our database, and the recovery’s slow so far…
Simone (@hiddenspark) has been around and checking on the inbox.
1 Like
Linode have been answering our support tickets, and we have access to some of our servers, but they’re still working on it and they haven’t given us an ETA.
As a reminder, if you derail because of the downtime, reply to the legit check as soon as you can and we’ll take care of it. We’re also hoping to delay pending charges at least once when we’re back up, in order to give people more time.
And if anyone wants to socialise (or wants some accountability!) in the meantime, I made a thread for that.
We’re really sorry about the downtime – unfortunately it’s a case where we aren’t able to do a whole lot about it until the servers come back up, but nonetheless, it’s super disruptive and we’ll do our best to get back online as quickly as we can once it’s in our hands.
2 Likes
@shanaqui Should I respond to charge emails now, or wait for Beeminder to be back up so I can enter missed data on any of the goals where that applies?
Good question, @narthur. I guess waiting will make things easier on us when everything comes back.
1 Like
Ah, let me contradict Danny from the support point of view! I’d rather people reply to the emails as soon as they can. Even if it means a bit more back-and-forth admin, there’s less risk of people forgetting altogether that way.
I’m not too worried about what’s easiest for the support team, we’re good at recovering from this kind of thing. I have faith in us! What’s more important is that we get charges sorted out as quickly as possible, so nobody gets charged because of this situation.
So yes, if you have derailments to reply to already, then go ahead!
4 Likes
We’re about to be back up for real! Bee’s carefully restarting everything. ETA: minutes!
1 Like
And we’re back! 
Really sorry for the extended downtime there! 
7 Likes
A couple of updates just to wrap everything up!
-
It actually turns out that our servers could contact each other before we could contact our database server, so some things were turning over even though the site itself was inaccessible. We could probably have got things working again sooner if we’d realised, but it really wasn’t clear at the time. In terms of learning from the situation, this one’s going on our checklist!
-
Charges have been delayed as planned, and emails gone out to those affected so that the delays won’t come as a surprise (hopefully).
-
The support team have been working through the backlog of emails; Simone (@hiddenspark) zeroed it out during the UK night-time (thank goodness for night owls!), and I’m working toward zero again now. If I don’t get it to zero during the time I have available today, someone else will.
If you have already contacted us but not yet received a reply, please do not email us again!
We’ll get to you as soon as we can, I promise, and emailing again will delay our response to you due to the chronological sorting in our inbox.
Hopefully that’s the end of it, but as ever, if you’re seeing any weirdness or anything seems buggy, send a message through to support and we’ll pick it up and dig into it as soon as we can!
4 Likes