Dec 27 2005
yet another power outage
Our colocation facility bounced the power again (apparently in order to deal with the failed UPS that caused our previous incident.) We are on route to the facility to bring servers back online now and that should be done shortly. We have been and will continue to work with Yahoo! to migrate to better facilities as fast as humanly possible.
(chronology moved inside the post)
Update (10:50 PM EST): We are at the facility and are bringing up machines now.
Update (11:14 PM EST): Bringing up slaves now. Should be done shortly.
Update (12:09 PM EST): All but one server up now. Bringing up inbox, search, etc.
Update (12:20 PM EST): The facility will be cutting power back over to the UPS at 1 AM. Supposedly this will not affect service, but neither was the test that brought everything down… cross your fingers.
Update (2:04 AM EST): Still waiting for the cutover back to the UPS. They’re planning to give us an hour warning, and have not done so yet. We’ll bring down the site beforehand so we can boot cleanly in case they drop power again.
Update (6:37 AM EST): They just cut over power to the UPS and bounced power again. We’re bringing stuff back online again, everything should be good in a few minutes.
Update (6:57 AM EST): Everything is back online.
Joshua Schachter · joshua Tags: service-alert bookmark this
106 Comments
Mateja | Dec 27 2005 at 10:56 pm
Thanks for the update. Good luck and don’t forget your Red Bull.
TheTraveller | Dec 27 2005 at 11:00 pm
New year resolution number one for del.icio.us: get on the safe side with the most basic infrastructural needs. Happy New Year!
armadda | Dec 27 2005 at 11:09 pm
why you always has power problem?
thansk you coming back
Slayer6 | Dec 27 2005 at 11:12 pm
i’m sure yahoo can afford a generator
ok..now that we are back up
add this to your delicious
http://theunderpantsgnome.blogspot.com
Hoan Ton-That | Dec 27 2005 at 11:12 pm
Hey, looks like its back up.
Well done guys, that was much quicker!
anon | Dec 27 2005 at 11:13 pm
Why did these power outages start happening after you were bought by Yahoo?
Phil | Dec 27 2005 at 11:31 pm
Co.ngra.ts on getting the system back up so quickly!
whiner | Dec 27 2005 at 11:51 pm
very cool. take your time on the migration. plan accordingly, etc. we’re with y’all … despite all the whiners on the “continued hiccups” post.
Rk | Dec 28 2005 at 12:13 am
Thanks guys. Happy new year to you :)
max | Dec 28 2005 at 6:01 am
Mail thingie is still down :(
Philipp Keller | Dec 28 2005 at 6:07 am
Connection refused..?
Is it just me?
While trying to retrieve the
The following error was encountered:
* Connection Failed
zodiac | Dec 28 2005 at 6:33 am
hmm, my tag bundles are gone…
Hazelip | Dec 28 2005 at 6:36 am
Bear in mind that all our collective usage of this site, all our bookmarks, all our eyeballs and efforts to use, promote, and improve this site through feedback suggestions have just you millions of dollars.
OUR efforts have made this site profitable FOR YOU.
Expecting the site to remain functional so that we can continue to help you make money is fucking far from reasonable. We are basically free labor for you people, and our expectation is that this site remains fully functional so we can continue giving places like Yahoo a reason to think this is anything but a money pit for them.
Restore tag intersections, please.
Go research the history of the CDDB if you think my attitude needs an adjustment…
andrew | Dec 28 2005 at 6:47 am
last straw. I am gone!!!!
Andrea | Dec 28 2005 at 6:47 am
Geez folks, give them a break!
This post was written because they DO care. And if people just bothered to read and understand, they’d see that: 1) they do have UPS; 2) they are not owned by del.icio.us, rather, by their colo provider; 3) they are malfunctioning, so no, a generator would NOT help; 4) del.icio.us is aware their colo provider sucks, and they are moving ASAP. What more do you want?
I work for a BIG colo provider in Europe and I can tell you that shit happens. We have spent millions of euros in the power grid, but still, if basic things (like switches) fail, you can have a failure. Last year IBM had a severe failure in a low-tech component of their power system, which started a fire and destroyed a large part of one of their colo facilities; and I can assure you that it didn’t happen because they were cheap or incompetent.
Bye,
Andrea
Philipp Keller | Dec 28 2005 at 7:18 am
The provider should come to this blog and apologize big time.
I mean what they did were mistakes after mistakes after mistakes. And not just “ship happens” mistakes.
And here the users are angry because they don’t understand.
Mark | Dec 28 2005 at 7:31 am
Joshua - what company are you using for colo?
Mara | Dec 28 2005 at 7:41 am
Sorry guys, it’s *not* working :-) I only see my “home” with the latest link but nothing else. No other links (I get a “no item”)and no tags either.
Thanks for listening.
alf | Dec 28 2005 at 8:44 am
The RSS feeds don’t seem to be working.
dermod | Dec 28 2005 at 9:14 am
All my tags are there - but no posts. A frightening sight, all my bookmarked posts vanished. Makes me realise how much I depend on del.icio.us. Where’s that backup gizmo gone?
JC Mann | Dec 28 2005 at 9:25 am
I’m having problems with additive tag filters (e.g. “java+source”). I can find bookmarks with *one* tag, but no results when I add another tag to the search.
Lizi | Dec 28 2005 at 9:29 am
I can’t see my bookmarks either. Please keep us updated on how much time it will take.
Thanks and have a Happy New Year.
Charme | Dec 28 2005 at 9:33 am
Is it just me being silly, or are bundles not showing up in the right-hand panel?
elloelli | Dec 28 2005 at 9:41 am
log in, even if you think you already are. i couldn’t see any items, yet the right hand panel was fine. clicking on them resulted in “no items.” after logging in again, all was fine.
Itto | Dec 28 2005 at 9:50 am
@elloelli : doesn’t seem to help for me
uruloki | Dec 28 2005 at 9:56 am
I am having the same problem as the last few users. If I am not logged in, I cannot see my own bookmark list. Logging in lets me use my own list, but the RSS feeds are empty and I can’t see any other user’s lists.
Additionally, the Blogroll/Linklist functions that del.icio.us provides (http://del.icio.us/help/linkrolls) does not recognize any of my bookmarks even when constrained to a single tag (which the box will recognize but the feed won’t). The Tag roll function (http://del.icio.us/help/tagrolls) works fine, but I’m faced with the same problem of people-that-aren’t-me being unable to see any of the bookmarks under the tag when they click on it.
I was just about to replace a lot of my hand-built lists with del.icio.us’ functionality, now I am glad that I did not.
elloelli | Dec 28 2005 at 10:02 am
Yeah, oops, “all was fine” was way too premature. all is fine in my own list world as far as eyeballing, but no history for any urls, no one else’s lists…
Andrew | Dec 28 2005 at 10:07 am
Searches don’t seem to be working too well, even within my own bookmark list. Also, when I click on a link that says 4 other people tagged a particular URL, it doesn’t list anything.
Yee | Dec 28 2005 at 10:09 am
Is there another problem with Del.icio.us? I logon, and I couldnt see any of my links. I see my tags and bundles, but they seem to be empty.
Search doesnt work either. I search for “geek”, but nothing comes up. It still work 1 hour ago.
What’s going on?
CyberCoder | Dec 28 2005 at 10:27 am
No bookmarks here either, a couple of tags will show some bookmarks, but, as a whole everything is blank.
Blog Jones | Dec 28 2005 at 10:37 am
Just to let you know, inboxes and search are still down. Hope y’all get the problems resolved soon!
Love the service. Thanks for giving it to us for free without ads.
Scott | Dec 28 2005 at 10:38 am
I’m getting the words ‘No items’ when I try and view other people’s del.icio.us pages.
JUS | Dec 28 2005 at 10:50 am
Great service and I know you’ve had it rough at a busy time of year. Congratulations on getting it all back together and (apparently) keeping your humor.
Are you still working on tag indeces? I notice that URL history seems to be missing for many sites… you probably knew that.
Best of luck with Yahoo and the new year!
charme | Dec 28 2005 at 11:27 am
“log in, even if you think you already are. i couldn’t see any items, yet the right hand panel was fine. clicking on them resulted in “no items.” after logging in again, all was fine.”
Nope, already tried that trick…still not working…not at the minute anyway…Tried unclicking and then clicking ’show bundles’, but no change…Funny thing is, if I keep clicking ‘favourites’, my bundles will sometimes show, but that’s only happened once or twice, and I can’t spend all day clicking it!!!
zodiac | Dec 28 2005 at 11:31 am
the bundles are there on the configuration page, but “show bundles” in the main view has no effect
crankysysadmin | Dec 28 2005 at 12:03 pm
Anybody who has ever worked for a colo or had their machines hosted at one knows that power failures like these are extremely frustrating experiences for the company being hosted which are often met with very little sympathy by the colo folks, who shrug and point to the loopholes in the Service Level Agreement. I, for one, know that these problems have absolutely nothing to do with the quality of del.icio.us as a service and wish them all the best in switching colos as soon as possible.
glauber | Dec 28 2005 at 3:23 pm
Down again? (Dec 28 - 14:20 US Central)
bl1ndsqrrl | Dec 28 2005 at 3:29 pm
still down for me. even the main site.
Chuck | Dec 28 2005 at 3:30 pm
IT’S DOWN AGAIN. Goodbye del.icio.us! BACK TO DIGG AGAIN: http://www.digg.com
technodolt | Dec 28 2005 at 3:32 pm
Down down down. Good luck, guys.
tenderlight | Dec 28 2005 at 3:38 pm
http://www.scuttle.org ….This is why I use Scuttle.
Brad Root | Dec 28 2005 at 3:41 pm
I love all the anger over a free service being unavaliable, so silly. Sucks to be without del.icio.us, but thankfully I have a life and other things to do outside of the internet so I don’t get all bent out of shape and need to rant about silly things such as this.
As I like to think, fairweather users aren’t really users at all.
davide | Dec 28 2005 at 3:45 pm
I don’t think I’ll keep using del.icio.us for any longer.
I discovered this just a couple of weeks ago: it looks and feels better and much more friendly than any other online bookmarking site. But honestly… this is just not working anymore. I cannot have to keep all of my bookmarks locally just hoping that del.icio.us will be online again one day or another. It has been down more time than up and running, lately. I work with IT and I really have simpathy for you guys, and I wish you all of the best, but without me.
Sorry… it was quick, but nice, till it lasted.
james | Dec 28 2005 at 4:08 pm
Telehouse sucks.
jackie | Dec 28 2005 at 4:11 pm
Guys. The whole thing is Yahoo’s fault. I will no longer use Yahoo or any of her services. Thank you del.icio.us for showing me the light.
Aundre Harris | Dec 28 2005 at 4:20 pm
Yeah! Down with Yahoo!!!! Down with so-called “Web2.0″!!
TheTraveller | Dec 28 2005 at 4:45 pm
Stop whining!
Read the post. Joshua said that they are moving “to better facilities as fast as humanly possible”, okay?
whiner | Dec 28 2005 at 4:56 pm
i agree w. brad root. i’m so amused at all the fuss! and the whining! this has been so entertaining see people getting mad over this. some folks are even apologetic for leaving. that cracks me up!! the pains folks go thru. amazing. get a life! yahoo’s fault. hilarious!
Blews Dawg | Dec 28 2005 at 4:57 pm
Jackie: The post clearly indicates that del.icio.us has not yet been migrated to Yahoo! facilities. But if you are intellectually predisposed to blame the big successful business, facts do not deter that in any way.
Sjaak | Dec 28 2005 at 5:05 pm
Since the last power outage I’m still having problems with http://del.icio.us/api/posts/all. Most of the times it returns an empty list, sometimes yesterdays list. This means that direc.tor doesn’t work anymore for me.
Julie | Dec 28 2005 at 5:51 pm
Glad you’re back up. What scared me a bit is during yesterday’s malfunction, when I went to del.icio.us I got not a status message nor simple error message, but a whole bunch of error messages related to Perl/Mason which exposed file paths and other information about the server. I hope if there are more outages that such server errors can be hidden for security and aesthetics.
Hazelip | Dec 28 2005 at 7:01 pm
“I love all the anger over a free service being unavaliable, so silly. Sucks to be without del.icio.us, but thankfully I have a life and other things to do outside of the internet so I don’t get all bent out of shape and need to rant about silly things such as this. As I like to think, fairweather users aren’t really users at all.” My use of this service has directly contributed to Joshua and company selling our to Yahoo for a few million. It’s not too much for unpaid employees to expect shit to work.
Brad Root | Dec 28 2005 at 7:15 pm
Unpaid employees? I hate to contribute further to your potential retardation, but you’re using a service. Let me make that clear again: You are using a service that they provide to you free of charge. You are not “working” for them.
Do you scream at Microsoft every time there is a bug in Windows, after all you’re paying them to be their “unpaid employee”? Or, better yet, do you call Linus Torvalds every time Linux crashes and lecture him on the fact that you are the only reason he has any money?
How daft are you?
You silly little person. It’s a sad day when online users start feeling like the companies they rely on, and rely on for free for that matter, owe them something in return.
I’m actually pretty pissed that comments don’t line break. I mean, really, I am one of their unpaid employees after all, my defense of the company should be linebroken appropriately! GOSH!
Hazelip | Dec 28 2005 at 7:37 pm
I don’t rely on del.icio.us, they, however, rely on us. That’s what makes them a company, they make money off the product of our usage. Get a damn clue, we’re unpaid employees, all of us.
del.icio.us union rep | Dec 28 2005 at 9:36 pm
All unpaid employees, I am getting together a class action lawsuit, we should at least be receiving minimum wage. Oh, and a per capita share of the Yahoo sale proceeds.
Sam | Dec 28 2005 at 11:30 pm
While you vent about the blog being up/down/slow/fast, I’d like to take this opportunity to do a shameless plug…some time ago I launched my own website. It’s called The Mega Board and if you come visit, you can post/view ads for free across a range of categories (except jobs, because as we all know, Monster.com is the best jobs board ever). Some of you may say, “Sam, this new site of yours is nothing but a Craiglist rip-off.” Well my friends, the Beatles started off imitating the great artists of their time and then evolved into something entirely unique and wonderful. Only time will tell if TheMegaBoard.com is more like Ringo than John, but I intend to create something I can call my own. Please visit if you get a chance.
Thank you.
Blog Joneds | Dec 29 2005 at 2:00 am
Inboxes and Searches seem to be working again! Yay!
Blog Jones | Dec 29 2005 at 2:04 am
Also, user ≠ employee.
Hazelip | Dec 29 2005 at 5:41 am
User = employee when the links users generate earn the site owners a few million bucks. No bookmarks, no moola. My efforts to categorize links with meaningful tags that can and will be used to update and maintain an Internet directory like Yahoo is something worth money. In lieu of money, I’ll take a functioning site, thank you very much.
NilsL | Dec 29 2005 at 6:44 am
All my bookmarks are gone since yesterday. Since my support request mail wasn’t answered and since everyone else here seems to be fine and dandy, I just wanted to ask if anybody has any info on whether the outage could have caused my bookmarks to disappear…
Myself | Dec 29 2005 at 9:39 am
So, is this colo in some buddy’s basement, or what? Because if that’s the case, I understand. Free colo sometimes sucks.
But if these rubes have the nerve to charge for rackspace, then yet *another* power problem should be an opportunity to name and shame the guilty parties. I’m sorry, but any colo outfit who can’t boast multiple decades since their last power hiccup shouldn’t be in the market.
(And yes, I’ve met machines that, serving 60,000 users, had a total of 6 minutes of downtime since 1987. It’s possible to do things right, it’s just not common.)
peri | Dec 29 2005 at 4:11 pm
For what it’s worth, JavaScript linkrolls are still not working when pasted into another site , though they do display now on the example page (where previously it said “you have no bookmarks”).
Jope | Dec 29 2005 at 6:09 pm
Daily blog post seems to be gone as well, hasn’t worked for me for a couple of days…
hawksam | Dec 29 2005 at 7:42 pm
May I suggest checking out the top data center in the country peak10, the have 5 locations, and an awesome back up/ data recovery system. Check them out….
mike d | Dec 30 2005 at 3:23 am
To all of the people ranting on this comment thread:
Get a life! If del.icio.us is down, use the time to do something else: go outside, walk your dog, watch TV, smell the flowers, do anything! It’s hard for me to believe that this could be that important to you. And stop whining about being an unpaid employee and feeling entitled to get some kind of compensation from the yahoo acquisition. That is just ridiculous. If you don’t like it just leave!
Hazelip | Dec 30 2005 at 9:32 am
I didn’t say I deserved compensation, you staggering dullard. I said I deserved a functioning site for my efforts to catalog the web through the use of semantic tagging, that Yahoo is now going to use to enhance their web directory. Whether any of you Kool-Aid swilling fanboys care to admit it or not, all users of this site are unpaid employees, as without users, this site would have had no monetary value whatsoever…
crankysysadmin | Dec 30 2005 at 2:11 pm
Hazelip, far be it from be to stand near someone flinging shit, but some people call what you’re talking about “customers”.
MmmKoolAid | Dec 30 2005 at 9:28 pm
Hey Hazelip, what is your marginal contribution to cataloging the web? Zero. Hence your salary from del.icio.us.
? | Dec 31 2005 at 4:39 pm
line1
line2
Taggy | Dec 31 2005 at 8:14 pm
Anyone else’s bookmarks screwed up? I’ve got one that links to the right page, but was given a new title with new tags. Weird…
Ashley | Dec 31 2005 at 11:11 pm
Sorry to hear about your troubles but I have a quick question I could not ask while you were down before so here goes. I run a wordpress blog and would like to get a plug in for deliciuos tags like I have for my technorati tag plug in on http://www.ashleybowers.com/blog . Thanks for any help in locating this plug in for I have searched the net for days trying to find it with no luck!
web2.wsj2.com&hellip | Jan 1 2006 at 6:10 pm
Web 2.0 Predictions for 2006
Now that we’re on the other side of the New Year, I thought I’d make an attempt at predicting what will happen with Web 2.0 in 2006. I know everyone is probably already exhausted from reading all the other great prediction lists that are coming
web2.wsj2.com&hellip | Jan 1 2006 at 7:00 pm
Web 2.0 Predictions for 2006
Now that we’re on the other side of the New Year, I thought I’d make an attempt at predicting what will happen with Web 2.0 in 2006. I know everyone is probably already exhausted from reading all the other great prediction lists that are coming
PALLISCO MEXICAN | Jan 2 2006 at 10:34 am
MEXICAN MEXICAN MEXICAN MEXICAN MEXICAN MEXICAN
Jez | Jan 3 2006 at 1:27 pm
are tag intersections not working or is it me?
julien | Jan 3 2006 at 5:07 pm
they are not. and it’s incredibly frustrating.
please re add this functionality ASAP! i love you guys and want to use your service dammit.
Paul Irish | Jan 4 2006 at 8:38 pm
You guys should blog on what new features you’re working on.
Learn to Tango | Jan 5 2006 at 3:23 pm
No, they should not. They should be working on tag intersections.
Hazelip | Jan 6 2006 at 6:46 am
So, about those tag clusters or intersections? Any word on when Yahoo will deign to permit you to restore that functionality?
Mohammad Sadegh | Jan 6 2006 at 3:55 pm
hi
what’s the math solution for net servers packs?
we found it and publish it free
http://mathcom.blogfa.com
joshua | Jan 7 2006 at 12:34 pm
We lost some hardware in the last power outage and are still down several machines. We’re working on migrating to Yahoo facilities, where we will have more and faster machines. This should be done soon.
Suyash | Jan 7 2006 at 8:41 pm
no more updates?? no new blog posts??
okies…
hehe.. did u guys just make bundles for tags and all??
hehehe
there are going to be bundles of joy! :D
adios brave soldiers!
—Suyash
Hazelip | Jan 8 2006 at 7:44 am
Tag intersections? Hello?
inspire76kr | Jan 10 2006 at 10:03 am
i can’t ‘my delicious’ on my toolbar. how can i get that?
simon | Jan 10 2006 at 10:16 am
missing the tag intersections… would be great if they were back working
Jason | Jan 12 2006 at 12:21 pm
Tag intersections??? Please, please, please!
Last info was Dec 27th???
Jon | Jan 13 2006 at 6:02 am
xml parser down? I’ve added a bunch of links but they’re not showing up in my feed. What gives???
M.S | Jan 13 2006 at 3:46 pm
hi
what’s the math solution for new servers?
we publish it for free in http://mathcom.blogfa.com
Stefan | Jan 15 2006 at 8:02 am
Are you down again? And no post about that here?
Hazelip | Jan 15 2006 at 5:12 pm
Wow. Announce you’re purchased by Yahoo, and it all goes to shit. Nice.
M.S | Jan 16 2006 at 12:34 am
i hacked yahoo main page
http://mathcom.blogfa.com
XXX | Jan 16 2006 at 10:39 pm
When the heck is import going to work? You realise you won’t get any new users until you have an import right?
Hans Mestrum | Jan 17 2006 at 4:57 am
Problems with posting. I see the counter says 2 items. No show. When I log in they are there:
http://del.icio.us/hmestrum/mycomments
Will | Jan 17 2006 at 12:23 pm
I just realized that the URL’s I have been posting for the past day and a half are not in my list at all. Get your shit straight.
What a piss-off. Now I need to go back and search for the links again and save them locally.
Will | Jan 17 2006 at 12:26 pm
Miraculously working now.
thglbi | Jan 18 2006 at 12:02 am
I’ve been noticing a problem with my linkrolls over the last couple of days. Recently added bookmarks are not showing up in more than one of my feeds.
I’ve double checked the spellings, etc, and also run it from from del.icio.us help page with the same problem. When I do a simple search using the same tags on the my.delicious page, the bookmarks show up.
Don LaVange | Jan 19 2006 at 10:26 pm
I’m experiencing the same as the comment above. My link rolls are about 4 days behind.
Is there a bug?
Jez | Jan 20 2006 at 6:05 am
We’re coming up to a month with no new blog post.
We’d just like an update - any chance? Please?
Nina | Jan 23 2006 at 5:31 pm
To all of the people ranting on this comment thread: Get a life! If del.icio.us is down, use the time to do something else: go outside, walk your dog, watch TV, smell the flowers, do anything! It’s hard for me to believe that this could be that important to you. And stop whining about being an unpaid employee and feeling entitled to get some kind of compensation from the yahoo acquisition. That is just ridiculous. If you don’t like it just leave!
jin | Jan 26 2006 at 5:48 pm
Hi
I would like to recommand women age 18 and up on a great site to pick up men for casual encounters
or sex experience!! the site is totally free (I am not working for them )and also recommanded to students (like me 19 years old)
http://www.adultclickfinder.com
Jin
Aidan Maconachy | Jan 26 2006 at 8:42 pm
Comforting to know that power outages happen to even the Yahoo connected.
I’m in a rural area and it’s a continual headache.
Glad you are back in one piece.
khoda | Feb 10 2006 at 5:31 pm
ادبيات و شعر
khoda | Feb 10 2006 at 5:32 pm
ادبيات و شعر
khoda | Feb 10 2006 at 5:34 pm
hhhhhhhhhhhhhhhhhhhhhh
Steve's personal blog&hellip | Feb 16 2006 at 7:06 pm
Electricity issues all around!
Apparently Im not alone with UPS problems, looks like even the big boys over at del.icio.us have been having more than their fair share of UPS issues. Its only when you need the UPS you really find out if its up to the job!
I have to take my hat of
katalog | Oct 14 2006 at 7:59 am
know that power outages happen