A Case StudyOnce upon a time, a small group of people joined a club. This club was using a discussion board platform none of them had heard of but they signed up to its hosts, EzBoard, by setting up user accounts. Then one day they fell out with the board's Administrator/ezOp and left to form their own club. Not knowing any better, they too set their board up on EzBoard. Having looked at EzBoard's different packages, they decided to go for the premium Gold Community package from almost the outset. This promised:
They delved further and found that the features of the Gold Community included:
How good did that sound? They signed up. The price for Gold Community status was calculated by ezboard based upon the amount of bandwidth the board consumed although how those statistics were derived was never revealed to the Administrators, just the overall daily totals. Deducted from those usage figures were the page visits of ezSupporters. Of course, from time to time there were some serious glitches, usually when ezboard - as they re-branded themselves - decided to "upgrade" their service. There were often times when members couldn't access the board, getting instead a "Server maintenance alert" page. Other times, members would find themselves logged out by ezboard and would have to log in again, often many times a day. But the members were loyal and stuck with the club and the club stuck with ezboard. The members would donate money by way of ezboard's Community Chest feature and the Administrators would spend hundreds of dollars every year for the Gold Status - in 2004, the renewal cost was $254. In order to keep the cost of renewal as low as possible, the Administrators and the Moderators paid an additional $7 to $12 to buy ezSupporter status. Then on 31 May 2005, disaster struck!ezboard initially claimed it was a hacker: Content removed due to ezboard, Inc. threats On our board, we simply saw no posting or replying capabilities from about 10.00am BST to 4.30pm BST. We were able to read, post and reply that afternoon and evening and our board data was intact ... until 1 June 2005, when we realised that overnight and post the claimed attack, we had lost all our data from 31 May 2005 back to the end of April 2004, coincidentally around the same time that Rob Labatt took over as CEO and managed to get ezboard profitable... On the basis of the board's claimed stats., that amounted to some 70,000 posts by our members. Note that at that point how Robert Labatt referred to his customers as the "ezboard Family". Thank goodness for those backup systems, eh? Still, 48 hours and our Gold board should be up and running at least with thanks to the backups. But wait! The attacker "deleted current board and back-up data." How could that be? Surely the backup should be secure? Most sources on the Internet agree that the backup should be kept at different location(s) from the original data. 1 June 2005An update from ezboard: Content removed due to ezboard, Inc. threats What is wrong with that picture? "The attacker erased all historical post data on all boards and a significant amount of back-up data". Hmm. Surely if the data was backed up properly, any external attack could not have compromised them? Either that or the attacker was somehow able to use the Internet to log into ezboard's servers, delete the data, then leave their servers and mysteriously get into the data safes to delete the secured backup media. Wow! How l33t was the h4x0r! "...we have back up data from the May 9th incremental back-up (we perform back-ups every day, but the attacker deleted many of them)". See above. Also, what of the promise by ezboard that: "Weekly backups and free restores protect your board". We weren't feeling too 'protected' at this stage but Labatt was stating categorically that they had "restored a significant number of boards and we are continuing to restore boards", that they had "restored data to many boards at this point ... and will be able to restore over half of all Gold Boards". Our hackles did rise at the way he was using an important announcement to spin the data loss by saying he was diverting resources to working on the next release of the ezboard software for a "vastly improved experience". He did, however, promise a daily noon update. Another update the same day said: Content removed due to ezboard, Inc. threats So we carried on using the board as directed. 2 June 2005So, 48 hours into the data loss - remember this? - and ezboard post this update: Content removed due to ezboard, Inc. threats So, after 48 hours, the new ETA is a further 24 to 48 hours - twice the original estimate. Whither the backups now? 3 June 2005ezboard starts to realise that they have messed up: Content removed due to ezboard, Inc. threats What is interesting here is what ezboard will and won't admit to. Let's break it down: "it will be impossible to restore full data on all boards" "Data restoration began the day of the attack" "The process is automated and each server takes approximately 6
hours to restore. We can restore between 8 and 10 servers in parallel. To
some of you this may seem like a long time to restore data on a single
server. And it is. However, we are using an innovative process to maximize
the data recovery and testing it thoroughly to make sure that (a) we don't
erase the posts that have been made since the attack and (b) the data is
being properly restored." "With quick thinking and fast action on the part of your technical
team we were able to do an emergency backup of active data." "The attacker was successful in erasing historical board posts and
the backup systems, which were password protected and on a separate
network." Of course, the cynic in me simply thinks it's ezboard messing about with their systems and making an almighty cock-up.
"What more can you tell me about the
back-ups? "Since the attack we have put out five global messages and have had
two conference calls with ezOps and admins." "I would anticipate further news on credits to be available in the
next week." "...we do have a back up of that data from a few days before the
attack. We need to recalculate the actual Community Chest balances by
taking the balance as of the date of the backup, adding Community Chest
contributions and subtracting payments from Community Chest. We hope you
agree that restoring Community Chest data is a second priority to getting
the board posts restored" 4 June 2005Well, actually, the 5th: the noon announcement was made at just before 1.00am the following morning: Content removed due to ezboard, Inc. threats Good to see that ezboard decided not to work 9-5 when the whole of their network is in meltdown. Such dedication... "We were able to increase the number of server restorations to more
than ten at a time." "Talking time to generate timelines and debate the nuance of a
changing and complex restoral process will only delay the completion of the restores. Given
this thinking it is my hope that you agree with our decision to provide
you with regular updates and an announcement when the restoration is
complete." "If a thread was not replied to in the time between the attack and when the
server your board is on was rebooted, then the information in that thread
is most likely permanently lost. We released a fix for this on Thursday
last week. It should no longer be an issue, but you may just be seeing the
effects of this now." "I apologize for not getting the noon update out to you today. We
made security changes to the site that made it impossible for the Global
Announcements to go out for a few hours."
To be continued.... {last updated 12 Oct 2007}
16 February 2006ezboard, Inc. have threatened to take action because they claim breach of copyright in their statements previously reproduced above. Whilst we claim their use is allowable as comment or review, we have removed the 'offending' elements as requested. If the comments now appear out of context then so be it: it was their choice to demand the original elements' removal. 14 March 2006Oh dear... ezboard’s CEO has stated that - contrary to what its customers were being told by its Customer Services [sic]/Help [sic] Forum staff - all ezboards will have to move to Yuku when it’s “ready for prime time” (a curious expression that, especially for a supposedly market leading message board company). The message from the ezboard CEO - whose wife is or was CFO at the venture capitalist that supported it a few years ago - is here: http://www.ezboard.com/component/option,com_mamblog/Itemid,55/task,show/action,user/id,69/ and/or aliased here: http://www.ezboard.com/labatt.blog Except that that very important announcement wasn’t actually announced! ezboard has a system whereby important news used to be propagated as a “Global Announcement”, so that the ezOps who operate ezboard boards would see them when they visited their boards and be able to choose whether or not to show all the message board visitors. It wasn’t even announced in the help forums, but slipped in by one of their staff in one of their minor forums. See for yourself - take a look here and see if you can find anything: http://p080.ezboard.com/fezboardimportantannouncements That contains such earth-shattering things as suggesting that ezboard users upgrade their browsers. Woohoo! And yet, when it comes to its customers being forced to change their message boards or communities from one platform to another, it chooses to keep this quiet from the vast majority of its users, most of whom will never even look at the help [sic] forums or the main ezboard web site. Why? Maybe it’s to ensure that ezboard, Inc. continues to gain an income stream from its ezSupporter feature? ezboard operates its free boards on the basis of advertising revenue. Knowing that people don’t like advertisements all over the place, they presently offer two ways to for users and board ‘owners’ to operate ad-free:
In the latter case, visits by ezSupporters to Gold Communities are claimed to be deducted from the overall page views when ezboard calculates how much the annual or six-monthly cost will be so it’s always been the case that Gold Community ezOps are urged either to activate their Community Chest (members can make payments to ezboard that can then be used to fund the Gold status for that board and nothing else, so they cannot be paid out to the ezOps if the boards move away from ezboard) or to solicit their members to subscribe as individuals to ezSupporter. But if you read Labatt’s blog entry, he says that in order to remain ad-free, existing ezboard Gold Communities “will need to have a minimum of 6 months left on their subscription when they move”. So tough luck if you renew your Gold subscription and then have the move over to Yuku foist upon you the following week, for instance, as you’ll lose that benefit (and one that isn’t exactly cheap for a busy board). Likewise, if you renew for a year’s ezSupporter and the boards you visit transfer or are transferred over to Yuku, you’ll then see adverts on them because the ezSupporter programme isn’t being carried over to Yuku. Cynical? Maybe, but then ezboard promised Gold Communities weekly backups as part of our paid subscription and yet when push came to shove and 70,000 or so messages on our ezboard alone were lost (according to ezboard’s average stats. for it) the backups weren’t available. A year’s messages and threads gone. ezboard’s answer to criticism (apart from banning those critical of them) was that the service was provided “as-is”, so tough: they were only transitory messages anyway. Oh and one of their support/customer service [sic] staff has posted a message saying that she thought it “hilarious” that people who’d been saying that ezboard wasn’t working properly - it isn’t, by ezboard’s own admission - were now complaining that “if it ain’t broke, don’t fix it”. I’m sure it’s “hilarious” that people don’t like change - one reason why people stay with ezboard - or that they’re not impressed by a new system that may well cost them more, will serve ads. at them and presently at least causes regular browser crashes and is slow as swimming in treacle.
In the meantime, you could always try some of these services: we have, to make sure our web sites and discussion forums work, unlike some other players in the market...
Here are some other ads. to help us make even more money. We've probably not used these advertisers' goods or services ourselves before, so we don't vouch for them.
|
