STH Turns 10 Years Old and Taking a Checkpoint in the Story

To celebrate our 10th anniversary, I am going to share the story of why STH is “ServeTheHome” when we review tons of data center gear. I also wanted to send a few thank yous and talk a bit more about the journey. Finally, I wanted to discuss a bit more about our plans for the next chapter in STH’s evolution.

The Untold Story: Why ServeTheHOME?

2008-2009 was a very interesting year for me personally. I was still working in PwC’s management consulting practice working on projects like the HP due diligence of EDS before it announced the acquisition and post-merger work. I was on planes constantly and that took a personal toll. I was engaged, then disengaged. During that period, my quite substantial DVD collection was no longer in my possession. Life lesson learned the hard (expensive) way: backups are important. My James Bond DVD collection was no more.

This was a time before the online movie services were great and took care of backups. I set out on a project to build a home backup server. Ten years ago, and on a senior associate budget living in Silicon Valley, that was not an easy task. That started a process of learning RAID controllers and SAS drives purchased off of ebay and hoping they were still good.

On June 8, 2009 STH started. We are not going back to the archives since ten years ago STH was not meant to be a large review site. It just happened. I had a shared hosting account and figured I could deploy WordPress easily so why not make a blog. STH was born.

For the first year or two, there were a few guides, a few benchmark pieces, but it was treated and being funded like a hobby.

Our entry into reviewing server products was simply buying parts and putting what we had online. The first product we reviewed for a company was after Douglas Herz at Supermicro sent me a note asking to review products after our Supermicro X8SI6-F about a year and a half into the STH journey. I drove over to Supermicro that had a very small San Jose campus then and met Doug in-person. Doug was the first person to have the vision that STH should start regularly reviewing server products. I had been keeping STH going for some time as a hobby with intermittent posts until then. My day job at PwC was going well, as I made manager in 2010 and would quickly make director in 2012. There was not a lot of free time, but I only wanted to take on the responsibility of reviewing someone’s products if I could not do it in a timely manner. Review photos were not done in a studio or a data center, instead, they were done on the kitchen table, handheld, and posted directly from camera media.

I had lunch with Doug a few months ago. In hindsight, he was one of the first mentors I had for STH and shared great bits of wisdom both on the STH side and at a personal level.

While STH started out as serve the “home”, it was all about taking data center gear and using it as it was intended outside the data center. Over time, that transitioned to reviewing a larger assortment of gear that may be used in a home office or SMB. The next natural place to look was to the larger segment of the market where the gear was intended for use.

A Few Rough Milestones on the STH Journey

Generally, the STH journey has been great. There were a few milestones however that were less than stellar.

Logos: Why I Should Not be a Graphic Designer

What happens when you need a logo, but you have no money for a graphics designer and do not want to invest in Photoshop/ Illustrator (Adobe Creative Cloud was not around yet, so those were big investments.) You download GIMP and think this is the best option.

In 2012, we had a minor update to the design:

Then Photoshop happened and with 3D text, the abbreviation to STH happened later in 2012. If this does not look like someone who just found a 3D text tool, I do not know what does.

Since 2012, ServeTheHome has essentially dropped the “home” bit and has used STH only in our logo.

In 2014, we had someone design the newer logo.

It turned out that the older gradient logo looked (terrible) in a lot of places. Over the next few years, we transitioned to our current logo design.

Dealing with Early Growth

STH started its life on shared hosting, the way many websites do. We actually moved STH to the cloud early using Amazon AWS EC2 instances for STH hosting. By early 2013, we were having issues. Our, at the time, hard drive backed instances were causing seg faults for WordPress leading to site instability. In January 2013, the spreadsheets dictated, it was time to get into the data center. We started our series: Falling From the Sky: Why STH is Leaving the Cloud.

Yes, we still had that logo. By February 2013, we had HP switches, and several Dell C6100 servers, retired from Twitter/ Facebook pre-OCP days, installed in a quarter cabinet.

These Nehalem-based servers were built by hand, and setup only a short flight away.

We later had the worst day for STH. More on that next. Over the period of the next two years, the colo changed quite a bit.

Later in 2014, we moved to a half cabinet. By 2015 we had added a full cabinet in another data center.

We talked about this a bit in last year’s Falling From the Sky Part 4 Leaving the Cloud 5 Years Later.

Our original business case for leaving AWS holds strong to this day. Since doing this project over six years ago, all of the hardware, labor, colocation fees, bandwidth, flights, remote hands, and shipping fees have been paid for just in what we have saved on AWS bandwidth costs (e.g. if compute and storage were free.)

This experience gave practical experience and also is a great case study on money that can be saved by self-hosting instead of using the cloud.

The Darkest Day: June 5, 2014

June 5, 2014 was the day that STH died. I remember the day vividly. I was walking out of Trimble Navigation’s offices after a meeting and suddenly I was getting alert after alert while getting in my car. STH did not respond on my phone.

We were running replication across multiple nodes which is usually a fairly safe configuration. There was one important issue. We were running all three nodes in the same Dell C6100 chassis. This was because we were preparing for an upgrade and thus we had all of the nodes in a chassis while we prepared for the upgrade. Remote hands could not diagnose the issue.

Remote hands even went to the point of unplugging drives and trying to get data off them. Every drive did not respond. We sent a drive that was the closest to functional to DriveSavers and they could not get anything off it.

We were using Kingston E100 400GB drives. These were the 400GB drives with an additional 400GB of reserve NAND for a total of 800GB of NAND with 400GB usable.

Although these were fancy drives of their day, we think what happened is that they experienced a power inrush event which killed the drives in each node. In the same nodes, our Intel SSDs did not fail because they had power inrush protection.

Lessons learned:

1. Use more than one chassis for replicas and backups
2. Use different types of drives
3. Backup offsite
4. Overbuild for hot spares
5. Did we mention more backups and replication?

At that point, we had to make a real decision. We had some older backups and an extremely painful way to get back almost all of the incremental missing data. It was, however, a painful process that took over a week to complete.

Even with overbuilding, the economics of self-hosting still work. Over time, this lesson was an important one to go through the pain of.

The day STH recovered was the day that I decided STH needed to grow. A little over a year later, I left PwC and started working on STH full-time. The alternative was to abandon STH and focus on the PwC partner process.

2019: Catching Fire in a Good Way

In 2019, as the team expanded, I have been pushing for more content. For most of 2018, we were still on a 2 review per week and 5 pieces of content per week schedule. In 2019, we are now publishing every day and pushing four reviews per week. We have had to do this just to stay ahead of the amount of content that we need to publish.

We now review hardware from every major server vendor except IBM Power and Huawei. We have been pushing on CPU reviews and in the next few months, you will see more types of content. Some may have noticed that John has taken over many of the CPU reviews and I am doing more perspective pieces such as this week’s Why the Intel Xeon Platinum 9200 Series Lacks Mainstream Support and AMD EPYC Rome NAMD and the Intel Xeon Response at Computex 2019.

A Few Thank Yous

First and foremost, I wanted to thank William Harmon for being awesome. He was the first one to start reviewing products for STH on a regular basis. Cliff is our rock that has been handling many of the product announcements that we cover across multiple domains. Rohit has focused on networking for STH and helped push that area of coverage. John recently took over CPU reviews. Eric has been working on some special pieces for us since joining a few weeks ago.

Virginia joined the STH team just over a year ago and has been teaching me a lot about the process of publishing. She previously worked with folks like Anand and has been teaching me a lot about the industry’s history.

I also wanted to say a quick thank you to Jeff, Mark, Stephen, Bryan, and Pieter who used to write for STH when we were much smaller. Your support in the early days helped keep STH going.

A few other thank yous are in order. TPM of The Next Platform sat me down about two years ago during a Voodoo Doughnut adventure and explained why I was, and still am, a moron at publishing. Fritz Nelson also helped me see a path to push STH just after. Ian Cutress helped increase my knowledge of chips, platforms, and some other use cases (e.g. gaming) as well as giving me a great perspective. Paul Alcorn likewise has helped shape STH with excellent perspective and taking over my old role reviewing Xeons at Tom’s Hardware, and much more. Also, I wanted to send another thank you to Bob O’Donnell for helping me understand more about the analyst industry that is often at the same events I attend.

Our STH Forum moderation team has changed over the years, and most want to remain anonymous. I am simply going to say thank you for making our forums a great place to hang out. The community is smaller, but it is generally excellent.

STH has some fairly broad support from the industry. We simply could not continue pushing boundaries at STH without this support from the community of vendors.

Finally, I wanted to thank our readers. For me, and most in a similar line of work, there are days that are more challenging than others. There are days that I am simply tired from too much travel and lack of sleep. Those are the same days that reader comments, a cool forum post, a cold e-mail out of the blue, or even just seeing an old article pop up popular elsewhere is extraordinarily energizing. In 2014 when STH died, this is what ultimately pushed the site to recover and grow. There are few things that one can find to do that energize them from a 4:25AM wake up to 11PM every day. STH is one of them.

Final Words and Next Steps

This was planned to be a 500-word quick post. Instead, it turned into a tome. It could have easily doubled in size adding experiences and thank you notes. Instead of looking at the past, I wanted a quick glimpse of where we are heading.

In the next few months, the plan is to fairly aggressively grow the site. We have been stuck on a ~2% compound monthly growth rate for some time. My next challenge is to push beyond that rate in a meaningful manner while not trading page views for lower quality. We have seen many sites go down that path and falter later for it. To this end, some of the projects we have been working on have taken longer than expected and I need to do a better job on execution.

As we travel to Asia, the US, and in Europe, we are getting signals that sound as though a slowdown may be coming. For the past six months, I have been working with our team to lay out, and execute on, a plan to ensure STH will be around through the downturn, and come out stronger on the other end. Given that we test servers, our operating expenses are very high, so a continuity plan is important.

One will have noticed that in 2019 beyond the editorial content we have started doing more interviews and other types of content. Now that we have a solid base, and frankly a robust pipeline to the point where we are planning two or more months out, we are now taking the opportunity to ask what else can we do. To that end, I have an ask. If there is something you like to see, or something you do not like, put it in the comments, send me an email, put a post in the forums. Even better, if you want to help, we are always looking for domain expertise we do not have.

Thanks for being an awesome community STH!