So many things can go wrong and often do, but I spend a good third of my time working on infrastructure, monitoring, and analytics so that they don’t.
Here’s what happened: At approximately 4:30pm PT feed fetching ceased. The feed fetchers were still working, which is why my monitors didn’t fire and alert anybody. But I have a second large Mongo database server used exclusively for collecting data about feeds being fetched. There are approximately 75 servers dedicated to feed fetching. These analytics look at average fetch times on a per task server basis. I use these analytics to ensure that my task servers are humming along, as they each use a ton of network, cpu, and memory.
This Mongo analytics servers works in a curious way. If you right-click on a feed and go to Statistics you’ll see the feed fetch history for a feed, stretching back a hundred fetches if the feed has had any issues in fetching. I keep these statistics on an analytics server separate from the regular Mongo server. I do this so that if the mongo analytics server goes down, everything will operate normally.
But the mongo server didn’t go down. It merely gave this error:
OperationError: Could not save document (Can't take a write lock while out of disk space)
Mongo continues serving read queries while not allowing write queries. I didn’t plan for that! And it gets worse. The way MongoDB stores data is that is just keeps growing, even as you delete data. NewsBlur only saves the last few fetches, but deleting old fetches doesn’t give you back any disk space. Every other database server I use has an autovacuum process that takes care of this maintenance work (PostgreSQL, Redis, Elasticsearch, but not MongoDB). It’s unfortunate that this is yet another instance of MongoDB being the cause of downtime, even though the fault lies with me.
The server that is meant to only be used to ensure things are working correctly was itself the culprit for feeds no longer being fetched. This is the ironic part.
Now comes the painful part. On Wednesday morning (yesterday) I packed my car and headed down to Big Sur to go backpack camping for the first time. I’ve car camped plenty of times, but I felt confident enough to pack my sleeping bag and tent into a big bag and head ten miles into the woods of coastal California.
I headed out, away from cellular service, at 4pm PT, half an hour before the analytics server ran out of disk space. And then returned nearly 24 hours later to a bevy of alarmed tweets, emails, direct messages, and a voicemail letting me know that things were haywire.
But the real problem is that I set a vacation reply on both my personal and work email accounts to say that I’d be out until September 3rd. Now, I hired a firm to watch the servers while I’m at Burning Man starting this Saturday. But I figured I could get away with leaving the servers for twenty four hours. And I neglected to tweet out that I’d be gone for a day, so theories cropped up that I was injured, dead, or worse, ignoring the service.
If you’re wondering, I think about NewsBlur first thing in the morning and last thing at night when I check Twitter for mentions. It’s my life and I would never just give up on it. I just got cocky after a year and a half of nearly uninterrupted service. NewsBlur requires next to no maintenance, apart from handling support requests and building new features (and occasionally fixing old ones). So I figured what harm could 24 hours of away time be? Boy was I wrong.
If you made it this far then you probably care about NewsBlur’s future. I want to not only assure you that I will be building better monitoring to ensure this never happens again, but to also offer anybody who feels that they are not getting their money’s worth a refund. Even if you are months away from payment, if you aren’t completely satisfied and think NewsBlur’s just about the best thing to happen to RSS since Brent Simmons released NetNewsWire back in 2004, then I want to give you your money back and let you keep your premium account until it expires.
I would like to also mention how much I appreciate the more light-hearted tweets that I read while on the frenetic three hour drive back to San Francisco from Big Sur. I do this for all of your happiness. If I did it for the money I’d probably find a way to juice the data so that I could at least afford to hire an employee. This is a labor of love and your payment goes directly into supporting it.
First comes feeds, then comes training, then comes social for those days when its raining.
The world’s most popular RSS feed reader now supports the world’s best RSS feed reader backend. Download Reeder 2 for Mac and iOS.
It was only a few months ago back in April that I launched the new font and style manager. Today I’m coming back to finish the job.
The style manager now has a new control for choosing the font size of feed titles and story titles.
Choose from extra small to extra large. These new font sizes not only give you extra customization, but they allow you to have bigger touch sizes for folks using touch screen monitors.
And please continue to talk about how much you use NewsBlur. Every tweet that talks positively about NewsBlur makes all of this work worthwhile.
Like it says on the tin, you can now go back and see what you’ve read with the Recently Read Stories feed. It’s on the bottom of your feed list, just above the saved stories.
This feed records all of the stories you’ve read in feeds, shared blurblogs, and even in feeds you’re not subscribed to. It does not record stories that are marked read as part of a mark all/previous as read action.
You can now temporarily turn off sites by going to Manage > Mute Sites. This is for those feeds you want to keep but ignore for a while.
I use this to mute craigslist feeds that I no longer want to see in my unread stories list, but I want to keep just in case I need to perform a specific craigslist search without having to go through the whole process of recreating the RSS feed.
It’s also useful for subscribing to high volume sites and only reading them when you want to know what’s happening but without having to remember to re-subscribe. Such a simple feature but with a ton of utility.
After launching Saved Story Tagging earlier this year, I realized that there is an inherent categorization to every saved story: the site. But instead of just allowing you to search by site, I created a new view that you can now use to see stories saved by site.
The neat thing about this new view is that it works for folders, too. If you click on a containing folder, you’ll see the saved stories from all of the sites inside that folder. And thanks to last month’s full text search, you can search all of these saved stories.
And you should expect that these features will eventually make their way to iOS and Android.
Last Summer Simple Search was launched, giving you the ability to search a single feed at a time. Obviously not ideal, but it was far less effort than the big enchilada: full text search across every site you subscribe to. It took a few months to come back to attack the full problem, but that means I could also take the time to do it right.
Full text search works just like the old simple search did. Open a folder or site and search at the top of the screen. Results come back instantly as you type. And this will automatically search in story titles, full text content, authors, and tags.
Because you’re searching for a story, hidden stories are also shown in the results, albeit with a red unread mark (as shown above). Also, in order to handle the demands of full text search, the first time you perform a search the system will index all of your feeds, which can take a couple minutes. It’s a one time process and you will see a progress bar that will update automatically when finished.
Since this was one of the most requested features since day one, I’m now looking to decide on the next big feature. There’s a few big ones to hit: an organizer that surfaces inactive feeds that no longer get read, a discovery platform that helps you find users, stories, and feeds related to what you like and what you may not even know you like yet, and porting many of these web-only features to Android and iOS.
NewsBlur has a free and open API that all of the native mobile apps and website are built on. But the API is not just for official apps. Numerous third-party developers have built apps on the API and today I’m proud to announce a new native iOS app has launched with NewsBlur support.
Unread, by Jared Sinclair, is a unique RSS reader, with a focus on simplicity and cleanliness. Sinclair built Unread because he wanted to see a different kind of reading experience, one where he could focus more on reading well written stories than consuming lots of content. Unread app has a different feel than any other news reader I’ve seen and I think many of the choices made by Sinclair work in a quiet harmony.
In Sinclair’s own words:
Most RSS apps are patterned after email. Noisy parades of dots, dates, and tags trample over their screens. Their source lists look like overflowing inboxes instead of stately tables of contents. Toolbars bristling with options obscure the text. Putting it bluntly, using these apps feels like work.
NewsBlur’s native iOS app certainly does fit this description. And if you want an incredibly well-made alternative, look no further than Unread, available on the App Store.
This is not just any font and style manager. You now have control over font size, line spacing, story layout, and type faces directly in the reading view. This nifty popover gives you quick access to all of these customizable features.
The type faces come from Hoefler & Co. (née Hoefler & Frere-Jones) which is the premier web type foundry. It’s also one of the most expensive, but that’s where the incredible font quality comes from.
The four new fonts are:
These new fonts are available immediately for premium users. Another great reason to pay for NewsBlur.
In this industry, you gotta be tough.
I’m just kidding. We’re a bunch of literates who enjoy reading so much that we built our own news readers. But when a behemoth like Google makes a call that places you at the business end of 100,000 frantic power users, reminding yourself how tough you are is one way of dealing with the madness.
Google announced Reader’s sunset at 4pm on March 13th, 2013. At that point I had spent three and a half years building my vision of a better news reader. I clearly wasn’t doing it for the money, since my paltry salary didn’t even cover my market rate rent in San Francisco. RSS was a decidedly stupid technology to piggyback off of to try and cover that financial disconnect.
Take a look at this graph. It shows NewsBlur’s income versus its expenses for the past 16 months. Just look at those few months before the Google Reader shutdown announcement in March 2013.
It was never hard to justify to others why I worked on a news reader for three-some years, partially because I’d been justifying it to myself for so long. I had the delusion that it would all work itself out in the end, so long as I kept pushing my hardest and shipping features users wanted. And, at the time, with 1,000 paying subscribers, it certainly felt like I was getting somewhere.
If you’re curious about why expenses are so high, think about what it takes to run a modern and popular news reader. This graph breaks down expenses for an average month from the past year.
Fast forward a year and let hindsight tell you what’s what. I was irrational to think that I could make it on my own in a decaying market, what with all the air sucked out by Google. But that three year hallucination kept me persevering to build a better product, which positioned NewsBlur well as a strong candidate for a Reader replacement. When the sunset announcement dropped, it didn’t take long to fortify the servers and handle all the traffic. NewsBlur permanently ballooned up to 20X the number of paid users. People flocked to NewsBlur because it was among the furthest along in creating real competition. As we say on NewsBlur, the people have spoken.
I run a very opinionated news reader. If you think somewhat like I do, you couldn’t be more pleased with the direction NewsBlur goes. But this is still a power tool, and in a world of casual readers who don’t care where their news is coming from so long as it’s in their interests and matches their biases, NewsBlur is the coffee equivalent of the AeroPress. Most people want drip coffee and they don’t bother wasting mental energy on caring about the difference in taste or quality. It’s a binary to them: coffee or no coffee. There’s nothing wrong with that, they just choose to focus on other things more important to them than the sourcing or control they have of their coffee.
Many competing news readers are visual and offer a similar experience. When you want to give up control in exchange for the digested output of sophisticated and heartless algorithms, they’re your best bet. When you want to exert control and know what you want and from which sources, NewsBlur is the only option. No other reader gives you training, statistics, and sharing in one multi-platform app. Nobody else cares so much about RSS as to work on a news reader when it was still a financial inevitability of failure.
If the past is any indication, NewsBlur is going to continue to see many more improvements. This graph of contributions from the past 365 days shows my level of unwavering dedication.
Meanwhile, I’m using the windfall to develop a secret project that will complement NewsBlur in a way that hasn’t been tried before with any reader. And if that fails, I’ll find an even better way to make my users happy with their purchase. If you thought I was relentless before March 13th, 2013, just wait until you see what I’m capable of with the finances to build all the big ticket features I’ve been imagining for years.
And while you’re here, do me a favor and tweet about NewsBlur. Tell your followers, who are probably looking for a better way to read news, about how much you rely on NewsBlur. Reading positive tweets about NewsBlur every morning (and afternoon and evening and before bed) make this the best job I’ve ever had.
Today marks the release of version 4.0 of the NewsBlur iOS app. To illustrate the significance of this release I’d like to talk about where the app has been.
The first version of the NewsBlur iOS app was actually just an iPhone app. It launched in October 2011 and only allowed you to read your feeds individually. Within months the river of news (reading by folder) was added, along with better sharing controls, intelligence training, sharing to your blurblog, and the May 2013 re-design.
But similarly to the web, I wasn’t particularly proud of the app until I hit a major milestone. On the web, I hit that milestone with the May 2013 re-design. In the blog post announcing the wide ranging re-design, I wrote:
Not to say that NewsBlur was ugly before today, but it certainly didn’t have the loving embrace of a talented designer. So without waiting another moment (or month) I proudly present the NewsBlur redesign.
Fast forward to today where I am finally releasing an iOS app that I’m proud of. This one comes with a ton of great features, both for iPhone and iPad.
If you want to tweet something along the lines of “Holy crap, the NewsBlur iOS app is nice! Only took four years” that would be alright. Download the free NewsBlur iOS app, and if you find yourself enjoying it, please leave a positive five star review on the App Store.
Wouldn’t it be nice if there was a way to automatically copy your saved stories over to Evernote or Twitter or Pinboard? What about an automatic way to keep track of your unread focus (trained) stories in Buffer or Delicious or Dropbox?
IFTTT (if this then that) is a free web service that automates the movement of data from NewsBlur to dozens of other web services. It also handles movement of data from dozens of other web services back into NewsBlur. Today IFTTT is launching a NewsBlur channel where you can take advantage of their incredible glue that binds dozens of web service like NewsBlur in unique and customizable ways.
The best part about these IFTTT recipes is that the triggers can be customized to fit a specific saved story tag or blurblog or folder or feed. When you create a recipe using NewsBlur, you can choose which of your own feeds or tags you want to use.
You can also choose to only use unread focus (trained) stories. So you can filter out stories you don’t want to read and highlight the stories you do want to read using the Intelligence Trainer.
Here’s a sample of IFTTT recipes that you can use today for moving your data from NewsBlur out to other services:
You can also take data from other services and move it into NewsBlur:
Mix and match and create your own IFTTT recipes using the NewsBlur channel. And if you put together a good recipe, it’ll get featured both on IFTTT and on NewsBlur.
It’s one thing to follow a handful of sites and use NewsBlur’s training to only read the stories you want to read. But sometimes you want to come back to stories long after you’ve read them. You could save the story, but then you would have to either scroll down your saved story list to find the story, or use the new search feature to find it by title or author.
Today I would like to introduce a big new feature: saved story tagging. It works like this. When you save a story, it is automatically tagged with all of the folders that the site is part of.
After the story is auto-tagged by folder, you can then add your own tags, which are autocompleted by previously used tags. You can also click a button to auto-tag the story with its own tags, easily clicking on the tags you want to remove.
You can also search per-tag and soon you will be able to rename and delete tags. There are more plans to integrate tagging into both iOS and Android apps.
Saved story tags also have RSS feeds. This is a big deal, since this allows you to save stories and have them automatically ingested by IFTTT for use elsewhere. You already could share stories over your blurblog, but now you have a private RSS feed that gives you far greater control.
Stay tuned because there are even more big features coming soon. Telling your friends about NewsBlur, whether on Twitter and Facebook or when spying over their shoulder and noticing that they’re hitting the same dozen sites over and over again without letting the sites come to them with new stories, goes a long way towards getting new features built. More users means more premiums which means more features getting built. It’s a vicious cycle.
A few small new features to get your first full week of the new year started off right.
A common feature request has been automatic syntax highlighting in blog posts. This feature uses auto-detection to highlight code. For short snippets of code, this means that they could have inaccurate highlighting, although in testing it’s not so bad to have a few words in various colors. But for any substantial amount of code, you’ll have an easier time following along with syntax highlighting.
Videos from Youtube and Vimeo (and a few others) are now stretched to fit the window, resulting in bigger video previews.
Previously, if you clicked the little footnote reference1, you would be taken to a new window. The Feed view now smoothly scrolls you to the footnote, where you can then return back to the footnote. Easy!
Like this. ↩
The single biggest criticism I’ve heard of the Android app is that it can be slow when loading feeds and then loading stories. That changes today with the release of version 3.5 of the NewsBlur Android app.
Here’s what’s new:
And remember that only a month and a half ago, version 3.0 hit the Play Store and the Kindle Store with these great features:
There’s more coming to the NewsBlur Android app, now that it’s reached speed parity with the iOS app. Look forward to lots of updates, improvements, and new features.