Blog – Stack Overflow

As you know, we sort answers (and sometimes questions) in simple descending score order by default. Score is defined as upvotes minus downvotes. Way back in February, Mike Schiraldi of Reddit emailed us about an alternate sorting mechanism.

After about 6 months of testing, It looks like Reddit has implemented this algorithm, and you can read about it courtesy of Reddit guest blogger Randall Munroe (aka XKCD):

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there’s not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes — figuring that by the time it’s also gotten 40 upvotes, it’s almost certain it will have fewer than 20 downvotes. And the best part is that if it’s wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top — and when it gets that data, it will quickly correct the comment’s position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom.

The original article, How Not To Sort By Average rating, elaborates on the math.

We need to balance the proportion of positive ratings with the uncertainty of a small number of observations. Fortunately, the math for this was worked out in 1927 by Edwin B. Wilson. What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? Wilson gives the answer. Considering only positive and negative ratings (i.e. not a 5-star scale), the lower bound on the proportion of positive ratings is given by:

He also provided some sample Ruby code that implements the above formula:

def ci_lower_bound(pos, n, power)
    if n == 0
        return 0
    end
    z = Statistics2.pnormaldist(1-power/2)
    phat = 1.0*pos/n
    (phat + z*z/(2*n) - z * Math.sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
end

pos is the number of positive rating, n is the total number of ratings, and power refers to the statistical power: pick 0.10 to have a 95% chance that your lower bound is correct, 0.05 to have a 97.5% chance, etc.

(other implementations in different languages were provided in this reddit thread.)

I met Mike in person at the LA DevDays, where he presented on Python. He reminded me about this article, and we discussed whether it would work on Stack Overflow. I generally like it, but there are some important differences between Reddit and Stack Overflow:

1. Statistically speaking, it is quite rare for us to get a question with more than 30 answers.
2. Since votes are limited to 30 per user per day, we have a much lower volume of voting overall than Reddit.
3. As downvotes cost reputation on Stack Overflow, the overall incidence of downvotes is probably much lower here than it is on Reddit, where downvoting costs nothing.
4. By the time a question gets to more than 30 answers, and has tons of voting, it’s arguably not a very appropriate question for Stack Overflow.
5. I worry that a sort order where lower scoring items are ranking higher than higher scoring items will confuse users. Score has its problems, but it is immediately understandable — low numbers are low, high numbers are high.

While this algorithm is definitely cool — and a clear improvement for Reddit users — I am not sure it’s as clearly useful for Stack Overflow.

A collection of clips recorded at the San Francisco DevDays conference, including Joel Spolsky, Mark Harrison, Jeff Atwood, Scott Hanselman and Rory Blyth. This episode runs a bit longer than usual.

• Joel Spolsky on web usability
• Mark Harrison on Python and the Norvig spell checker
• Rory Blyth on iPhone development
• Scott Hanselman on ASP.NET MVC 2.0
• Jeff Atwood on Stack Overflow
• Ad-hoc roundtable podcast with Scott, Rory, Joel, and Jeff backstage at DevDays. Warning: extreme ramblosity ahead!
• Joel explains his Duct Tape Programmer post. Apparently DevDays is a duct tape conference, and this section of the recording is a duct tape podcast.
• Some discussion of the ubiquity of mobile code. Also, if you are nostalgic for the era “when development was hard”, the consensus is that you should be doing mobile development today on iPhone, Android, Windows Mobile, or Symbian.
• Rory elaborates on his experience with (and effusive opinions on) iPhone development to date. Is coding in Objective-C best accompanied by a flux capacitor, New Coke, and Max Headroom? Also, his excitement for MonoTouch.
• Joel and Scott put on their amateur language designer hats and have a spirited discussion of type inference and Fog Creek’s in-house DSL, Wasabi.
• Scott covers some of the highlights of new and shiny features coming in the Visual Studio 2010 IDE, the C# 4.0 language, and the ASP.NET MVC 2.0 web framework.

Our favorite questions this week:

• How do I create unicode smileys? So far beyond :) it isn’t even funny. Who knows, you might even learn some typography along the way!

If you’d like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have a dedicated phone number you can call to leave audio questions at 646-826-3879.

The transcript wiki for this episode is available for public editing.

 

 Podcast 71 [01:46:47m]: Play Now | Play in Popup | Download

Our apologies for the all-site outage today. According to our Pingdom monitors, we were down from 7:18 PM PST to 9:43 PM PST. There goes our vaunted envy-of-the-industry three nines uptime guarantee!

Apparently there was a router meltdown at our ISP, Peak Internet. They promised pictures of the (literally?) melted router via an update on their Twitter account. If they come through, I’ll post the pictures here for our viewing pleasure.

(not as dramatic as I had hoped, but there are some definite scorch marks around that solder!)

At any rate, if you guys and gals could send a few less fiery packets of network doom to our ISP’s routers, we’d appreciate it.

We made a few key technology bets when we created Stack Overflow:

• OpenID
• Markdown
• ASP.NET MVC

I’ll defer the discussion on the other two items for another day, but after spending a year immersed in Markdown — the lightweight markup language we use to format posts on all Trilogy sites — I have some thoughts I’d like to share.

We knew early on that there were a handful of Markdown Gotchas, thanks to the sage advice of John Fraser (who, sadly, I have completely lost contact with.) Based on those gotchas, we quickly adjusted our Markdown support to fix a few obvious things:

1. Removed support for intra-word emphasis like_this_example
2. Added auto-hyperlink support for https:// URLs in posts

Apparently github also uses Markdown, and they independently arrived at some of the same conclusions we did — synthesizing something they call GitHub Flavored Markdown.

1. Removed support for intra-word emphasis like_this_example
2. Added auto-hyperlink support for https:// URLs in posts
3. Automatic return-based linebreaks instead of “two spaces at end of line” linebreaks
4. Support for some magic strings that auto-convert to GitHub specific links

Since GitHub and Stack Overflow match exactly on #1 and #2, it’s fairly safe to say that those are in fact deficiencies in Markdown, at least for a programming audience. (Though I’d argue they apply to general audiences, too.)

As for #3, that’s one I hadn’t considered. In normal Markdown, this:

Roses are red¶
Violets are blue¶

Will render like this:

Roses are red violets are blue

The Markdown answer is to add two spaces at the end of the line (or a literal <br>, I suppose).

Roses are red  ¶
violets are blue¶

Although it’s easy once you know the trick, this is far from intuitive to most. I’m reminded a bit of the double-click mouse problem. I wonder if we should adopt the GitHub linebreak approach here.

As for the fourth item, when text is entered in these specific formats …

* SHA: be6a8cc1c1ecfe9489fb51e4869af15a13fc2cd2
* User@SHA ref: mojombo@be6a8cc1c1ecfe9489fb51e4869af15a13fc2cd2
* User/Project@SHA: mojombo/god@be6a8cc1c1ecfe9489fb51e4869af15a13fc2cd2
* \#Num: #1
* User/#Num: mojombo#1
* User/Project#Num: mojombo/god#1

… those magic strings are detected by the GitHub Flavored Markdown and auto-converted into GitHub specific hyperlinks. Something similar has been proposed on meta for internal Stack Overflow references, so this is an idea we’ve been entertaining for some time as well.

Markdown is remarkably flexible, because it allows you to intermix a narrow list of whitelisted HTML tags with Markdown “fancy ASCII” syntax in a fairly logical way, at least most of the time.

So, now that you’ve had a chance to mess around with Markdown for a year — what are your thoughts?

In this episode of the podcast, Joel and Jeff discuss DevDays, the diversity of Stack Exchange sites, the debut of CVs and careers on Stack Overflow, and the viability of WiFi at tech conferences.

• Stack Exchange is now officially in public beta! There are a huge number of sites running on the Stack Overflow engine. Far more than I expected at this early stage, anyway.
• The Stack Exchange sites are pushing the boundaries of the specific audience (that is, programmers) we designed it for. Consider the audience overlap between answers.onstartups.com, epicadvice.com, and moms4mom.com. I was getting usability reports from my wife on that last one, which was quite surreal. Also surreal: that Jon Skeet is a top user on one of the above. You’ll never guess which one!
• Do some of the Stack Exchange sites compete with Stack Overflow? Such as ask.sqlteam.com and snippetgood.com? Not necessarily; if you’re particularly enthusiastic about some niche, you’ll get more questions and tighter focus of community by going to site dedicated to that topic. 
• Joel feels that Stack Exchange works so well as a support forum that he’s shutting down all the other online FogBugz web support tools in favor of fogbugz.stackexchange.com.
• What’s the minimum number of knowledgable, invested users you need to have a functional online Q&A community? Joel says one (!). I think it’s more on the order of a few dozen. The software part is easy, the real hurdle is this: can you rustle together a core community of a few dozen enthusiastic, knowledgable folks?
• An extended discussion of our new careers section of Stack Overflow, which we launched last week. Joel sort of wrote the book on this topic, with Smart and Gets Things Done: Joel Spolsky’s Concise Guide to Finding the Best Technical Talent. Our careers approach grows out of Joel (and my) dissatisfaction with the current status quo. It sucks, and we’d like to build something better.
• This is the philosophy behind careers.stackoverflow.com : smart companies should be pursuing good programmers, and not the other way around. We also want to cut out the cheesy for-pay contingency recruiters (or any other middlemen, for that matter) from the mix, and directly connect passionate programmers with companies that understand the value of programmers who hit the high notes.
• This is Fog Creek’s guarantee for every service they charge money for: “The Fog Creek Promise: If you’re not satisfied, for any reason, within 90 days you get a full refund, period, no questions asked. We don’t want your money if you’re not amazingly happy.” Stack Overflow has adopted this promise as well. Why don’t all companies do this? Why would you want to keep an unsatisified customer’s money — it generates ill will far out of proportion to the tiny amount of money involved.
• As a part of careers, we’re planning to roll out free, public CVs with user-selectable “vanity” URLs in a week or two. In retrospect, we should have done this from day one, as it compliments the public record of your Q&A on Stack Overflow. As Joel notes, the best way to control your online presence is to fill it yourself with all the cool stuff you’ve been doing! Don’t let others tell the story of you when you can tell it yourself.

Our favorite question this week is from Server Fault:

• Why is Internet access and Wi-Fi always so terrible at large tech conferences? Based on Joel’s recent DevDays experience, reliable WiFi at tech conferences seems to be rare. Why? How can this be fixed? What does it take?

If you’d like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have a dedicated phone number you can call to leave audio questions at 646-826-3879.

The transcript wiki for this episode is available for public editing.

 

 Podcast 70 [01:06:56m]: Play Now | Play in Popup | Download