Why the HTML5 ‘Video’ Element Is Effectively Unusable, Even in Browsers Which Support It

Monday, 21 December 2009

[Update, 22 Dec 2009: In the original version of this article, I incorrectly reported that Firefox auto-buffers HTML5 video content by default, as Safari and Chrome do. It does not, as shown by Christopher Blizzard’s simple tests here. I regret the error, and have revised the article accordingly. For posterity, the source for the original version is preserved here. You can create a precise list of changes by doing a diff against the source for the current revision.]

I seldom post video to DF, but when I do, I refuse to embed Flash,¹ I want the markup to be sane and standard, I want the video to play in popular standards-compliant web browsers, and I don’t want the video to download/buffer automatically. Here’s an example from a year ago, using QuickTime.

What you see upon page load is a poster frame (a still image), then you (the user) click the poster frame to actually download and watch the video. Here’s the markup I used then:

<embed
   width="320" height="256"
   type="video/quicktime"
   pluginspage="https://www.apple.com/quicktime/download/"
   src="dtk-panic-1-poster.jpg"
   href="dtk-panic-1.mov"
   target="myself"
   controller="false"
   autoplay="false"
   scale="aspect"
/>

That markup met all of my aforementioned desires but for one: the <embed> tag is not standard. Worse, it now has a new significant problem: it doesn’t work at all in Chrome (at least on the Mac).

So I’ve been paying attention to the new <video> element in HTML5. In a nut, it attempts to make embedding a video in a web page just as easy markup-wise as embedding an image with the <img> tag. (Likewise for audio with the new <audio> element.)

The obvious downside to relying solely on the <video> element to embed web video is that because it’s new, the only browsers that support it are recent releases of Safari, Firefox, and Chrome. This isn’t one of those things that just doesn’t work in IE6 or IE7 — it doesn’t work in IE period. Therefore few sites are using HTML5 video in production now, and of those, nearly all are doing so with fallback markup, often of significant complexity, that presents the video using a Flash player for other browsers. Because (a) I don’t post much video; (b) the overwhelming majority of DF’s audience is in fact using an HTML5-compatible version of Safari, Firefox, or Chrome;² and (c) I’m willing to be a dick about this; I do not care about fallbacks for browsers that don’t support <video>.

What I’d like to do is just use <video>, with two source elements — MP4 and OGV — for all the cross-browser reasons specified in Mark Pilgrim’s fine chapter on video in his in-progress HTML5 book. (Short version: Safari and MobileSafari support only MP4, Firefox supports only OGV, Chrome supports both MP4 and OGV.)³

So I decided to try this last week with the screencast videos I created to illustrate my piece on PastryKit. I tried markup like this:

<video height="475"  width="407"  controls  poster="iphoneguide-mac.png">
   <source src="iphoneguide-mac.mp4" type="video/mp4" />
   <source src="iphoneguide-mac.ogv" type="video/ogg" />
</video>

The good news: The above markup results in video that plays in Safari, Chrome, and Firefox. It also works perfectly in MobileSafari on iPhone OS. Safari and Chrome play the MP4 video, Firefox plays the OGV. (Chrome supports both formats, and plays the one listed first. I want it to play the MP4 version because the video and audio are of noticeably superior quality.) That is the entirety of the necessary markup; if you’re unfamiliar with the sort of nasty markup typically used to embed video, try a little View Source on a few web pages that embed video.

The bad news: In two of the three browsers (Safari 4.0.4 and Chrome 4.0.249.43), with the above simple markup, the video content buffers automatically on page load. What I mean is that as soon as you load the web page, the browsers download the actual video files that are embedded. As stated at the outset, I don’t want that. Instead, on page load, I want the browser to render only the poster image, and load the video only after the user has clicked to initiate playback.

The HTML5 spec defines an autobuffer attribute for the video and other media elements (bold emphasis added):

The autobuffer attribute is a boolean attribute. Its presence hints to the user agent that the author believes that the media element will likely be used, even though the element does not have an autoplay attribute. (The attribute has no effect if used in conjunction with the autoplay attribute, though including both is not an error.) This attribute may be ignored altogether.

Firefox honors the autobuffer attribute. Omit the attribute from your markup, and video content will not auto-buffer in Firefox. Include it, and it will.

But alas, in my testing, Safari and Chrome take the spec up on the aforebolded offer to ignore this attribute. Even if you do not explicitly turn this attribute on, Safari and Chrome will still auto-buffer the content for your <video> (and <audio>) elements. There is no way to suppress this using HTML markup.

You might be thinking, “Hey, but default auto-buffering sounds like a good feature, because that way users won’t have to wait as long for the video to be ready to play.” I presume this sort of thinking is what led the Safari and Chrome teams to do this.

But this browser behavior is very much undesirable for both publishers and users in common contexts. Users loading the page over a slow connection, or a pay-by-the-megabyte metered connection (which is common with wireless networks), should not be forced to download a potentially large video every time they load the page. Likewise, publishers should not be forced to pay for the bandwidth to transmit videos that won’t be watched.

Think, in particular, of the nature of publishing embedded video on a weblog. I, the publisher, post an entry containing embedded video. That post may remain on my home page for a week. Regular readers may load the home page dozens of times during the period when the video appears on the page. With auto-buffering, they’re going to download the full video every time they load the page. Local caching may alleviate some of that, but for sites with high traffic and/or which frequently embed video, the difference is enormous.

This is why embedded video from YouTube, Vimeo, and all similar services works on a click-to-load basis. Auto-buffering is fine as an optional attribute, but for many (probably most) contexts, click-to-load is essential behavior.

But as far as I can see, there’s no way to get click-to-load video in Safari or Chrome using just a <video> element. The only workaround I could think of was to do something like this:

1. In the HTML markup, rather than a <video> element, instead use an <img> element with the intended poster frame.
2. Add an onclick JavaScript handler to the <img> element, which, when invoked, does some DOM jiggery-pokery to remove the just-clicked-upon <img> element and replace it with a <video> element that sources the intended video files.

And, in fact, that is exactly what I resorted to for my PastryKit videos. Do a View Source on that page to see the solution. There goes the nice clean just-as-easy-to-include-a-video-as-an-image markup. (My sincere thanks to Faruk Ateş and Paul Irish for helping with the jiggery-pokery implementation.)

This WebKit bug filed back in April indicates I’m not the first person to stumble on this shortcoming. That’s about <audio> instead of <video>, but the principle is exactly the same. And the example cited in this bug report seems like a perfect scenario where everyone should agree that media content should not buffer automatically: a podcast archive page with an <audio> element for every previous episode.

A big part of the appeal of the <video> and <audio> elements is that they should be easier to use. As it stands today, though, these elements are unusable in popular contexts without resorting to JavaScript DOM manipulation to effectively turn auto-buffering off.

I think the HTML5 spec should be changed such that the value of the autobuffer attribute must be respected. And even if the spec is not changed, web browsers should not choose to ignore it. Web browsers should only buffer HTML5 media content when the autobuffer or autoplay attribute has been explicitly turned on in the markup. ★

More on PastryKit

Thursday, 17 December 2009

A bit more regarding PastryKit, Apple’s web framework for creating iPhone web apps with a look and feel very close to that of native iPhone apps.

Apple’s Various JavaScript Frameworks

Jonathan “Wolf” Rentzsch posted this terrific overview of the known JavaScript frameworks in use by various teams at Apple today. Some are public. Others, like PastryKit, are not. There’s a bit of overlap between them; in particular, several of them are very much patterned after Cocoa, not just in terms of MVC, but right down to details like class names. (I hadn’t heard of Gianduia before, but it seems very deep. Apple’s online retail team is using it for things like the Genius Bar reservation system.)

But PastryKit is the only one that’s specifically targeted at building native-style iPhone user interfaces. Of the others, one does seem closely related to PastryKit: TuneKit, the now-public framework for building iTunes LP and iTunes Extras content bundles. About 40 percent of the code in PastryKit and TuneKit is identical (if you discount the difference between the “PK” and “TK” class name prefixes). DF reader Greg Sadetsky emailed with the following comparison, after running the code through a prettifier to de-minimize it. First, this from PastryKit:

const PKTransitionDefaults = {
    duration: 0.5, delay: 0,
    removesTargetUponCompletion: false,
    revertsToOriginalValues: false
};
const PKTransitionStyles = [
    "-webkit-transition-property", 
    "-webkit-transition-duration",
    "-webkit-transition-timing-function",
    "-webkit-transition-delay",
    "-webkit-transition"
];

And then this identical-but-for-the-PK/TK-prefixes bit from TunesKit:

const TKTransitionDefaults = {
    duration: 0.5, delay: 0,
    removesTargetUponCompletion: false,
    revertsToOriginalValues: false
};
const TKTransitionStyles = [
    "-webkit-transition-property", 
    "-webkit-transition-duration",
    "-webkit-transition-timing-function",
    "-webkit-transition-delay",
    "-webkit-transition"
];

Et cetera. Either PastryKit and TuneKit are sibling projects, or they’re the work of the same person. (Also, Daniel Dilger found references to “PastryKit” in the Wall-E iTunes Extras bundle back in September.)

iDisk Help

A few readers emailed to point out another instance of PastryKit in action: the help pages for the iDisk app for iPhone OS. If you have the iDisk app from the App Store installed, you can see this by tapping “Settings”, then “iDisk Help”. Looks just like the rest of the app, but there are a few tell-tale signs that it’s PastryKit running in a full-screen web view (e.g. tapping on the status bar doesn’t scroll to the top).

Is PastryKit a Leftover From the Pre-SDK Days?

One common question I’ve seen (for one example, in this Hacker News thread) is whether PastryKit could be a leftover from the “web apps are a really sweet solution to creating iPhone software” days. I’d say no. For one thing, there’s always been a user guide, and as I recall, the guide has always sort of basically followed the look-and-feel of a native app, but this current version for iPhone OS 3 is much better. In fact, if you set your user agent in Safari to MobileSafari 2 (rather than 3) and then load the User Guide you get an older version from iPhone OS 2.0, and it is far less impressive technically. No scrolling magic, no iPhone-style blue gradients when highlighting selected list rows, etc. Further, PastryKit takes advantage of features in mobile WebKit that were only added in iPhone OS 3.0. PastryKit is new.

Apple’s Take on Web Apps Competing With the App Store

Lastly, there’s the question of how concerned Apple is, strategically, that a robust web app API and market would take away from the App Store. And if so, are they worried about the money? I’d guess probably not. I don’t think Apple’s 30 percent cut of App Store revenue is anything to sneeze at, and it’s growing fast. But there’s no question that the App Store exists to sell iPhones and iPod Touches, not the other way around.

A bigger concern might be the fact that web apps are cross-platform. Apple is currently using user-agent detection to prevent its PastryKit user guide apps from loading on anything other than MobileSafari, but I don’t think there’s any technical reason why these apps wouldn’t work just fine, and look almost the same, on both Android and WebOS. The font would be slightly different, since neither of those ship with Helvetica, but otherwise it’d look just like an iPhone app.

Cutting edge web development never seems to “just work” across different browsers, but these three — iPhone, Android, WebOS — are all based on the same WebKit rendering core. And even if per-platform tweaks are required, surely it’d be far less work than to develop separate native apps using the fundamentally different Cocoa Touch, Android, and Mojo APIs.

I would hope Apple isn’t concerned about this. For one thing, desktop-targeted web apps haven’t dampened demand for Macs. If anything, the Mac has thrived as the web has replaced Windows as the most widespread software platform. I think Apple would do well to encourage web apps for cross-platform mobile software, and that doing so would not slow interest in native Cocoa Touch development for performance (especially games) and access to hardware (like the camera) not available to web apps. I also think it would benefit both iPhone users and Apple itself to have no-approval-necessary web apps thrive as an alternative to the tightly-controlled App Store.¹ It’s human nature to emphasize conflict and drama, but there’s no reason why native iPhone apps and iPhone-optimized web apps can’t both be thriving, growing platforms.

But, well, we don’t know. ★

PastryKit

Tuesday, 15 December 2009

One of the recent themes here has been the topic of iPhone OS web apps — apps for the iPhone and iPod Touch that are built with nothing more than HTML, CSS, and JavaScript. And specifically, the issues of how good an experience iPhone web apps can offer, and how easy they are to develop, compared to native Cocoa Touch apps.

One area in particular where iPhone web apps fall short of native iPhone apps is scrolling. Take for example a long list, such as your full address book in Contacts, or all your songs in the iPod app. When you scroll these lists, you can fling the list, and the list will scroll at high speed after you let go. The effect is sort of like spinning a wheel with very little friction. With iPhone web apps, you can make a list that looks almost, maybe even exactly, like a native iPhone list view. But all web views on the iPhone scroll with almost no momentum. You can’t fling them. iPhone web views feel like they have a lot of scrolling friction.

This friction might make sense for regular web pages rendered on the iPhone’s small screen, where by “regular” I mean “not optimized specifically for display on the iPhone”. But it just feels slow — stuck — on iPhone-optimized apps.

If you’ve never taken notice of it, try it now, comparing something like your Contacts app list with an iPhone web app like Showtime. (In Showtime, tap “Watchlist” and then tap the “+” button to get a nice long list to scroll through.) Another good scrolling comparison: native iPhone Twitter clients like Tweetie and Birdfeed versus iPhone web app Twitter clients like Hahlo and the new Mobile Twitter. The difference is significant, and far more than cosmetic. As Justin Williams recently wrote:

I believe that with the current crop of Web technologies available in MobileSafari, apps like Hahlo, PocketTweets and Showtime could thrive as an alternative to their native counterparts if Apple allowed developers to adjust the scrolling/drag coefficient of Mobile WebKit. If you compare the scrolling speed of your Twitter timeline in Hahlo and Tweetie, the results are drastically different. Tweetie feels like it effortlessly scrolls based on how much momentum you exert in the scroll action, while Hahlo is being constrained by a fifty pound weight on its back.

Scrolling isn’t the only UI/experience issue where web apps seemingly can’t quite match what native iPhone apps can offer. Another is that MobileSafari doesn’t allow for CSS fixed-position elements, which means you can’t make a toolbar that stays put at the very top or bottom of the screen without having it scroll away when you scroll the content.

And that’s just talking about the user experience side of things. The other side is that of development. Last month I wrote:

When you write a Cocoa Touch app for the iPhone, you’re not starting from scratch. You’re starting with the Cocoa Touch framework. As Faruk Ateş astutely points out in his response to Koch, to discount the framework is to discount everything that sets the iPhone apart as a development platform. Not only are native iPhone apps faster and more capable than their web-app equivalents, but they’re easier to write.

Some readers objected to this, arguing, more or less, that no matter how good the Cocoa Touch framework is, native iPhone apps are harder to develop than web apps because one must learn both the app framework (Cocoa) and a new programming language, Objective-C. But that’s not really a fair comparison. It’s like saying it’s easier to run than to bike if you don’t know how to ride a bicycle.

Let’s clarify. Let’s specify that we’re talking about creating iPhone apps with great design and user experiences. Let’s acknowledge that to make anything great — anything at all, software or otherwise — one needs talent, experience, and familiarity and expertise with the necessary tools. If you’re already an expert web developer but have never programmed Cocoa software, then yes, there’s large time-consuming Step Zero in front of you before you can attempt to develop a native iPhone Cocoa Touch app. But, likewise, there are long-time Cocoa Mac (and a handful of even longer-time NeXT) developers who have no idea how to create a modern AJAX-y web app from HTML, CSS, and JavaScript — and who think JavaScript is the weird programming language.

What I’m saying is that talented Cocoa Touch developers have an easier job implementing the same iPhone app than talented web app developers do. The Cocoa Touch framework makes all sort of things free or easy. Things like smooth fast animation between views. Things like buttons and lists and toolbars that look just like other standard iPhone buttons, lists, and toolbars.

There do exist some open source frameworks for iPhone web app developers, so that one need not start from scratch implementing things to mimic Cocoa Touch UI elements. iUI, started by Joe Hewitt just a few weeks after the original iPhone debuted in July 2007, is one. jQTouch, by David Kaneda and based on jQuery, is another. (Showtime is built using jQTouch.)

But these frameworks don’t solve the problem with scrolling speed/friction, or fixed-position elements.

It ends up there is a company, however, that has developed an amazing iPhone web app framework which:

• Completely hides the address bar, even when running not from a saved-to-the-home app icon, but within a page in MobileSafari itself.

• Allows for fixed-position toolbars that never budge from the top when you scroll.

• And: sets its own scrolling friction coefficient, allowing you to fling long lists.

The company behind this web framework is Apple. And the framework is apparently named PastryKit.

The Best iPhone Web App in the World

If you have an iPhone or iPod Touch handy, stop reading this and go here:

https://help.apple.com/iphone/3/mobile/

This is not a secret web site. In fact, it may well already be in your iPhone bookmarks — it’s where you get redirected when you choose the “iPhone User Guide” bookmark that’s included as one of the defaults for MobileSafari. I don’t know when Apple launched this PastryKit-powered version of the site, but it’s been under our noses for a while, with very little notice.¹

If you don’t have an iPhone or iPod Touch handy, or if you do and you’re back after trying it out and want to poke at it with Safari’s Web Inspector, you can also load it in Mac or Windows Safari by setting the user agent string to MobileSafari 3.x in Safari’s Develop menu. (Without the MobileSafari user agent string, you’ll get redirected to a different iPhone help page; using a MobileSafari 2.x user agent string, you’ll see last year’s version of the User Guide, which is far less impressive technically.) Shrink your Safari window down to roughly iPhone dimensions before loading the site, because the UI will be laid out to fill the dimensions of the viewport when it loads.

The JavaScript source code has been minimized/optimized, but it’s not obfuscated, per se, so it’s easy to see that the framework is called PastryKit, and even a dilettante JavaScript hacker like me can follow along and see some of what’s going on. PastryKit accomplishes all three of the aforementioned things — hiding the MobileSafari address bar, providing fixed-position toolbars, and providing scrolling with momentum — by disabling regular scrolling and setting up its own view hierarchy and implementing its own scrolling.

From WebKit’s perspective, everything in this iPhone User Guide is in a view that is exactly the size of the viewport, so there’s nothing to scroll. PastryKit handles all of what the user sees as scrollable content. This is how on the iPhone it provides for lower-friction scrolling than provided by MobileWebKit itself — PastryKit does its own scrolling math. And it’s even more noticeable when running the app in desktop Safari with the user agent trick, because you never see Mac scrollbars, and can’t use your mouse scrollwheel to scroll the content. Everything related to scrolling is implemented within the app itself, in JavaScript.

After installing the User Guide app to your home screen and launching it from there, there’s really very little to suggest that it isn’t a native iPhone application. No MobileSafari address bar at the top, no MobileSafari toolbar at the bottom. Scrolling is fast and has momentum. It even works perfectly offline, because the contents of the user guide are stored locally in a database using HTML5.

It’s not perfect. Scrolling is smooth on my iPhone 3GS, but it’s a little janky on my old iPhone 3G and original iPhone. (Still better than the scrolling in any other web app I’ve ever seen, though — just not native-app-smooth.) Taps and gestures sometimes don’t register — this is most noticeable for me when I try to scroll as fast as I can with a quick finger swipe. Super-quick gestures, I suspect, sometimes slip through the JavaScript event filters.

The rubber-band “bounce” scrolling — where if the view is already at the top but you pull down in an attempt to scroll up and you see whitespace and it all just bounces back into place when you let go — breaks if you pull down all the way off the bottom of the display. What happens there is the view gets “stuck” in the position where it’s displaying the stretched-out whitespace at the top of the view; you can unstick it by just tapping somewhere in the content area.

And, lastly, it doesn’t work at all with the iPhone’s system-standard gesture where you can tap the status bar at the top of the screen (the bar that displays the carrier name, signal strength, time, and battery life) to scroll to the top. In MobileSafari, if you tap the status bar while running this app, the entire MobileSafari view scrolls to the top, which exposes the browser address bar. When running the User Guide app from the home screen, tapping the status bar simply has no effect. I presume the problem here is that there is no JavaScript event in mobile WebKit for status bar taps — the event goes to the iPhone app, and the web app running in WebKit inside the iPhone app can’t see it or register a handler to act upon it. From a user’s perspective, an iPhone web app launched from the home screen is just another app. But technically, it’s like a meta app — it’s a JavaScript app running inside a native iPhone app that just presents a full-screen WebKit view and loads the web app.

But on the whole this User Guide app and the PastryKit framework are rather amazing. The $64,000 question, though, is whether PastryKit is something Apple intends (or that a team within Apple hopes) to ship publicly. It seems like a lot of effort to build a framework this rich just for this iPhone User Guide, so I’m hopeful the answer is yes. Perhaps something integrated with the next major release of Dashcode? And, perhaps with integrated UI layout tools, along the lines of Interface Builder?

Here’s to hoping we haven’t heard the last of PastryKit, and that Apple continues work on making mobile WebKit an open alternative to the App Store.

PastryKit / iPhone User Guide Demos

For the sake of posterity and for those of you without access to an iPhone or Safari, I’ve made two screencasts showing the iPhone User Guide web app in action. ★