CARVIEW |
Popular topics:

Squeaky Clean Ajax and Comet with Lift
Focus on application development, not the plumbing
Lift is a web framework for Scala, and is probably best known for having great Comet and Ajax support.
I’ve been touring the features of Lift that I find appealing. Initially I looked at designer-friendly templates and REST services. Recently, I highlighted the great features for organising and controlling access to content.
Let’s now take a look at the Ajax and Comet features of Lift.
Ajax
When I think of Ajax, I think of interacting with a web server, but avoiding page reloads.
An example: suppose we had a site that allowed you to write poems, and a feature on that site might be a button you could click to get an associated word from a thesaurus. The thesaurus we have is large, we don’t want it loaded in the browser, so we need to call the server to use it.
The HTML template would be a field and a button:
<div data-lift="Associate">
<input id="word" type="text" placeholder="Type a word">
<input type="submit" name="submit">
</div>
So far, that’s probably similar across many web frameworks. What you might now expect is for us to define a REST endpoint, do some jQuery wiring to connect the service to the button.
In Lift, we can do that, but often the Ajax goodness comes via this kind of Scala code:
class Associate {
def render =
"@submit [onclick]" #> ajaxCall(
ValById("word"),
w => SetValById("word", Thesaurus.association(w))
)
}
This Associate
snippet is binding the submit button’s click event to a Lift Ajax call. That is, the left side of the replace function (#>
) is a CSS selector targeting the “onclick”, and the right side is arranging for the Ajax call to hit the server.
The first parameter to ajaxCall
is the value passed from the browser to the server. We’ve asked for the value of the input field with an ID of “word”. The second parameter is the function to run when the button is pressed. Here we’re taking the word we’ve been sent, and updating the field with whatever our Thesaurus.association
function gives us back.
Read more…

Community Leadership Summit tracks the forces that spread ideas
Inclusivity and recruitment among the themes at the unconference
American businesses, along with many others around the world, hustle to find enough programmers and computing staff. The gap widens precariously between the number of job openings and the number students graduating with the necessary skills. And yet, at the same time, we seem to drown in an overabundance of software packages. If you want JavaScript frameworks, large-scale data stores, bulk system configuration tools, mesh networking protocols, or even a new functional language, you have almost too many choices. So the shortage of programmers does not apply when people offer their code to their colleagues across the globe.
Open source software’s ever-expanding options show a programming culture that is proud of its accomplishments and eager to explore untapped potential. But above all, they reveal a desire to work together on problems. The existence of open source packages shows the strength of community.
The Community Leadership Summit (CLS), which started four years ago at the Open Source convention and has been held on the weekend preceding it ever since, drew more attendees than ever this year (I estimate some 200 at the opening plenary), and a large fraction of them actually work as community managers. The field has matured in other ways as well. At early conferences, people expressed aspirations and complained of problems that this year are finding solutions.

Rotating a UIView in 3D
OSCON 2013 Speaker Series
Jon Manning (@desplesda) and Paris Buttfield-Addison (@parisba) are co-founders of Secret Lab and authors of the forthcoming Learning Cocoa with Objective-C, 3rd Edition. They’ll be speaking at OSCON this week in Portland, OR. Here they explain how to rotate a UIView in 3D on iOS.
One of the simplest visual tricks you can do in iOS is to make a part of your UI pretend to be a 3D object. We’ve found that this is an excellent way to add some life and visual interest to both apps and games.
Below, you’ll learn how to make a view rotate in 3D and have a perspective effect.
First, your project needs to use the QuartzCore
framework.
Next, when you want the animation to begin, you do this:
CABasicAnimation* animation = [CABasicAnimation animation WithKeyPath:@"transform.rotation.y"]; animation.fromValue = @(0); animation.toValue = @(2 * M_PI); animation.repeatCount = INFINITY; animation.duration = 5.0; [self.rotatingView.layer addAnimation:animation forKey:@"rotation"]; CATransform3D transform = CATransform3DIdentity; transform.m34 = 1.0 / 500.0; self.rotatingView.layer.transform = transform;
To stop the animation, you do this:
[self.rotatingView.layer removeAnimationForKey:@"rotation"];
How It Works
CABasicAnimation
allows you to animate a property of a view from one value to another. In the case of rotating a view, the property that we want to animate is its rotation, and the values we want to animate from and to are angles.
When you create the animation using [CABasicAnimation animationWithKeyPath:
], you specify the property you want to animate. In this case, the one we want is the rotation around the Y axis.
CABasicAnimation* animation = [CABasicAnimation
animationWithKeyPath:@"transform.rotation.y"];
The animation is then configured. In this example, we made the rotation start from zero, and proceed through to a full circle. In Core Animation, angles are measured in radians, and there are 2 * π radians
in a full circle. So, the from value and to value are set thusly:
animation.fromValue = @(0); animation.toValue = @(2 * M_PI);
Next, the animation is told that it should repeat an infinite number of times, and that the full rotation should take 5 seconds.
animation.repeatCount = INFINITY; animation.duration = 5.0;
The animation is started by adding the animation to the view’s layer, using the addAnimation:forKey:
method. This method takes two parameters: the animation object that you want to use, and a key
(or name) that the animation should be referred by.
Don’t be confused by the similarity between the “key
” that you use when you add the animation, and the “key path
” you use when creating the animation. The former is just a name you give the animation, and can be anything; the “key path” describes exactly what the animation modifies.
[self.rotatingView.layer addAnimation:animation forKey:@"rotation"];
The last step to this is to give the rotating view a little perspective. If you run the code while omitting the last few lines, you’ll end up with a view that appears to horizontally squash and stretch. What you want is for the edge that’s approaching the user’s eye to appear to get bigger, while the edge that’s moving away from the user’s eye to get smaller.
This is done by modifying the view’s 3D transform. By default, all views have a transform matrix applied to them that makes them all lie flat over each other. When you want something to have perspective, though, this doesn’t apply, and we need to override it.
CATransform3D transform = CATransform3DIdentity; transform.m34 = 1.0 / 500.0; self.rotatingView.layer.transform = transform;
The key to this part of the code is the second line: the one where the m34
field of the transform is updated. This part of the transform controls the sharpness of the perspective. (It’s basically how much the z
coordinate gets scaled towards or away from the vanishing point as it moves closer to or further from the “camera
“.)
As you can see, adding 3D and perspective effects isn’t terribly difficult, but the results can provide an excellent payoff in terms of user immersion.
Securing User Content in the JavaScriptable Web
OSCON 2013 Speaker Series
Recent work by a W3 Working Group plans to expose many powerful cryptographic operations for web applications. Although the planned API adds much needed functionality to JavaScript, it doesn’t address the JavaScript runtime’s terrible security properties. For instance, any script running in the web application has the power to hijack the web app’s content and UX. Just last February a mistake in Facebook’s “like” button brought down millions of web sites. Further, if you are an online service provider wanting to support higher privacy use cases like encrypted chat or web-based PGP email, the trust model is fundamentally broken since you can’t serve cryptographic JavaScript code without making the server a potential attack point.
The security challenges faced by JavaScript are mitigated by Privly, also labeled “the Web Privacy Stack,” by addressing two issues: (1) scoping code to the data, and (2) pre-distributing the code to the clients when possible.
Scoping Code to the Data
Every website has its own navigation structure, layout, and audience, but when you strip away these unique attributes of websites, you are left with data– chats, emails, photos– that can be treated uniformly across all websites. Operations on these data like encryption and signing, can be performed with indifference to their context and their contents.
Privly uses data indifference to create the notion of “Injectable Applications,” which are full web applications that are injected into the context of other web application. Since these applications are scoped to data and not layout, their properties are simplified and usable across the web.
Privly works within browser extensions by scanning web pages for specially formatted hyperlinks. When the extension detects the hyperlink, it “injects” the link into an iframe that is served locally from the browser extension.(footnote 1: This is currently how the Chrome extension performs, but different platforms are at different levels of sophistication. This approach can be universally applied to all platforms and does not necessarily require locally-served JavaScript code. However, without serving the JavaScript code locally, the security properties of the system are lost) Since the injected application is a full web application, the app could potentially support any web-implementable feature, including APIs between the host page and the injected application.
In short, if you scope an application to the data, then the cryptography can be viewed in potentially untrusted contexts.
Pre-Distributing Client Code
Privly creates an ecosystem of apps with known properties because it allows us to reason about security uniformly across the web. However, security is only as strong as the weakest attack point, which is why great care must be taken to appropriately distribute these applications. By packaging a set of applications for integration into browser extensions and mobile apps, the code is not re-loaded from a remote source every time the browser loads a new page.
Requiring the pre-distribution of applications is not normally compatible with the free and open internet. However, pains must be taken to realize the differences between the limited use cases of injected applications, and the general cases supported by web applications in general. Distributing every website to the client before visiting a website is impossible, but distributing a set of general injectable applications is lightweight and perhaps the only way to achieve security within the modern JavaScript runtime.
Requiring users to install an extension before they can view content is likely an impediment for any security system looking to gain users. However, since Privly uses hyperlinks to reference the content, it provides opportunity for a hosted fallback application. Depending on the nature of the injectable application, clicking the hyperlink could either present the same application as normally delivered by the extension, or present a prompt to install the appropriate browser extension.
Read more…

How to Engage (and Delight) Your App Users
OSCON 2013 Speaker Series: The importance of being whimsical
What makes a good app? Sure, it should do what it claims to do, intuitively and efficiently. But that is just the base line. How do you stand out from the sea of apps? You need to go above and beyond, and whimsy may just be the secret ingredient you need.
Tumblr
At the keynote at AnDevCon Boston, Chris Haseman and Zack Sultan showed us how they designed and implemented the Tumblr app. They shared many tips and tricks to make your app look good and work well, but the recurring theme is to delight your user.
There are many whimsical touches to Tumblr, one being the pull-to-refresh animation:
Tumblr could have gone with a loading text or a standard spinny, but this custom animation brings the app a notch above others. It entices users to refresh more, increasing engagement. This is the power of whimsy.
Welcome animations
Animation is a great way to add whimsy to your app. At AltWWDC, Ben Johnson showcased many different apps with effective animations.
Read more…

Get Hadoop, Hive, and HBase Up and Running in Less Than 15 Minutes
OSCON 2013 Speaker Series
If you have delved into Apache Hadoop and related projects, you know that installing and configuring Hadoop is hard. Often, a minor mistake during installation or configuration with messy tarballs will lurk for a long time until some otherwise innocuous change to the system or workload causes difficulties. Moreover, there is little to no integration testing among different projects (e.g. Hadoop, Hive, HBase, Zookeeper, etc.) in the ecosystem. Apache Bigtop is an open source project aimed at bridging exactly those gaps by:
1. Making it easier for users to deploy and configure Hadoop and related projects on their bare metal or virtualized clusters.
2. Performing integration testing among various components in the Hadoop ecosystem.
More about Apache Bigtop
The primary goal of Apache Bigtop is to build a community around the packaging and interoperability testing of Hadoop related projects. This includes testing at various levels (packaging, platform, runtime, upgrade, etc.) developed by a community with a focus on the system as a whole, rather than individual projects.
The latest released version of Apache Bigtop is Bigtop 0.5 which integrates the latest versions of various projects including Hadoop, Hive, HBase, Flume, Sqoop, Oozie and many more! The supported platforms include CentOS/RHEL 5 and 6, Fedora 16 and 17, SuSE Linux Enterprise 11, OpenSuSE 12.2, Ubuntu LTS Lucid and Precise, and Ubuntu Quantal.
Who uses Bigtop?
Folks who use Bigtop can be divided into two major categories. The first category of users are those who leverage Bigtop to power their own Hadoop Distributions. The second category of users are those who use Bigtop for deployment purposes.
In alphabetical order, they are:
Read more…

Zero Downtime Application Updates with Ansible
OSCON 2013 Speaker Series
Automating the configuration management of your operating systems and the rollout of your applications is one of the most important things an administrator or developer can do to avoid surprises when updating services, scaling up, or recovering from failures. However, it’s often not enough. Some of the most common operations that happen in your datacenter (or cloud environment) involve large numbers of machines working together and humans to mediate those processes. While we have been able to remove a lot of human effort from configuration, there has been a lack of software able to handle these higher-level operations.
I used to work for a hosted web application company where the IT process for executing an application update involved locking six people in a room for sometimes 3-4 hours, each person pressing the right buttons at the right time. This process almost always had a glitch somewhere where someone forgot to run the right command or something wasn’t well tested beforehand. While some technical solutions were applied to handle configuration automation, nothing that could perform configuration could really accomplish that high level choreography on top as well. This is why I wrote Ansible.
Ansible is a configuration management, application deployment, and IT orchestration system. One of Ansible’s strong points is having a very simple, human readable language – it allows users very fine, precise control over what happens on what machines at what times.
Getting started
To get started, create an inventory file, for instance, ~/ansible_hosts that defines what machines you are managing, and which machines are frequently organized into groups. Ansible can also pull inventory from multiple cloud sources, but an inventory file is a quick way to get started:
[webservers] www01.example.com www02.example.com # add more webservers here [monitoring] nagios1.example.com [lbservers] haproxy1.example.com haproxy2.example.com
Now that you have defined what machines you are managing, you have to define what you are going to do on the remote machines.
Ansible calls this description of processes a “playbook,” and you don’t have to have just one, you could have different playbooks for different kinds of tasks.
Let’s look at an example for describing a rolling update process. This example is somewhat involved because it’s using haproxy, but haproxy is freely available. Ansible also includes modules for dealing with Netscalers and F5 load balancers, so this is just an example — ordinarily you would start more simply and work up to an example like this:
Read more…

A Birds-eye View with Lift
Organize and control content with the Lift web framework
Lift is a web framework built for the Scala programming language, running on the Java Virtual Machine. Version 2.5 recently shipped, and I’m highlighting features of the framework that I find appealing.
Last time it was transforms and REST services, and this time it’s two features around organizing and controlling access to content.
SiteMap
How do you manage the HTML pages on site? Maybe some pages are available to everyone, others require login or some condition for access, and often pages are grouped in some way. Lift has an optional mechanism for expressing all this in one place, and it’s called SiteMap. This is how it looks in code:
lazy val home = Menu.i("home") / "index"
lazy val poets = Menu.i("poets") / "list" submenus (
Menu("Larkin") / "a-z" / "larkin",
Menu("Tennyson") / "a-z" / "tennyson"
)
LiftRules.setSiteMap(SiteMap(home, poets))
We’ve defined a site made up of a home page, and pages for poets we like. Why is this a good thing?
First, it’s a pretty readable representation of the site in (compiler checked) Scala code. You can probably figure out that we have a page called “home” found at /index.html (or / as it’s usually known).
What might not be obvious is that Menu.i
is using “home” as a key into your translation resources, meaning the link text will be the localized version—if you’re building an internationalized site. The link to the page can be generated by a Menu.builder
, a snippet you can use on pages to give your users navigation of the SiteMap.
The second menu item is called “poets,” served up from a template called list.html. On the home page, the navigation snippet would give you a link to the “poets” page, but by default is smart enough to hide the submenus until you visit the list of poets.
SiteMap also performs access control: if you use SiteMap, HTML pages have to be listed in it to be accessed. It’s a simple line of defense, but you can improve on this by adding location parameters. These are rules for controlling access to a page.
Let’s say we wanted to only offer some content to customers paying for our premium plan:
val PremiumCustomersOnly = If(
() => false,
() => S.redirectTo("https://shop.example.org/") )
This is an if-then-else construct, made up of two functions. The first function is a test for what qualifies as a premium customer. It’s an arbitrary computation, and in this example it’s false
, meaning nobody is a premium customer (in reality, we’d perhaps check a property of the logged user). If you nevertheless try to get to the page, the second function kicks in, and you’d be sent to our store front.
We can apply the “premium customer” rule to any part of the site, but we’re going to just restrict Tennyson poems:
Menu("Tennyson") / "a-z" / "tennyson" >> PremiumCustomersOnly
If you don’t like the >>
method name, use rule
instead. If you just don’t like this domain specific language (DSL), you can construct a SiteMap in longer-form method calls.
What’s great about SiteMap is the flexibility it gives you: if you can’t get the behavior you want out of the box, you can write custom functions and classes, and drop them into the SiteMap. If you want menus displayed in a completely different way, not handled by Menu.builder
customization, you can write a different navigation snippet. Location parameter rules allow you to isolate logic, and combine them as needed to parts of your site.

Enterprise Data Workflows with Cascading
OSCON 2013 Speaker Series
Paco Nathan (@pacoid) is Director of Data Science at Concurrent, O’Reilly Author, and OSCON 2013 Speaker. In this interview we talk about creating enterprise data workflow with Cascading. Be sure to check out Paco’s book on the subject here
NOTE: If you are interested in attending OSCON to check out Paco’s talk or the many other cool sessions, click over to the OSCON website where you can use the discount code OS13PROG to get 20% your registration fee.
Key highlights include:
- Cascading is an abstraction layer on top of Hadoop [Discussed at 0:23]
- Define your business logic at a high level [Discussed at 1:21]
- Is Cascading good for enterprise? [Discussed at 2:31]
- Test-driven development at scale [Discussed at 3:35]
- Cascalog and the City of Palo Alto Open Data portal [Discussed at 7:39]
You can view the full interview here:

Augmenting Unstructured Data
OSCON 2013 Speaker Series
Our world is filled with unstructured data. By some estimates, it’s as high as 80% of all data.
Unstructured data is data that isn’t in a specific format. It isn’t separated by a delimiter that you could split on and get all of the individual pieces of data. Most often, this data comes directly from humans. Human generated data isn’t the best kind to place in a relational database and run queries on. You’d need to run some algorithms on unstructured data to gain any insight.
Play-by-Play
Advanced NFL Stats was kind enough to open up their play-by-play data for NFL games from 2002 to 2012. This dataset consisted of both structured and unstructured data. Here is a sample line showing a single play in the dataset:
20121119_CHI@SF,3,17,48,SF,CHI,3,2,76,20,0,(2:48) C.Kaepernick pass short right to M.Crabtree to SF 25 for 1 yard (C.Tillman). Caught at SF 25. 0-yds YAC,0,3,0,27,7 ,2012
This line is comma separated and has various queryable elements. For example, we could query on the teams playing and the scores. We couldn’t query on the unstructured portion, specifically:
(2:48) C.Kaepernick pass short right to M.Crabtree to SF 25 for 1 yard (C.Tillman). Caught at SF 25. 0-yds YAC
There are many more insights that can be gleaned from this portion. For example, we can see that San Francisco’s quarterback, Colin Kaepernick, passed the ball to the wide receiver, Michael Crabtree. He was tackled by Charles Tillman. Michael Crabtree caught the ball at the 25 yard line, gained 1 yard. After catching the ball, he didn’t make any forward progress or yards after catch (“0-yds YAC
“).
Writing a synopsis like that is easy for me as a human. I’ve spent my life working with language and trying to comprehend the meaning. It’s not as easy for a computer. We have to use various methods to extract the information from unstructured data like this. These can vary dramatically in complexity; some use simple string lookups or contains and others use natural language processing.
Parsing would be easy if all of the plays looked like the one above, but they don’t. Each play has a little bit different formatting and varies in the amount of data. The general format is different for each type of play: run, pass, punt, etc. Each type of play needs to be treated a little differently. Also, each person writing the play description will be a little different from the others.
Augmenting Data
As I showed in the synopsis of the play, one can get some interesting insight out of the structured and unstructured data. However, we’re still limited in what we can do and the kinds of queries we can write.
I had a conversation with a Canadian about American Football and the effects of the weather. I was talking about how the NFL now favors domed locations to take weather out as a variable for the Superbowl or NFL championship. With the play-by-play dataset, I couldn’t write a query to tell me the effects of weather one way or another. The data simply doesn’t exist in the dataset.
The physical location and date of the game is captured in the play-by-play. We can see that in the first portion “20121119_CHI@SF
“. Here, Chicago is playing at San Francisco on 2012-11-19. There are other datasets that could enable us to augment our play-by-play. These datasets would give us the physical location of the stadium where the game was played (the team’s name doesn’t mean that their stadium is in that city). The dataset for stadiums is a relatively small one. This dataset also shows if the stadium is domed and the ambient temperature is a non-issue.
Read more…
Get the Programming Weekly Newsletter
Stay informed. Receive weekly insight from industry insiders.
Ebook Deals of the Day
Video Deal of the Week
Get the O'Reilly
Programming Newsletter
Weekly insight from industry insiders.
Plus exclusive content and offers.