Haskell positions at Facebook

Want to write Haskell for a living? At Facebook we’re looking for Spam Fighters. A large part of this job involves writing Haskell code to run on our Sigma/Haxl platform.

It’s a fascinating and exciting area to work in, using state of the art tools and systems, working with amazing people, and of course you get to write Haskell every day. Come and see what it’s like to write Haskell code that runs at Facebook scale!

Job description and application

Note: this is for the Menlo Park (California, USA) office.

Stack traces in GHCi, coming in GHC 8.0.1

tl;dr

In the upcoming GHC 8.0.1 release, if you start GHCi with ghci -fexternal-interpreter -prof (any packages you use must be built for profiling), then you get access to detailed stack traces for all the code you load into GHCi. Stack traces can be accessed via assert, error, Debug.Trace.traceStack, and the API in GHC.Stack.

Background

Haxl users at Facebook do a lot of development and testing inside GHCi. In fact, we’ve built a customized version of GHCi that runs code in our Haxl monad by default instead of the IO monad, and has a handful of extra commands to support common workflows needed by our developers.

Some of our codebase is pre-compiled, but the code being actively worked on is just loaded on the fly into GHCi during development and run with the interpreter. This works surprisingly well even for large codebases like ours, especially if you enable parallel compilation and use a bigger heap (e.g. ghci -j8 +RTS -A128m). This is a pretty smooth setup: right inside GHCi we can test the production code against real data, and interact with all of the services that our production systems talk to, while having a nice interactive edit/compile/test cycle.

However, one thing is missed by many developers, especially those coming from other languages: easy access to a stack trace when debugging. So, towards the end of last year, I set about finding a workable solution that we could deploy to our users without impacting their workflows.

Show me a stack trace!

To cut to the chase, in GHC 8.0.1 you can fire up ghci like this:

$ ghci -fexternal-interpreter -prof

and you have stack traces on, by default, for all the code you load into ghci. Let’s try an example.

import Control.Exception

myTail xs = assert (not (null xs)) $ tail xs

myMap :: (a -> b) -> [a] -> [b]
myMap f [] = []
myMap f (x:xs) = f x : myMap f xs

main = print (myMap myTail [[3],[]])

We have a map-alike function called myMap, and a tail-alike function called myTail. We want to find out if myTail is called with an empty list, so we added an assert. Ok, so it’s a contrived example, but let’s see what happens:

$ ghci -fexternal-interpreter -prof
GHCi, version 8.1.20160127: https://www.haskell.org/ghc/  :? for help
Prelude> :l ~/scratch/tailtest.hs 
[1 of 1] Compiling Main             ( /home/smarlow/scratch/tailtest.hs, interpreted )
Ok, modules loaded: Main.
*Main> main
[[],*** Exception: Assertion failed
CallStack (from ImplicitParams):
  assert, called at /home/smarlow//scratch/tailtest.hs:3:13 in main:Main
  myTail, called at /home/smarlow//scratch/tailtest.hs:9:21 in main:Main
CallStack (from -prof):
  Main.myTail (/home/smarlow/scratch/tailtest.hs:3:13-34)
  Main.myTail (/home/smarlow/scratch/tailtest.hs:3:13-44)
  Main.myMap (/home/smarlow/scratch/tailtest.hs:7:18-20)
  Main.myMap (/home/smarlow/scratch/tailtest.hs:7:18-33)
  Main.main (/home/smarlow/scratch/tailtest.hs:9:15-35)
  Main.main (/home/smarlow/scratch/tailtest.hs:9:8-36)
*Main> 

Now, we got two stack traces, both printed by assert. The first comes from ImplicitParams, which knows the location of the call site of assert because assert has a special ?callStack :: CallStack constraint in its type.

The second stack trace is the new one, generated by GHCi running in -prof mode, and has the full call stack all the way from main, including the fact that myTail was called by myMap. That is, it’s a dynamic call stack, not a lexical one.

Dumping the stack from anywhere

Using assert is one way to get access to a stack trace, but sometimes you just want to print out the stack when a particular condition is hit, or when a function is called, to see what’s going on. For this reason we have Debug.Trace.traceStack. This is like trace, but it also prints out the current stack trace. For example, I just picked a random place in the code of Happy, inserted a call to traceStack, loaded Happy into ghci -fexternal-interpreter -prof, ran it and got this:

closure1
CallStack (from -prof):
  LALR.closure1.addItems.fn (LALR.lhs:106:28-48)
  LALR.closure1.addItems.fn (LALR.lhs:(106,28)-(110,84))
  LALR.closure1.addItems.fn (LALR.lhs:(104,40)-(111,31))
  LALR.closure1.addItems.new_new_items (LALR.lhs:100:59-74)
  LALR.closure1.addItems.new_new_items (LALR.lhs:100:37-75)
  LALR.closure1.addItems.new_new_items (LALR.lhs:(99,33)-(101,53))
  GenUtils.mkClosure (GenUtils.lhs:28:28-36)
  GenUtils.mkClosure (GenUtils.lhs:28:20-36)
  LALR.closure1 (LALR.lhs:91:16-67)
  LALR.closure1 (LALR.lhs:91:11-68)
  LALR.genActionTable.possActions (LALR.lhs:489:44-64)
  LALR.genActionTable.possActions (LALR.lhs:(489,33)-(490,60))
  LALR.genActionTable.actionTable (LALR.lhs:471:34-53)
  LALR.genActionTable.actionTable (LALR.lhs:(469,26)-(471,54))
  LALR.genActionTable.actionTable (LALR.lhs:(468,23)-(472,61))
  Main.main2.runParserGen.action (Main.lhs:114:49-77)
  Main.main2.runParserGen.action (Main.lhs:114:27-78)
  Main.main2.runParserGen (Main.lhs:(96,9)-(276,9))
  Main.main2.runParserGen (Main.lhs:(90,9)-(276,10))
  Main.main2.runParserGen (Main.lhs:(86,9)-(276,10))
  Main.main2.runParserGen (Main.lhs:(85,9)-(276,10))
  Main.main2 (Main.lhs:74:20-43)
  Main.main2 (Main.lhs:(64,9)-(78,61))
  Main.main (Main.lhs:57:9-18)

You’ll notice that each function appears on the stack multiple times—this is because the the annotations are based on scopes, and GHC tries to insert annotations in useful-looking places. There might well be room for refinement here in the future.

Any drawbacks?

1. You have to compile your packages with profiling. Use --enable-library-profiling when running Cabal, or set library-profiling: True in your .cabal/config, or do the Stack equivalent.

2. Results with calls to error are mixed, because the error calls are often lifted to the top level as a CAF, which breaks the stack simulation that the profiler does. I have ideas for some workarounds for this that I plan to try in the future.

3. Interpreted code will run more slowly. But this is only for debugging—we didn’t change the source code, so everything still runs at full speed when compiled normally. You can also pre-compile some of your code; don’t forget to use -prof, and add -fprof-auto-calls to get stack-trace annotations for the code you compile. You can :set -fobject-code -fprof-auto-calls inside GHCi itself to use compiled code by default.

How does it work?

We’re using the existing stack-simulation that happens in GHC’s profiler, called “cost-centre stacks”. However, running in profiled mode wasn’t supported by the interpreter, and there were some serious shenanigans involved to make it possible to run profiled code in GHCi without slowing down GHCi itself.

There are various differences in the way the Haskell code runs in profiling mode. The layout of heap objects is different, because every heap object points to information about the call stack that created it. This is necessary to get accurate stack simulations in the presence of things like higher-order functions, but it’s also important for the heap profiler, so that it can tell who created each heap object. When running in profiling mode, we have to do various things to maintain the runtime’s simulation of the call stack.

The first step was to make the interpreter itself work in profiling mode (as in, interpret code correctly and not crash). Fortunately this wasn’t nearly as difficult as I’d anticipated: the interpreter and byte-code compiler were already nicely abstracted over the things that change in profiling mode. At this point we can already do things that weren’t possible before: profile GHCi itself, and profile Template Haskell.

Next, I had to make the interpreter actually simulate the call stack for interpreted code. Again, this was reasonably straightforward, and involved using the breakpoints that GHCi already inserts into the interpreted code as SCC annotations for the profiler.

So far so good: this actually worked quite nicely, but there was one huge drawback. To actually use it, we have to compile GHC itself with profiling. Which works, except that it slows down GHCi when compiling code by a factor of 2-3. That was too big a hit to deploy this as part of the standard workflow for our Haxl users at Facebook, so I needed to find a way to make it work without the overhead on the compiler.

Enter Remote GHCi

The solution is to separate the compiler from the interpreter, using a scheme that I’ve called Remote GHCi. The idea is that by putting the compiler and the interpreter in separate processes, the compiler can be running at full speed on a normal non-profiled runtime, while the interpreter is running in a separate process using the profiled runtime.

The main complication is arranging that all the interactions between the compiler and the interpreter happen via serialized messages over a pipe. We currently have about 50 different message types, you can see them all here. We’re currently using the binary library together with Generic instance generation, but serialization and deserialization using binary is definitely a bottleneck so I’m looking forward to moving to the new CBOR-based serialization library when it’s ready.

It turns out that making this separation has a number of advantages aside from stack traces, which are listed on the RemoteGHCi wiki page.

GHCJS has been doing something similar for a while to support Template Haskell. In fact, I used the GHCJS Template Haskell code as a starting point, integrated it with GHC proper and built it out to fully support GHCi (with a couple of exceptions, notably the debugger doesn’t currently work, and dynCompileExpr in the GHC API cannot be supported in general).

Remote GHCi also works for Template Haskell and Quasi Quotes, and has the advantage that when compiling TH code with -prof -fexternal-interpreter, you don’t need to first compile it without -prof, because we can run the -prof code directly in the external interpreter process.

Three kinds of stack trace in GHC 8.0.1

There’s a lot happening on the stack trace front. We now have no less than three ways to get a stack trace:

• Profiling: ghc -prof -fprof-auto and ghci -fexternal-interprter -prof
• ImplicitParams, with the magic ?callStack :: CallStack constraint (now called HasCallStack).
• DWARF: ghc -g

Each of these has advantages and disadvantages, and none of them are subsumed by any of the others (sadly!). I’ll try to summarise:

• Profiling

  • Detailed, dynamic, call stacks

  But:

  • Requires recompiling your code, or loading it into GHCi
  • 2-3x runtime overhead compiled, 20-40x interpreted
  • Not so great for error and undefined right now

• ImplicitParams

  • Good for finding the call site of particular functions, like error or undefined

  But:

  • Requires explicit code changes to propagate the stack
  • Some runtime overhead (stacks get constructed and passed around at runtime)
  • Shows up in types as HasCallStack constraints
  • Lexical, not dynamic. (In g = map f, g calls f rather than map calling f)

  Could you change GHC so that it automatically adds HasCallStack constraints everywhere and also hides them from the user, to get the effect of full call-stack coverage? Maybe - that would be an alternative to the scheme I’ve implemented on top of profiling. One difficult area is CAFs, though. If a constraint is added to a CAF, then the CAF is re-evaluated each time it is called, which is obviously undesirable. The profiler goes to some lengths to avoid changing the asymptotic cost of things, but trades off some information in the stack simulation in the process, which is why calls to error sometimes don’t get accurate call stack information.

• DWARF

  • No runtime overhead, can be deployed in production.
  • Good when you’re not willing to sacrifice any performance, but having some information is better than none when something goes wrong.

  But:

  • Gives the raw execution stack, so we lose information due to tail-calls and lazy evaluation.

Conclusion

We now have full stack traces inside GHCi, provided you compile your packages for profiling, and use ghci -fexternal-interpreter -prof.

Remote GHCi is not the default in GHC 8.0.1, but it’s available with the flag -fexternal-interpreter. Please try it out and let me know how you get on!

Fun With Haxl (Part 1)

This is a blog-post version of a talk I recently gave at the Haskell eXchange 2015. The video of the talk is here, but there were a lot of questions during the talk which aren’t very audible on the video, so hopefully this post will be useful to folks who weren’t at the event.

If you want to play with the examples yourself, the code is available on github, and to run the examples you’ll need to cabal install haxl sqlite first, or the stack equivalent.

What is Haxl?

Haxl is a library that was developed for solving a very specific problem at Facebook: we wanted to write purely functional code, including data-fetching operations, and have the data-fetches automatically batched and performed concurrently as far as possible. This is exactly what Haxl does, and it has been running in production at Facebook as part of the anti-abuse infrastructure for nearly a year now.

Although it was designed for this specific purpose, we can put Haxl to use for a wide range of tasks where implicit concurrency is needed: not just data-fetching, but other remote data operations (including writes), and it works perfectly well for batching and overlapping local I/O operations too. In this blog post (series) I’ll start by reflecting on how to use Haxl for what it was intended for, and then move on to give examples of some of the other things we can use Haxl for. In the final example, I’ll use Haxl to implement a parallel build system.

Example: accessing data for a blog

Let’s suppose you’re writing a blog (an old-fashioned one with dynamically-generated pages!) and you want to store the content and metadata for the blog in a database. I’ve made an example database called blog.sqlite, and we can poke around to see what’s in it:

$ sqlite3 blog.sqlite
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
postcontent  postinfo     postviews  
sqlite> .schema postinfo
CREATE TABLE postinfo(postid int, postdate timestamp, posttopic text);
sqlite> .schema postcontent
CREATE TABLE postcontent(postid int, content text);
sqlite> select * from postinfo;
1|2014-11-20 10:00:00|topic1
2|2014-11-20 10:01:00|topic2
3|2014-11-20 10:02:00|topic3
...
sqlite> select * from postcontent;
1|example content 1
2|example content 2
3|example content 3
...

There are a couple of tables that we’re interested in: postinfo, which contains the metadata, and postcontent, which contains the content. Both are indexed by postid, an integer key for each post.

Now, let’s make a little Haskell API for accessing the blog data. I’ll do this twice: first by calling an SQL library directly, and then using Haxl, to compare the two.

The code for the direct implementation is in BlogDB.hs, using the simple sqlite package for accessing the sqlite DB (there are other more elaborate and type-safe abstractions for accessing databases, but that is orthogonal to the issues we’re interested in here, so I’m using sqlite to keep things simple).

In our simple API, there’s a monad, Blog, in which we can access the blog data, a function run for executing a Blog computation, and two operations, getPostIds and getPostContent for making specific queries in the Blog monad. To summarise:

type Blog a  -- a monad

run :: Blog a -> IO a

type PostId = Int
type PostContent = String

getPostIds     :: Blog [PostId]
getPostContent :: PostId -> Blog PostContent

The implementation of the API will print out the queries it is making, so that we can see what’s happening when we call these functions. Let’s use this API to query our example DB:

GHCi, version 7.11.20150924: https://www.haskell.org/ghc/  :? for help
[1 of 1] Compiling BlogDB           ( BlogDB.hs, interpreted )
Ok, modules loaded: BlogDB.
*BlogDB> run getPostIds
select postid from postinfo;
[1,2,3,4,5,6,7,8,9,10,11,12]
*BlogDB> run $ getPostIds >>= mapM getPostContent
select postid from postinfo;
select content from postcontent where postid = 1;
select content from postcontent where postid = 2;
select content from postcontent where postid = 3;
select content from postcontent where postid = 4;
select content from postcontent where postid = 5;
select content from postcontent where postid = 6;
select content from postcontent where postid = 7;
select content from postcontent where postid = 8;
select content from postcontent where postid = 9;
select content from postcontent where postid = 10;
select content from postcontent where postid = 11;
select content from postcontent where postid = 12;
["example content 1","example content 2","example content 3","example content 4","example content 5","example content 6","example content 7","example content 8","example content 9","example content 10","example content 11","example content 12"]
*BlogDB> 

The problem: batching queries

Now, the issue with this API is that every call to getPostContent results in a separate select query. The mapM call in the above example gave rise to one select query to fetch the contents of each post separately.

Ideally, rather than

select content from postcontent where postid = 1;
select content from postcontent where postid = 2;
select content from postcontent where postid = 3;

What we would like to see is something like

select content from postcontent where postid in (1,2,3);

This kind of batching is particularly important when the database is remote, or large, or both.

One way to solve the problem is to add a new API for this query, e.g.:

multiGetPostContents :: [PostId] -> IO [PostContent]

But there are several problems with this:

• Clients have to remember to call it, rather than using mapM.

• If we’re fetching post content in multiple parts of our code, we would have to arrange to do the fetching in one place and plumb the results to the places that need the data, which might involve restructuring our code in an unnatural way, purely for efficiency reasons.

• From a taste perspective, multiGetPostContents duplicates the functionality of mapM getPostContent, which is ugly.

This is the problem that Haxl was designed to solve. We’ll look at how to implement this API on top of Haxl in the next couple of sections, but just to demonstrate the effect, let’s try it out first:

Prelude> :l HaxlBlog
[1 of 2] Compiling BlogDataSource   ( BlogDataSource.hs, interpreted )
[2 of 2] Compiling HaxlBlog         ( HaxlBlog.hs, interpreted )
Ok, modules loaded: HaxlBlog, BlogDataSource.
*HaxlBlog> run $ getPostIds >>= mapM getPostContent
select postid from postinfo;
select postid,content from postcontent where postid in (12,11,10,9,8,7,6,5,4,3,2,1)
["example content 1","example content 2","example content 3","example content 4","example content 5","example content 6","example content 7","example content 8","example content 9","example content 10","example content 11","example content 12"]
*HaxlBlog>

Even though we used the standard mapM function to perform multiple getPostContent calls, they were batched together and executed as a single select query.

Introduction to Haxl

You can find the full documentation for Haxl here, but in this section I’ll walk through the most important parts, and then we’ll implement our own data source for the blog database.

Haxl is a Monad:

newtype GenHaxl u a

instance Functor (GenHaxl u)
instance Applicative (GenHaxl u)
instance Monad (GenHaxl u)

It is generalised over a type variable u, which can be used to pass around some user-defined data throughout a Haxl computation. For example, in our application at Facebook we instantiate u with the data passed in with the request that we’re processing.

Essentially there is a Reader monad built-in to Haxl. (this might not be the cleanest design, but it is the way it is.) Throughout the following we’re not going to be using the u parameter, and I’ll often instantiate it with (), like this:

type Haxl a = GenHaxl () a

The most important operation in Haxl is dataFetch:

dataFetch :: (DataSource u r, Request r a) => r a -> GenHaxl u a

This is how a user of Haxl fetches some data from a data source (in our example, from the blog database). The Haxl library is designed so that you can use multiple user-defined data sources simultaneously.

The argument of type r a is a request, where r is the request type constructor, and a is the type of the result we’re expecting. The r type is defined by the data source you’re using, which should also supply appropriate instances of DataSource and Request. For example, the request type for our blog looks like this:

data BlogRequest a where
  FetchPosts       :: BlogRequest [PostId]
  FetchPostContent :: PostId -> BlogRequest PostContent

Note that we’re using a GADT, because we have two different requests which each produce a result of a different type.

Next, our request type needs to satisfy the Request constraint. Request is defined like this:

type Request req a =
  ( Eq (req a)
  , Hashable (req a)
  , Typeable (req a)
  , Show (req a)
  , Show a
  )

That is, it is a synonym for a handful of type class constraints that are all straightforward boilerplate. (defining constraint-synonyms like this requires the ConstraintKinds extension, and it’s a handy trick to know).

The other constraint we need to satisfy is DataSource, which is defined like this:

class (DataSourceName req, StateKey req, Show1 req)
       => DataSource u req where
  fetch
    :: State req
    -> Flags
    -> u
    -> [BlockedFetch req]
    -> PerformFetch

DataSource has a single method, fetch, which is used by Haxl to execute requests for this data source. The key point is that fetch is passed a list of BlockedFetch values, each of which contains a single request. The BlockedFetch type is defined like this:

data BlockedFetch r = forall a. BlockedFetch (r a) (ResultVar a)

That is, it contains a request of type r a, and a ResultVar a which is a container to store the result in. The fetch implementation can store the result using one of these two functions:

putSuccess :: ResultVar a -> a -> IO ()
putFailure :: (Exception e) => ResultVar a -> e -> IO ()

Because fetch is passed a list of BlockedFetch, it can collect together requests and satisfy them using a single query to the database, or perform them concurrently, or use whatever methods are available for performing multiple requests simultaneously.

The fetch method returns PerformFetch, which is defined like this:

data PerformFetch
  = SyncFetch  (IO ())
  | AsyncFetch (IO () -> IO ())

For our purposes here, we’ll only use SyncFetch, which should contain an IO action whose job it is to fill in all the results in the BlockedFetches before it returns. The alternative AsyncFetch can be used to overlap requests from multiple data sources.

Lastly, let’s talk about state. Most data sources will need some state; in the case of our blog database we need to keep track of the handle to the database so that we don’t have to open a fresh one each time we make some queries. In Haxl, data source state is represented using an associated data type called State, which is defined by the StateKey class:

class Typeable f => StateKey (f :: * -> *) where
  data State f

So every data source with request type req defines a state of type State req, which can of course be empty if the data source doesn’t need any state. Our blog data source defines it like this:

instance StateKey BlogRequest where
  data State BlogRequest = BlogDataState SQLiteHandle

The State req for a data source is passed to fetch each time it is called.

The full implementation of our example data source is in BlogDataSource.hs.

How do we run some Haxl?

There’s a runHaxl function:

runHaxl :: Env u -> GenHaxl u a -> IO a

Which needs something of type Env u. This is the “environment” that a Haxl computation runs in, and contains various things needed by the framework. It also contains the data source state, and to build an Env we need to supply the initial state. Here’s how to get an Env:

initEnv :: StateStore -> u -> IO (Env u)

The StateStore contains the states for all the data sources we’re using. It is constructed with these two functions:

stateEmpty :: StateStore
stateSet :: StateKey f => State f -> StateStore -> StateStore

To see how to put these together, take a look at HaxlBlog.hs.

Trying it out

We saw a small example of our Haxl data source working earlier, but just to round off this first part of the series and whet your appetite for the next part, here are a couple more examples.

Haxl batches things together when we use the Applicative operators:

*HaxlBlog> run $ (,) <$> getPostContent 1 <*> getPostContent 2
select postid,content from postcontent where postid in (2,1)
("example content 1","example content 2")

Even if we have multiple mapM calls, they get batched together:

*HaxlBlog> run $ (,) <$> mapM getPostContent [1..3] <*> mapM getPostContent [4..6]
select postid,content from postcontent where postid in (6,5,4,3,2,1)
(["example content 1","example content 2","example content 3"],["example content 4","example content 5","example content 6"])

In Part 2 we’ll talk more about batching, and introduce the upcoming ApplicativeDo extension which will allow Haxl to automatically parallelize sequential-looking do-expressions.