Hi @robertmeta. Thanks for the input! Would you mind reviewing the thread on issue #49 regarding concurrency levels and perhaps add to that conversation? It's my opinion that higher concurrency levels beyond what we have provided here would be useful if we were ready to benchmark high-connection low-utilization Websockets. But presently we are testing high-traffic traditional HTTP where responding to requests as quickly as possible is the paramount objective.

As with anything though, I'm prepared to be proven wrong. :)

I'm with bhauer, this isn't, "how many users can we serve per server on our chat service".

Some things I would like to see in the future:

• msgpack tests. Msgpack is rapidly becoming an alternative to JSON particularly with non-browser clients.
• multirow reads/writes, perhaps computing a mathematical function from a table or updating a hundred rows. Almost all of the requests I serve are multirow reads or writes, and frameworks have some per-row overhead usually
• Making an "onward" request to another server. This tests the outbound HTTP stack.
• A relationship (join) test, where you are using the ORM to relate two or more entities in a parent/child configuration. Frameworks take different approaches for eager vs lazy loading; the results may be interesting. Maybe construct a loop of entities and then check it for cycles.

Hi @drewcrawford. Thanks for the ideas!

No rush, since this is just long-term planning, but I am curious about your second idea concerning multi-row reads and writes. In my head I conceive of that as executing a single UPDATE but I am probably misunderstanding you. Would you be able to draft up some quick pseudo-code to allow me to visualize what you mean?

Your third idea was echoed by another reader, so that's got "high demand" from my perspective. :)

A test of relationships is a great idea too.

A commenter on HN named Terretta suggested the following. I'm just copying this here for easy future reference.

1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D.
2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example. (The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.)
3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc.
4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered.
5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache.
6. For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.

Another HN commenter named kbenson suggested a goal of defining a simple blog-style application. This is ambitious but we've already passed the threshold at which we require community contributions in order to move forward (even adding one more simple test will require several pull requests from the community to see that test implemented in more than a small sampling of the frameworks).

With that in mind, I think it's a great item to have on the long-term plan. If we keep the requirements simple, it could be done.

Would you be able to draft up some quick pseudo-code to allow me to visualize what you mean?

a = 0
b = 1
for i from 1 to 100
    insert into table values(a+b)
    c = b
    b = a + b
    a = c

or

i = 0
not_quite_sum = 0
for row in table
    if i is even:
        not_quite_sum += row.field
    else:
        not_quite_sum -= row.field

The key insight being

• there's a for loop
• each pass of the for loop operates on one row
• the overall operation is simple, but not so simple that it's natural to do in a SQL one-liner

The interesting thing about this test is that it does reads/writes in the same connection. Whereas in the single row access case the dominating factor might be setting up the connection or acquiring it from a shared pool, here the test is about how quick the ORM bindings are once they're in place and how fast you can move memory between the DB process and the application process.

@drewcrawford, thanks! I understand what you have in mind now. For some reason, I read your original statement to imply something much more complicated.

But you simply mean for a test that operates over multiple rows in a single resultset, in the case of reading, with per-row functionality occurring within the application rather than within SQL functions. Your pseudo-code illustrates the idea well.

Note that requirements for each test type are now posted at the results web site: https://www.techempower.com/benchmarks/#section=code

I just edited this issue to indicate the updates test is "present," and to add quick notes about the need to implement plaintext tests (both small and large payloads) and a larger work-load JSON test (something involving a complex and large data structure).

A test that exercises form rendering, validation, and CSRF protection could be interesting. I'm pretty sure most of the full stack frameworks have utilities for those. Maybe the test would have three parts? One server-side implementation, but three sets of wrk parameters: (a) GET the form, (b) POST the form with errors, (c) POST the form successfully.

I've added test type 16, a more complex routing table/map/tree based on Christopher Lord's comment on the Google Group:

https://groups.google.com/d/msg/framework-benchmarks/r0B3tPaCMPs/_PG1_p1McbwJ

Heavy model test:

• Make 10 ActiveRecord or RowGateway classes. Instantiate them from each table.
• Make additional 100 classes with single methods. Instantiate and call them while request.

This test may reveal cost of class loading [1], method calling and GC.

[1] Some languages like php loads class for each request.

I think a pretty simple additional test would be serving static context of different sizes (100k, 1m, 10m, 100m?). Frameworks that perform similar to (maybe even better than) Apache httpd in this domain could make life a lot easier for full-blown web applications than those that serve small content extremely well but degrade significantly when content sizes get large. With our framework PLAIN, for instance, we generate dynamic content of 3D data (JT, 3DXML, CATIA) that quickly reaches sizes >100m. Streaming those to files and then serve them with an httpd or the like would be a big drawback in terms of performance and complexity.

I agree that HTTPS between microservices, or between the load balancer and web frontend is a good idea in general. Benchmarking this would bring visibility to the performance overhead of frameworks in this area (an aim of the test would be to be directly comparable to a http test so the https overhead could be measured for each framework).

In production" many environments have TLS between layers and in service to service calls. Infact some industries (especially heavily regulated ones" require secure communication between these layers...

I think is Redis a great use case example. Communication is performed securely using a secure tunnel (such as using Spiped, Ghostunnel, etc'.

The separation of concerns between security and services is important. It allows security patches to be easily implemented across the board without patching each framework / service separately. Also, the security implementation is often superior to a BYO solution.

What's the best reverse proxy setup to use for frontend server Y and framework X, what do they support: localhost vs named pipes vs unix sockets vs shared memory etc.

I would love have benchmarks about TLS/SSL, but I'm not sure the benchmarks should include any frameworks. Once we use a framework's internal TLS/SSL in a benchmark, we're telling developers that this is a production grade setup.

I'm not even sure I want to know how many applications in the wild use security implementations with known vulnerabilities... it a source for worry for me on SO whenever someone asks about implementing TLS/SSL in a backend application server.

Than again, I'll probably implement TLS/SSL and let security be damned - the pressure to add TLS/SSL is too great and (re)explaining why security concerns should be separated from the framework is getting tiring.

Right an article on a site dedicated to edge services (F5) ... Also most of your "edge" services will use the exact same underlying lib. Eg ngnix using openssl. And if we take openssl it's often part of the os.

So the big issue is ... How did they implement it? Did they do a lot of copying and allocations? Are there added bottle necks?

As someone who works in a very secure environment we are required to have a high security edge layer (no one said I was ditching that and we actually have many layers there) but we are also required to have ssl/TLS "Inside" that security ring. To assume a security layer and you are done in banking, health, government, education is dangerous and in some cases illegal

Also most of your "edge" services will use the exact same underlying lib. Eg ngnix using openssl. And if we take openssl it's often part of the os.

And... if memory serves, openssl uses insecure defaults... but maybe that changed.

As someone who works in a very secure environment...

In your experience, do you use a framework's SSL/TLS implementation for production (to secure internal network communication), or do you tunnel the communication through secure channeling (i.e., spiped)?

I agree that if secure production environments rely on the framework's SSL/TLS layer, than it's important to both test and benchmark that option.

websockets +1

Commenters above who've thoughts about TLS/SSL tests please see and weigh in on the stand-alone issue (#3290).

Many microservices are running in 'tiny' containers. It would be interesting to see how the stacks behave in such an environment. For example: allocated 1 CPU and 500MB RAM.

For some time I've been running some of the TechEmpower benchmarks and trying to make sense of the results. One thought that keeps coming back is that these benchmarks are stressing the frameworks at 100% cpu. That should not be the normal operating mode of a server. I'd be interested to see latency vs throughput numbers. wrk2 can be used for such benchmarks.

HTTP2 +1

It seams the only performance you measure is response time.
Can you also measure

• CPU
• Energy
• Memory

as it was done as described in this article : https://jaxenter.com/energy-efficient-programming-languages-137264.html

Github Repo : https://github.com/greensoftwarelab/Energy-Languages

This only issue about greensoftwarelab tests is they used the Debian Benchmarks Game which only apply to few use case far away of the web use.

I am also interested in the performance of HTTP/3. Though there are few libraries supporting it now and HTTP/2 is still on the waiting list, adding this test type could give the library contributors a motive and speed up its support, especially in the more popular web frameworks. There is a WRK variant supporting HTTP/3 now so the tool should be ready.

Building on "13. Future: JSON responses with larger workloads (complex data structure serialization).", I'd like to propose a variant using Protobuf, as complex data structure serialization is where Protobuf shines.

I think that's better to first fix the actual test without fake apps.
Later we can think with more tests.