| CARVIEW |
Select Language
HTTP/2 200
date: Thu, 31 Jul 2025 09:22:00 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
etag: W/"2cb35c84097019671fca7afce40af6a4"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com wss://alive-staging.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=gMneT9GVnpV4PedgA3nKIuL6FcaAdO7lEqxfOnXBAkpnh0pVVnFWUbdnNwq%2FlVpfmswotiNrdlfnun6jFLbwP8DSsES7bpakQBgy1c%2FLII7PliOnnluXrAqvkViZQmifVx9EV99DE8CgWTH0Xqv7pOwgQGfVUf%2BDFEUQomkJ9IfhcQ0oVeJEgODeecobBDyCDz1ySyKXHrYmYvfN2Pn6xto13LOkVhQviEt6uTAaPZqoc7I4QSrxBar91BlDDBsEoRddCev3c0MeehpiJ78Z3Q%3D%3D--ovF7i81fMtiGQP7V--fOQdB2iLjqhk6ppu%2FBrEzw%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.71019626.1753953720; Path=/; Domain=github.com; Expires=Fri, 31 Jul 2026 09:22:00 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Fri, 31 Jul 2026 09:22:00 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: BE80:3FE9C3:679144:7D4DC2:688B35B8
Optimizations on the Producer Graph · twitter/summingbird Wiki · GitHub
Skip to content
Navigation Menu
{{ message }}
This repository was archived by the owner on Jan 20, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 267
Optimizations on the Producer Graph
Ian O'Connell edited this page Feb 27, 2014
·
4 revisions
- optionMap pushed up to sources.
- flatMap/optionMap composition
- map-side caching and combining in flatMappers immediately before sumByKey
- batching of results from flatMaps prior to sumByKey to send Map[K, V] rather than many individual (K, V) pairs (this reduces thread contention in storm).
- flatMapKeys followed by sumByKey causes a partial sum of the values before the flatMap, when the keys should have better locality (scalding only right now).
- Diamond collapse: (a.flatMap(fn1) ++ a.flatMap(fn2)) could be collapsed into a single flatMap (similarly with optionMap).
- flatMapKeys partial sum in online mode.
- leftJoin -> mapValues -> sumByKey could be pushed to the summer so that keys are colocated before hitting the service. In scalding, this is almost certainly a win, online when local caches are used, it may give better locality.
- push flatMap/leftJoin up to sources
- online, if a store is a converted store, go ahead and do the key Injection on the flatMappers before the sumByKey (i.e. we are often converting an object to bytes before putting into memcache, why not go ahead and serialize once, and do the cheap bytes-only serialization between the flatMappers and reducers.
- Offline: Frozen keys, we have some code/PR's on it, but needs to be finished up tested.
- Online: Full batching between Spouts, Flatmaps and Summers in online mode
- Online: Collapse final flat map's into spouts when we have enough parallelism
- Online: Online tuning to auto move operations in/out of a spout
- Online: See we have a CAS store and collapse the whole tree into the spouts/sources
You can’t perform that action at this time.