| CARVIEW |
Select Language
HTTP/2 302
server: nginx
date: Mon, 22 Dec 2025 08:33:16 GMT
content-type: text/plain; charset=utf-8
content-length: 0
x-archive-redirect-reason: found capture at 20100211203533
location: https://web.archive.org/web/20100211203533/https://code.google.com/p/redis/wiki/FAQ
server-timing: captures_list;dur=0.621199, exclusion.robots;dur=0.057845, exclusion.robots.policy;dur=0.047704, esindex;dur=0.008430, cdx.remote;dur=19.910463, LoadShardBlock;dur=192.484339, PetaboxLoader3.datanode;dur=85.555458, PetaboxLoader3.resolve;dur=35.266323
x-app-server: wwwb-app242-dc8
x-ts: 302
x-tr: 335
server-timing: TR;dur=0,Tw;dur=0,Tc;dur=0
set-cookie: wb-p-SERVER=wwwb-app242; path=/
x-location: All
x-as: 14061
x-rl: 0
x-na: 0
x-page-cache: MISS
server-timing: MISS
x-nid: DigitalOcean
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()
HTTP/2 200
server: nginx
date: Mon, 22 Dec 2025 08:33:16 GMT
content-type: text/html; charset=UTF-8
x-archive-orig-date: Thu, 11 Feb 2010 20:35:32 GMT
x-archive-orig-pragma: no-cache
x-archive-orig-expires: Fri, 01 Jan 1990 00:00:00 GMT
x-archive-orig-cache-control: no-cache, must-revalidate
x-archive-orig-x-content-type-options: nosniff
x-archive-orig-server: codesite
x-archive-orig-x-xss-protection: 0
x-archive-guessed-content-type: text/html
x-archive-guessed-charset: utf-8
memento-datetime: Thu, 11 Feb 2010 20:35:33 GMT
link: ; rel="original", ; rel="timemap"; type="application/link-format", ; rel="timegate"
content-security-policy: default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org web-static.archive.org wayback-api.archive.org athena.archive.org analytics.archive.org pragma.archivelab.org wwwb-events.archive.org
x-archive-src: 52_14_20100211182945_crawl102-c/52_14_20100211203517_crawl103.arc.gz
server-timing: captures_list;dur=0.529713, exclusion.robots;dur=0.023962, exclusion.robots.policy;dur=0.014492, esindex;dur=0.008708, cdx.remote;dur=9.592865, LoadShardBlock;dur=121.614162, PetaboxLoader3.datanode;dur=158.279807, PetaboxLoader3.resolve;dur=142.727543, load_resource;dur=241.599798
x-app-server: wwwb-app242-dc8
x-ts: 200
x-tr: 432
server-timing: TR;dur=0,Tw;dur=0,Tc;dur=0
x-location: All
x-as: 14061
x-rl: 0
x-na: 0
x-page-cache: MISS
server-timing: MISS
x-nid: DigitalOcean
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()
content-encoding: gzip
FAQ -
redis -
Project Hosting on Google Code
|
|
|
Search
|
FAQ
Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?Memcachedb is basically memcached done persistent. Redis is a different evolution path in the key-value DBs, the idea is that the main advantages of key-value DBs are retained even without a so severe loss of comfort of plain key-value DBs. So Redis offers more features:
Isn't this key-value thing just hype?I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent. When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?). Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve. Can I backup a Redis DB while the server is working?Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the cp unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well. What's the Redis memory footprint?Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly. With large keys/values the ratio is much better of course. 64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required. I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?Short answer: If you are using a Redis client that supports consistent hashing you can distribute the dataset across different nodes. For instance the Ruby clients supports this feature. There are plans to develop redis-cluster that basically is a dummy Redis server that is only used in order to distribute the requests among N different nodes using consistent hashing. Why Redis takes the whole dataset in RAM?Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM. If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load from multiple clients with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower. Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory. Amazon EC2 now provides instances with 32 or 64 GB of RAM. If my dataset is too big for RAM and I don't want to use consistent hashing or other ways to distribute the dataset across different nodes, what I can do to use Redis anyway?You may try to load a dataset larger than your memory in Redis and see what happens, basically if you are using a modern Operating System, and you have a lot of data in the DB that is rarely accessed, the OS's virtual memory implementation will try to swap rarely used pages of memory on the disk, to only recall this pages when they are needed. If you have many large values rarely used this will work. If your DB is big because you have tons of little values accessed at random without a specific pattern this will not work (at low level a page is usually 4096 bytes, and you can have different keys/values stored at a single page. The OS can't swap this page on disk if there are even few keys used frequently). Another possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs. Update: it could be interesting to test how Redis performs with datasets larger than memory if the OS swap partition is in one of this very fast Intel SSD disks. Do you plan to implement Virtual Memory in Redis? Why don't just let the Operating System handle it for you?Yes, in order to support datasets bigger than RAM there is the plan to implement transparent Virtual Memory in Redis, that is, the ability to transfer large values associated to keys rarely used on Disk, and reload them transparently in memory when this values are requested in some way. So you may ask why don't let the operating system VM do the work for us. There are two main reasons: in Redis even a large value stored at a given key, for instance a 1 million elements list, is not allocated in a contiguous piece of memory. It's actually very fragmented since Redis uses quite aggressive object sharing and allocated Redis Objects structures reuse. So you can imagine the memory layout composed of 4096 bytes pages that actually contain different parts of different large values. Not only, but a lot of values that are large enough for us to swap out to disk, like a 1024k value, is just one quarter the size of a memory page, and likely in the same page there are other values that are not rarely used. So this value wil never be swapped out by the operating system. This is the first reason for implementing application-level virtual memory in Redis. There is another one, as important as the first. A complex object in memory like a list or a set is something 10 times bigger than the same object serialized on disk. Probably you already noticed how Redis snapshots on disk are damn smaller compared to the memory usage of Redis for the same objects. This happens because when data is in memory is full of pointers, reference counters and other metadata. Add to this malloc fragmentation and need to return word-aligned chunks of memory and you have a clear picture of what happens. So this means to have 10 times the I/O between memory and disk than otherwise needed. I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!This may happen and it's prefectly ok. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused. What happens if Redis runs out of memory?With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM. The INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions. You can also use the "maxmemory" option in the config file to put a limit to the memory Redis can use. If this limit is reached Redis will start to reply with an error to write commands (but will continue to accept read-only commands). How much time it takes to load a big database at server startup?Just an example on normal hardware: It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID. This can give you some kind of feeling about the order of magnitude of the time needed to load data when you restart the server. Background saving is failing with a fork() error under Linux even if I've a lot of free RAM!Short answer: echo 1 > /proc/sys/vm/overcommit_memory :) And now the long one: Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail. Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis. Are Redis on disk snapshots atomic?Yes, redis background saving process is always fork(2)ed when the server is outside of the execution of a command, so every command reported to be atomic in RAM is also atomic from the point of view of the disk snapshot. Redis is single threaded, how can I exploit multiple CPU / cores?Simply start multiple instances of Redis in different ports in the same box and threat them as different servers! Given that Redis is a distributed database anyway in order to scale you need to think in terms of multiple computational units. At some point a single box may not be enough anyway. In general key-value databases are very scalable because of the property that different keys can stay on different servers independently. In Redis there are client libraries such Redis-rb (the Ruby client) that are able to handle multiple servers automatically using consistent hashing. We are going to implement consistent hashing in all the other major client libraries. If you use a different language you can implement it yourself otherwise just hash the key before to SET / GET it from a given server. For example imagine to have N Redis servers, server-0, server-1, ..., server-N. You want to store the key "foo", what's the right server where to put "foo" in order to distribute keys evenly among different servers? Just perform the crc = CRC32("foo"), then servernum = crc % N (the rest of the division for N). This will give a number between 0 and N-1 for every key. Connect to this server and store the key. The same for gets. This is a basic way of performing key partitioning, consistent hashing is much better and this is why after Redis 1.0 will be released we'll try to implement this in every widely used client library starting from Python and PHP (Ruby already implements this support). I'm using some form of key hashing for partitioning, but what about SORT BY?With SORT BY you need that all the weight keys are in the same Redis instance of the list/set you are trying to sort. In order to make this possible we developed a concept called key tags. A key tag is a special pattern inside a key that, if preset, is the only part of the key hashed in order to select the server for this key. For example in order to hash the key "foo" I simply perform the CRC32 checksum of the whole string, but if this key has a pattern in the form of the characters {...} I only hash this substring. So for example for the key "foo{bared}" the key hashing code will simply perform the CRC32 of "bared". This way using key tags you can ensure that related keys will be stored on the same Redis instance just using the same key tag for all this keys. Redis-rb already implements key tags. What is the maximum number of keys a single Redis instance can hold? and what the max number of elements in a List, Set, Ordered Set?In theory Redis can handle up to 2^32 keys, and was tested in practice to handle at least 150 million of keys per instance. We are working in order to experiment with larger values. Every list, set, and ordered set, can hold 2^32 elements. Actually Redis internals are ready to allow up to 264 elements but the current disk dump format don't support this, and there is a lot time to fix this issues in the future as currently even with 128 GB of RAM it's impossible to reach 232 elements. What Redis means actually?Redis means two things:
Why did you started the Redis project?In order to scale LLOOGG. But after I got the basic server working I liked the idea to share the work with other guys, and Redis was turned into an open source project. |
||
► Sign in to add a comment
©2009 Google -
Code Home -
Terms of Service -
Privacy Policy -
Site Directory -
Project Hosting Help
Hosted by
Hello World = 11 bytes 1 million = 10.5 megabytes ????
lgr888999: there is the key too. Adding everything you get something like 18MB if stored as a big single string without separators IIRC. This is worst case scenario, add to this: pointers, length of the strings, hash table memory, objects reference counting (that makes we able to do zero-copy GETs and replication) and so on. With bigger strings it was observed that the ration is something like 1/2. This is not a Redis-specific problem, to store things in ram in a way that's fast to take them back costs a lot of space.
memory footprint of memcached would almost be the same for the given example
@ligachef thanks interesting information.
Perhaps the scenario when the memory is full could be clarified a bit? Will it swap mem or refuse new entries/updates?
Hello cjbystrom, when Redis runs out of memory it exists. It's just like a system crash. In the real world basically malloc() will hardly return NULL on a modern operating system like Linux, so you'll know you are out of memory when the system starts to perform poorly due to swapping. When this happens it's time to upgrade the hardware or to use a partitioning schema in order to REDIStribute the load across different servers.
p.s. it is absolutely possible to add a configuration parameter that tells Redis to stop accepting new queries if Redis is using more than N gigabytes of memory, but this is hardly useful.
Cheers, Salvatore
Note: I'm writing a new FAQ entry about this and other issues related to why Redis holds the dataset in memory.
Thanks for the reply. I really admire your work, keep it up :)
Thanks cgbystrom! Actually thanks to your question I'm adding support for Redis awareness of how much memory is using just now, so the new INFO command that will report a number of statistics will provide this very useful information in Kbytes. We'll see later what to do when we are low on memory, but at least now it will be possible to write simple scripts that will check the memory of all the Redis boxes and possibly signal that something is odd.
The EXPIRE command, that's also a work in progress, is another issue about being low on memory. Probably Redis gets an out of memory condition it should try to flush expired entries ASAP and retry to allocated.
I'm curious what is the nature of the replication between redis servers? Can it be synchronous, or not?
@spiderbill: yes it's syncronous, every change done in the master is replicated as soon as possible (it depends on the master <-> slave link speed) in the slave, exactly in the same order.
Hi, nice project! I know that the theory of memcached is that authentication and authorization are beyond the scope of the project, I wonder if that is necessarily true for redis as well?
I hope not (I strongly believe that relying on obscurity or virtual networking for security in production systems is daft) but either way there should be something in the FAQ about security and how to keep others from writing to or silently changing your database.
Hi, this is nice... The redis server loads the data into memory when it is started... right? But how long will it take to start the redis server if I have say for ex. 10 GB of data?
In the spirit of REDIStributing data, amazon's dynamo system seems like it would make a great complement to redis. https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
That paper describes their scheme for redistributing what appears to be a very simple vanilly key-value store across a cluster of servers. They have a hashing scheme using the idea of a ring keyspace (as it looks like Ezra has just done for redis) but add the ability to automatically move servers in and out of a cluster and redistribute the keys accordingly, and to build in redundancy by making keys map to a configurable number of servers.
I don't see any reason this couldn't be done for redis. I know some open source projects based on dynamo (dynomite and voldemort) exist, and it'd be awesome to add the use of higher-level data structures into the mix, as redis does. I wonder if it would be easy to combine redis with dynomite and/or voldemort?
It seems like the necessary primitives (like redis' MOVE operation) are there already.
Could you please go through an example when the node fails? How do you recover? What gets lost? What happens with replication on a failure?
@poomalairaj.r
it takes about 45 seconds to restore a 2 GB database on my system (fairly standard, no RAID).
I guess the problem with current redis-rb hashring is that you loose the beauty and potential of SORT. Lists, sets and simple key-value should work fine across the cluster, but since SORT can be fed with BY and GET, it would not work, if referenced objects are on separate nodes.
So in this aspect clustering would require hard-coding into Redis itself. With dynamic cluster resizing (adding/removing nodes on-the-fly) it would simply rock. My humble advice is to go in this direction, since clustering + redundant nodes is what many project need, especially in the cloud environment.
About SORTing and Redis-db hashring, actually we figured out a way to make this working most of the time.
If the key contains {....} pattern, like foo{bared} then only "bared" is hashed. So basically all the keys containing {myuserid} are guaranteed to be on the same server. So if you need sorting only for "local" things, like stuff associated to a specific user, what you can do is to make sure all the keys related to a given sorting are going to be stored on the same server.
Also to use a single Redis server for things that require to be sorted and use a cluster for "raw data" is a workable approach.
I would like to store data, indexed with an 512 bit key (SHA512). From a quick glance it seems (because of the text protocol) redis can't store binary keys, and doing a conversion of the SHA512 checksum into base64 would introduce a major storage room hit.
So my questions are: - will redis support arbitrary binary keys/values? - will/is it be possible to use plain numbers as keys, which are not stored as strings, but arbitrarily big numbers? That would be great for storing data according to their checksum.
A slightly related question: for memcached one can set a maximum memory size, and when memcached hits it, it starts to evict entries from memory (I think in an LRU-like fashion). Are the developers interested in anything similar for redis? Doing this housekeeping on the client side (when not needing strong persistency for the objects) is just a real big pain.
Oh, I see in the protocol docs that the response for eg. the GET operation does specify the result's size. I think a similar extension (which can even be compatible with the current one) could be done, so SET would look: SET $keylength key $valuelength value eg: SET $10 binaryblob $34 anotherbinaryblobwitharbitrarybits instead of the current SET key value
I want to implement a service that will export data in redis to mysql. The reason is that I want to use Redis because of if speed, but I also want to be able to get back with mysql (just in case, :D). So, I think I will write a service that read dump.rdb or appendonly.log file, then update data in MySQL. So can you tell me that is there already a project like that or how can I read and understand the dump.rdb (or appendonly.log) file. Thanks a lot.