Redis: Reducing Memory Usage

High Level Tips for Redis

Most of Stream-Framework's users start out with Redis and eventually move to Cassandra because Redis becomes too expensive to host. Stream, our hosted API for building scalable newsfeeds & activity streams is also based on Cassandra.

There are quite a few things you can do to reduce Redis' memory usage though. Here are my favourite 6 high level tips for reducing Redis memory usage. For lower level tips, see the Redis docs on memory optimization.

Tip 1 - Serialization

It is often possible to reduce your Redis memory usage by using the right serialization. Serialize objects using built-in serialization mechanism is very tempting (eg. Python's pickle, PHP's serialize...) but it will likely waste lot of memory for nothing. You should stay away from it and make sure you only serialize what you need. Using something like JSON or Protobuf is often a good idea.

Tip 2 - Compression

If the data you're storing is large enough and contains a lot of text you can often reduce memory usage by adding compression. LZO or Snappy are good options for this use case.

Tip 3 - Fallback to disk based storage (aka your rdbms)

Many times you don't need to store all the data in Redis. If your model allows it, try to store only hot or recent data and hit the database when a request for missing data is performed.

If you go for this approach be sure to include an endmarker in your data. Assuming there are 3 items in your database, Redis would store the 3 items and the end marker. This way you know you don't have to run a database query after item number 3. The end marker approach vastly reduces how frequently you fall back to the database.

Tip 4 - Normalize your storage

Memory is expensive and Redis only uses memory. Take this into account when designing your data structure. A normalized approach is often the better choice compared to a denormalized option.

Tip 5 - Expire and trim the data

If possible, set expirations on everything. This presents data which is rarely used from sticking around in your cluster. It can be a few days, a few weeks, but definitely set some sort of expiration on your data. If you're inserting into lists or sets be sure to trim them occasionally.

Tip 6 - Use the right eviction policy

If the amount of data that you store in Redis grows over time and you can't afford to keep it all in memory, you probably want to configure Redis as a LRU cache storage. Redis allows for 6 different eviction policies, make sure you pick the right one!

Most of these tips are obvious, but before you switch away from Redis, be sure to give them a try.