Redis: Reducing Memory Usage
High Level Tips for Redis
Most of Stream-Framework's users start out with Redis and eventually move to Cassandra because Redis becomes too expensive to host. Stream, our hosted API for building scalable newsfeeds & activity streams is also based on Cassandra.
There are quite a few things you can do to reduce Redis' memory usage though. Here are my favourite 6 high level tips for reducing Redis memory usage. For lower level tips, see the Redis docs on memory optimization.
Tip 1 - Serialization
It is often possible to reduce your Redis memory usage by using the right serialization. Serialize objects using built-in serialization mechanism is very tempting (eg. Python's pickle, PHP's serialize...) but it will likely waste lot of memory for nothing. You should stay away from it and make sure you only serialize what you need. Using something like JSON or Protobuf is often a good idea.
Tip 2 - Compression
If the data you're storing is large enough and contains a lot of text you can often reduce memory usage by adding compression. LZO or Snappy are good options for this use case.
Tip 3 - Fallback to disk based storage (aka your rdbms)
Many times you don't need to store all the data in Redis. If your model allows it, try to store only hot or recent data and hit the database when a request for missing data is performed.
If you go for this approach be sure to include an endmarker in your data. Assuming there are 3 items in your database, Redis would store the 3 items and the end marker. This way you know you don't have to run a database query after item number 3. The end marker approach vastly reduces how frequently you fall back to the database.
Tip 4 - Normalize your storage
Memory is expensive and Redis only uses memory. Take this into account when designing your data structure. A normalized approach is often the better choice compared to a denormalized option.
Tip 5 - Expire and trim the data
If possible, set expirations on everything. This presents data which is rarely used from sticking around in your cluster. It can be a few days, a few weeks, but definitely set some sort of expiration on your data. If you're inserting into lists or sets be sure to trim them occasionally.
Tip 6 - Use the right eviction policy
If the amount of data that you store in Redis grows over time and you can't afford to keep it all in memory, you probably want to configure Redis as a LRU cache storage. Redis allows for 6 different eviction policies, make sure you pick the right one!
Most of these tips are obvious, but before you switch away from Redis, be sure to give them a try.
Redis: Reducing Memory Usage的更多相关文章
- Reducing and Profiling GPU Memory Usage in Keras with TensorFlow Backend
keras 自适应分配显存 & 清理不用的变量释放 GPU 显存 Intro Are you running out of GPU memory when using keras or ten ...
- Shell script for logging cpu and memory usage of a Linux process
Shell script for logging cpu and memory usage of a Linux process http://www.unix.com/shell-programmi ...
- 5 commands to check memory usage on Linux
Memory Usage On linux, there are commands for almost everything, because the gui might not be always ...
- SHELL:Find Memory Usage In Linux (统计每个程序内存使用情况)
转载一个shell统计linux系统中每个程序的内存使用情况,因为内存结构非常复杂,不一定100%精确,此shell可以在Ghub上下载. [root@db231 ~]# ./memstat.sh P ...
- Why does the memory usage increase when I redeploy a web application?
That is because your web application has a memory leak. A common issue are "PermGen" memor ...
- GPU Memory Usage占满而GPU-Util却为0的调试
最近使用github上的一个开源项目训练基于CNN的翻译模型,使用THEANO_FLAGS='floatX=float32,device=gpu2,lib.cnmem=1' python run_nn ...
- 【转】C++ Incorrect Memory Usage and Corrupted Memory(模拟C++程序内存使用崩溃问题)
http://www.bogotobogo.com/cplusplus/CppCrashDebuggingMemoryLeak.php Incorrect Memory Usage and Corru ...
- Memory usage of a Java process java Xms Xmx Xmn
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html 3.1 Meaning of OutOfMemoryError O ...
- detect data races The cost of race detection varies by program, but for a typical program, memory usage may increase by 5-10x and execution time by 2-20x.
小结: 1. conflicting access 2.性能危害 优化 The cost of race detection varies by program, but for a typical ...
随机推荐
- Ubuntu 16.04.4下安装apache服务
Ubuntu 16.04.4下安装apache服务: 一.首先,准备需要的预装环境 需要c++,make,gcc,apr apr-util pcre.(如果后面报错缺少什么组件,可以百度搜方法. ...
- C++面试常见问题——12虚函数
虚函数 虚函数的工作原理 虚函数的实现要求对象携带额外的信息,这些信息用于确定运行时调用哪一个虚函数,这一信息具有一种被称为虚函数表指针(vptr)的指针形式.vptr指向一个被称为虚函数表(vtbl ...
- xfpt 连接Linux失败问题
首先切换到root用户 1. su 未设置root密码的可以使用一下命令 sudo passwd root 一.上传文件失败(一动不动) 1.安装ftp服务 apt-get install vsftp ...
- BSGS&ExBSGS
BSGS&ExBSGS 求解形如 \[a^x\equiv b\pmod p\] 的高次同余方程 BSGS 假装\(gcd(a,p)=1\). 设\(m=\lceil\sqrt p \rceil ...
- postman 请求get post方法的 区别
1.HTTP的五种请求方法:GET, POST ,HEAD,OPTIONS, PUT, DELETE, TRACE 和 CONNECT 方法. GET请求:请求指定的页面信息,并返回实体主体.(通常用 ...
- AS3.0判断数组中最大值
function getMax(Arr) { if (typeof Arr !="object") { return null; } for (var i=0,max= ...
- 吴裕雄 Bootstrap 前端框架开发——Bootstrap 字体图标(Glyphicons):glyphicon glyphicon-time
<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta name ...
- NO8 find结合sed查找替换企业案例多方法精讲&命令总结!
·sed #替换 eg: sed 'sed 's#已有的内容#更改的内容#g' oldboy.txt s 代表替换,g代表全局,sg就是全局替换 ...
- Linux学习《第五章 用户身份与文件权限》
- Web安全测试学习笔记 - vulhub环境搭建
Vulhub和DVWA一样,也是开源漏洞靶场,地址:https://github.com/vulhub/vulhub 环境搭建过程如下: 1. 下载和安装Ubuntu 16.04镜像,镜像地址:htt ...