Optimizing shaper — hashing filters (HTB)
I have a very nice shaper in my linux box :-)
How the configurator works — it’s another question, here i will try to describe how one could configure her shaper with hash-based filtering.
So, this page about configuring linux kernel traffic shaper, which uses filter hashing for massive IP processing.
Introduction
In a couple of words::
Shaper, in some sense, is a queueing discipline, which processes queue(s) of packets. Discipline may be classless or classful. Classful means that certain traffic may be shaped by certain class.
For certain traffic to be shaped by certain class, there is certain filter should be configured.
So, in some sense, classful discipline consists of two trees — tree of classes and tree of filters. Filters filter packets to classes, classes shape traffic.
Anyway, these topics discovered in a lot of manuals :-)
Why hashes.
«Filters tree» is a defined sequence of filters, with possible jumps between branches. But if tree has a lot of filters, it takes a lot of time to reach mathing one.
If while filters tree is a sequence of check an IP address for a match, it is far more effective to use hashes. Hash is a table of keys «matching» their «values», so for every key there may be only one value, and hash searching (»which value corresponds to this key?») is very fast.
In our case key is an IP address and value for a key is a filter, which filters a packet to a certain shaping class.
Surely, see also a LARTC HOWTO chapter for this.
Idea
So, if we create a table with 256 cells for /24 network and look for neede cell using IP address as a key, every needed filter we will find by one step. It would be 128 steps (in average) with no hash table.
If we need to manage two /24 networks, we can create three tables, two of them for 256 cells (for /24’s) and one with two cells. First select «proper» large table by «network» key and then find proper filter by IP (host part particularly).
Like this:
Lets take two /24, 192.168.1.0/24 and 192.168.2.0/24.
Third bytes are 00000001 and 00000010 (in binary format) respectively.
So, kernel with the very first filter chooses a cell in two-cells table — by third byte in IP address. In that cell there is a filter, which points to one of two 256-cells table, and uses fourth byte as a key. With this filter kernel can find the last (possibly) filter in particular cell of particular table. (»Posibly» — because that filter can point to elsewhere, be chained to other filter(s)). The last filter filters a packet to a class, which shapes a packet.
The task
We have:
- Five networks /24: 192.168.1.0/24, 192.168.2.0/24 .. 192.168.5.0/24
- Interface $DEV , directed to these networks
We need:
- To shape packets depending on destination IP address
- Unclassified traffic should be shaped on a special class
Configuring
We will want to have a script, which will configure our kernel… Or to configure kernel directly:
# it may be useful to distinct between
# configuring and creating script:
#tc="/sbin/tc"
tc="/bin/echo /sbin/tc"
Create root qdisc (class 90 — for unclassified traffic):
$tc qdisc add dev $DEV root handle 1: htb default 90
Root class:
$tc class add dev $DEV parent 1:0 classid 1:1 htb rate 100Mbit
Now we need to create «root filter» — other filters need this:
$tc filter add dev $DEV parent 1:1 prio 10 protocol ip u32
Create a table with five cells, one cell for every /24:
# divisor 5 --- table for 5 cells:
$tc filter add dev $DEV parent 1:1 protocol ip prio 10 handle 8: u32 divisor 5
Now let’s create five tables with 256 cells each:
for i in 1 2 3 4 5; do
$tc filter add dev $DEV parent 1:1 prio 10 handle ${i}: protocol ip u32 divisor 256
done
Now let’s fill upper table (with 5 cells), it must contain jumps to particular 256-cells table:
for i in 1 2 3 4 5; do
$tc filter add dev $DEV parent 1:1 protocol ip prio 10 \
u32 ht 8:$[i-1]: \
match ip dst 192.168.${i}.0/24 \
hashkey mask 0x000000ff at 16 \
link $i:
done
This means: put in cell $[i-1] of table 8 (ht 8:$[i-1]:
) a filter, which takes a fourth byte
(hashkey mask 0x000000ff
) of destination IP address (match ip dst 192.168.${i}.0/24
), and uses it as a key for searching in table $i (link $i:
).
Now — a «master filter», which uses tree bits of third byte as a key for searching in a main table:
$tc filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 800:: \
match ip dst 192.168.0.0/21 \
hashkey mask 0x00000700 at 16 link 8:
It means — root filter (ht 800::
) must check destination IP address and, if it matches one our networks, (match ip dst 192.168.0.0/21
) use last three bits of third byte of address (hashkey mask 0x00000700 at 16
) as a key for searching in table 8 (link 8:
).
Now we may create classes/filters for our clients.
This is a very individual process (every administrator decides in which format she will store configuration data — classids, rates etc.), so for the sake of demonstration we will create one clients class and a filter for it:
# class 1: 320 --- parent for clients' classes:
$tc class add dev $DEV parent 1:1 classid 1:230 htb rate 30Mbit ceil 50Mbit quantum 1500 . . .
#
# particular client:
$tc class add dev $DEV parent 1:230 classid 1:431 htb rate 2Mbit ceil 10Mbit quantum 1500 burst . . .
$tc filter add dev $DEV protocol ip parent 1:0 prio 100 u32 ht 3:4: match ip dst 192.168.3.4 flowid 1:431
Last command means — In a fifth cell (numbered from zero!) of third table (for 192.168.3.0/24) out a filter, which will filter packets in class 1:431.
So in a loop we can create classes and filters for every client — and we actually fill our 256-cells tables with these filters.
Real case
In a real case the administrator may want to keep configuration data in a SQL database, real scripts may differ a lot but perform the same job. Such shapers scale quite well, we may double (tripe, quadruple etc ….) number of networks, but filters will work very fast.
Probably you will configure your shaper at a bridge — HTB works quite nice at bridged interfaces, with no IP address assigned.
Notes
- When i worked on this configuration, i noticed that it is necessary to create a «whole» number of 256-cells tables: it should be equal to (2^^n – 1), where n is a number of /24 networks. So, i have to create more tables than i actually need, but otherwise it doesn’t work :-)
- Cells are numbered starting with zero; cells numbers must be in hex. So, in the example above — u32 ht
3:4: match
— four (4) is in hex, too :-) - This is the first beta draft :-)
Optimizing shaper — hashing filters (HTB)的更多相关文章
- Hashing filters for very fast massive filtering
If you have a need for thousands of rules, for example if you have a lot of clients or computers, al ...
- 【Ansible 文档】【译文】模版(Jinja2)
Templating (Jinja2) 正如在 variables 部分描述的那样, Ansible 使用Jinja2模版来启用动态表达式和访问变量. Ansible 扩展了许多 filtes 和 t ...
- 【Ansible 文档】【译文】常见问题
http://docs.ansible.com/ansible/latest/faq.html 如何为一个task或者整个Playbook设置PATH或者任意其他环境变量? 通过environment ...
- 布隆过滤器(Bloom Filters)的原理及代码实现(Python + Java)
本文介绍了布隆过滤器的概念及变体,这种描述非常适合代码模拟实现.重点在于标准布隆过滤器和计算布隆过滤器,其他的大都在此基础上优化.文末附上了标准布隆过滤器和计算布隆过滤器的代码实现(Java版和Pyt ...
- ABP(现代ASP.NET样板开发框架)系列之13、ABP领域层——数据过滤器(Data filters)
点这里进入ABP系列文章总目录 基于DDD的现代ASP.NET开发框架--ABP系列之13.ABP领域层——数据过滤器(Data filters) ABP是“ASP.NET Boilerplate P ...
- ASP.NET MVC Filters 4种默认过滤器的使用【附示例】
过滤器(Filters)的出现使得我们可以在ASP.NET MVC程序里更好的控制浏览器请求过来的URL,不是每个请求都会响应内容,只响应特定内容给那些有特定权限的用户,过滤器理论上有以下功能: 判断 ...
- Unity性能优化(3)-官方教程Optimizing garbage collection in Unity games翻译
本文是Unity官方教程,性能优化系列的第三篇<Optimizing garbage collection in Unity games>的翻译. 相关文章: Unity性能优化(1)-官 ...
- Unity性能优化(4)-官方教程Optimizing graphics rendering in Unity games翻译
本文是Unity官方教程,性能优化系列的第四篇<Optimizing graphics rendering in Unity games>的翻译. 相关文章: Unity性能优化(1)-官 ...
- [Algorithm] 局部敏感哈希算法(Locality Sensitive Hashing)
局部敏感哈希(Locality Sensitive Hashing,LSH)算法是我在前一段时间找工作时接触到的一种衡量文本相似度的算法.局部敏感哈希是近似最近邻搜索算法中最流行的一种,它有坚实的理论 ...
随机推荐
- FZU 1911 Construct a Matrix
题目链接:Construct a Matrix 题意:构造一个矩阵,要求矩阵的每行每列的和都不相同.矩阵的边长是前n项斐波那契的和. 思路:由sn = 2*(fn-1)+(fn-2)-1,只要知道第n ...
- 使用ContentProvider管理多媒体-----查看多媒体数据中的所有图片
import java.util.ArrayList;import java.util.HashMap;import java.util.List;import java.util.Map; impo ...
- admob 广告Android不显示
我弄个了ane,iOS上好好的,Android上打死不显示, 最后发现是少在xml里面增加一个配置,百度半天没搜索到. <android> <manifestAddi ...
- django.test.client 使用随记
import os,sys,django; sys.path.append("\\path\\to\\mysite")#ATTETION!,Err: "unable to ...
- Oracle ODP.NET 篇
1.C# 使用 System.Data.OracleClient 连接 Oracle 需要安装 instantclient , 并配置相应环境变量.重启,方可使用. 2. 使用 System.Data ...
- javax.servlet.ServletException: Error instantiating servlet class Compiler 错误
javax.servlet.ServletException: Error instantiating servlet class Compiler org.apache.catalina.authe ...
- 经典线程同步 信号量Semaphore
阅读本篇之前推荐阅读以下姊妹篇: <秒杀多线程第四篇一个经典的多线程同步问题> <秒杀多线程第五篇经典线程同步关键段CS> <秒杀多线程第六篇经典线程同步事件Event& ...
- ie浏览器兼容性快速处理小招
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/ ...
- matlab中如何保留指定的变量的数据
在使用matlab时,通常要将计算的结果保存到mat文件中.使用的命令是: save filename 但是用上述命令保存的时候,会将计算过程中的其他变量一起保存了.这中粗糙的保存有以下缺点: (1) ...
- Oracle Data Integrator 12c (12.1.2)新特性
改进特性如下: 基于流程界面的声明式设计 在12c中,以前的接口(interface)已经改为映射(mapping),新的基于流程声明的设计方式更灵活,也更容易使用.在12c中,映射的实现是通过使用J ...