Hello everybody,
I have a problem which is similar:
450 customers with 450 HTB classes and their corresponding 450 filters in a
subclass (1:3) of the root qdisc (PRIO) of the interface.
The problem is manifesting at times, when I try to ping a host behind the router
from a host before the router, the latency becomes 1-1,5 msec.
On the machine is running also iptables firewall with a bunch of rules for
dropping/accepting/natting specific traffic, plus routing about 30 Mbits/sec.
When I remove the HTB qdisc, the latency is normal, 0,3-0,4 msec.
Anyone has an idea what could cause this?
Any input much appreciated!
Regards,
Stoimen
--------------
Pablo,
Here we have HTB being used for more than 10.000 customers. The
difference, is that we use tc and u32 filters to classify the packets..
I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz)
on it :)
# uptime
13:18:08 up 16 days, 12:32, 1 user, load average: 0.02, 0.02, 0.00
mpstat says:
01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %
steal %idle intr/s
01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81
0.00 85.61 10568.88
And as you can see.. the use of cpu is not that big..
On May 26, 2007, at 6:54 AM, Pablo Fernandes Yahoo wrote:
> Hello,
>
>
>
> have HTB „rules“ in 4 different ISPs and i control for each
> customer this way:
>
>
>
> Flush and 1:0 class
>
> tc qdisc del dev eth0 root
>
> tc qdisc add dev eth0 root handle 1:0 htb
>
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit
>
> tc qdisc del dev eth1 root
>
> tc qdisc add dev eth1 root handle 1:0 htb
>
> tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit
>
>
>
> Upload and Download: user1
>
> tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil
> 150kbit
>
> tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil
> 50kbit
>
> tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j
> CLASSIFY --set-class 1:5
>
> iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --
> set-class 1:5
>
>
>
> Upload and Download: user2
>
> tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil
> 150kbit
>
> tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil
> 50kbit
>
> tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j
> CLASSIFY --set-class 1:8
>
> iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --
> set-class 1:8
>
>
>
> (…)
>
>
>
> This rules works fine, but just for less than 1.700 customers. More
> than 1.700 customers, i have my load avarage in the sky and
> Ksoftirqd process (top information) in 100% fulltime. I don’t know
> why. I used to use CBQ instead HTB because i had the same problem
> and Ron (a guy in this list) gave this rules and told me that he
> uses this for more than 3.000 customers. I tested it in more than 7
> different computers (but the same hadware specifications) and i had
> the same problem with either CBQ or HTB rules. The computers that i
> have are all of them DELL PowerEdge 1850. I will put some hardware
> iformations here:
>
>
>
> top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 3 root 39 19 0 0 0 R 100 0.0 5316:20
> ksoftirqd/0
>
>
>
> [root at fw ~]# uptime
>
> 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65,
> 1.51 (unfortunately when i took this, the load average was
> „pretty good“, but minutes ago, it was more than 11.0
>
>
>
> [root at fw ~]# lspci
>
> 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub
> (rev 09)
>
> 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A (rev 09)
>
> 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port
> B (rev 09)
>
> 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1
> (rev 09)
>
> 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev
> 09)
>
> 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #1 (rev 02)
>
> 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #2 (rev 02)
>
> 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #3 (rev 02)
>
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB2 EHCI Controller (rev 02)
>
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
>
> 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC
> Interface Bridge (rev 02)
>
> 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> IDE Controller (rev 02)
>
> 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (A-Segment Bridge) (rev 06)
>
> 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (B-Segment Bridge) (rev 06)
>
> 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID
> controller 4 (rev 06)
>
> 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge A (rev 09)
>
> 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge B (rev 09)
>
> 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon
> RV100 QY [Radeon 7000/VE]
>
>
>
> [root at fw ~]# free -m
>
> total used free shared buffers
> cached
>
> Mem: 2021 1479 542 0
> 400 654
>
> -/+ buffers/cache: 424 1597
>
> Swap: 1027 0 1027
>
>
>
> [root at fw ~]# cat /proc/cpuinfo
>
> processor : 0
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5990.78
>
>
>
> processor : 1
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5985.13
>
>
>
>
>
> Any help/Tipp/hint will be very welcome.
>
>
>
> Thanks in Advance!
>
>
>
> Pablo Fernandes
---------------------------------------------------
Webmail of Bulsat Ltd. at http://mail.bulsattv.com/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
|