kernel_optimize_test/net/ipv6
Eric Dumazet 2b85a34e91 net: No more expensive sock_hold()/sock_put() on each tx
One of the problem with sock memory accounting is it uses
a pair of sock_hold()/sock_put() for each transmitted packet.

This slows down bidirectional flows because the receive path
also needs to take a refcount on socket and might use a different
cpu than transmit path or transmit completion path. So these
two atomic operations also trigger cache line bounces.

We can see this in tx or tx/rx workloads (media gateways for example),
where sock_wfree() can be in top five functions in profiles.

We use this sock_hold()/sock_put() so that sock freeing
is delayed until all tx packets are completed.

As we also update sk_wmem_alloc, we could offset sk_wmem_alloc
by one unit at init time, until sk_free() is called.
Once sk_free() is called, we atomic_dec_and_test(sk_wmem_alloc)
to decrement initial offset and atomicaly check if any packets
are in flight.

skb_set_owner_w() doesnt call sock_hold() anymore

sock_wfree() doesnt call sock_put() anymore, but check if sk_wmem_alloc
reached 0 to perform the final freeing.

Drawback is that a skb->truesize error could lead to unfreeable sockets, or
even worse, prematurely calling __sk_free() on a live socket.

Nice speedups on SMP. tbench for example, going from 2691 MB/s to 2711 MB/s
on my 8 cpu dev machine, even if tbench was not really hitting sk_refcnt
contention point. 5 % speedup on a UDP transmit workload (depends
on number of flows), lowering TX completion cpu usage.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-06-11 02:55:43 -07:00
..
netfilter netfilter: Use frag list abstraction interfaces. 2009-06-09 00:23:58 -07:00
addrconf_core.c
addrconf.c IPv6: Add 'autoconf' and 'disable_ipv6' module parameters 2009-06-01 03:07:33 -07:00
addrlabel.c
af_inet6.c IPv6: Add 'autoconf' and 'disable_ipv6' module parameters 2009-06-01 03:07:33 -07:00
ah6.c netns xfrm: AH/ESP in netns! 2008-11-25 17:59:27 -08:00
anycast.c
datagram.c netns xfrm: lookup in netns 2008-11-25 17:35:18 -08:00
esp6.c netns xfrm: AH/ESP in netns! 2008-11-25 17:59:27 -08:00
exthdrs_core.c
exthdrs.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
fib6_rules.c net: Remove unused parameter from fill method in fib_rules_ops. 2009-05-20 17:26:23 -07:00
icmp.c net: fix xfrm reverse flow lookup for icmp6 2009-01-27 22:30:19 -08:00
inet6_connection_sock.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
inet6_hashtables.c ipv6: don't use tw net when accounting for recycled tw 2009-02-26 03:35:13 -08:00
ip6_fib.c ipv6: Fix fib6_dump_table walker leak 2009-01-13 22:17:51 -08:00
ip6_flowlabel.c ipv6: Disallow rediculious flowlabel option sizes. 2009-02-06 00:49:55 -08:00
ip6_input.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
ip6_output.c net: No more expensive sock_hold()/sock_put() on each tx 2009-06-11 02:55:43 -07:00
ip6_tunnel.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
ip6mr.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
ipcomp6.c netns xfrm: state lookup in netns 2008-11-25 17:30:50 -08:00
ipv6_sockglue.c ipv6:remove useless check 2009-04-14 02:21:41 -07:00
Kconfig trivial: fix typos/grammar errors in Kconfig texts 2009-03-30 15:22:01 +02:00
Makefile
mcast.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
mip6.c netns xfrm: KM reporting in netns 2008-11-25 17:51:01 -08:00
ndisc.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
netfilter.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
proc.c snmp: add missing counters for RFC 4293 2009-04-27 02:45:02 -07:00
protocol.c
raw.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
reassembly.c ipv6: Use frag list abstraction interfaces. 2009-06-09 00:20:05 -07:00
route.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
sit.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
syncookies.c syncookies: remove last_synq_overflow from struct tcp_sock 2009-04-20 02:25:26 -07:00
sysctl_net_ipv6.c remove lots of double-semicolons 2009-01-08 08:31:14 -08:00
tcp_ipv6.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
tunnel6.c
udp_impl.h
udp.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
udplite.c
xfrm6_input.c netns xfrm: per-netns MIBs 2008-11-25 17:59:52 -08:00
xfrm6_mode_beet.c
xfrm6_mode_ro.c
xfrm6_mode_transport.c
xfrm6_mode_tunnel.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
xfrm6_output.c net: skb->dst accessors 2009-06-03 02:51:04 -07:00
xfrm6_policy.c net: replace uses of __constant_{endian} 2009-02-01 00:45:17 -08:00
xfrm6_state.c ipv6: fix sparse warning: Using plain integer as NULL pointer 2009-02-21 23:37:10 -08:00
xfrm6_tunnel.c xfrm6_tunnel: join error paths using goto 2008-12-14 23:13:48 -08:00