-
Copyright IBM Corporation 2013 TrademarksSimulating dropped
TCP/IP packets on IBM AIX Page 1 of 26
Simulating dropped TCP/IP packets on IBM AIX
Jerry Stevens ([email protected])Software Developer,
WebSphere Service Registryand Repository Development teamIBM
Skill Level: Intermediate
Date: 15 Mar 2013
Commercial software products are often developed and tested on
local high-speed, high-capacity networks, where there is little
congestion or packet loss.The network conditions in deployment
environments may not be so ideal. Heavycongestion or hardware
errors on networking equipment might result in lostpackets, which
can severely affect product performance. This article shows youhow
to use and customize a provided facility for simulating dropped
TransmissionControl Protocol/Internet Protocol (TCP/IP) packets on
IBM AIX to allowyou to anticipate how well a product will behave
under less-than-ideal networkconditions.
IntroductionThis article presents:
An AIX kernel extension to permit a specified percentage of
TCP/IP packets toand from a designated host to be dropped randomly
so as to simulate adversenetwork conditions.
A utility to load, activate, and unload the kernel extension. C
and Java utilities to monitor total packet throughput to the target
host and
actual packets dropped. Full source for the kernel extension
with C and Java utilities so that the software
can be built, customized, and developed according to local
requirements.
All the components needed to build and run this packet dropping
simulator areprovided with this article as a downloadable
compressed file. The How to build anduse the kernel extension as
supplied section enables you to get started and use it assoon as
possible.
The kernel extension is designed to be activated on a single
source node from whichit will selectively drop packets either
outbound to a target node or received from it. It
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 2 of 26
is not necessary to install the packet drop simulator on the
target node. The targetnode itself does not even need to be
AIX-based but just needs to support the TCP/IPprotocol.
The article first presents some background to TCP/IP performance
and why droppedpackets can have a serious impact on application
performance. You will then see howto build and use the kernel
extension as provided. This enables you to get up andrunning
quickly with it.
The supplied functionality is deliberately very simple. It
allows you to set a givenpercentage of TCP/IP packets to be dropped
randomly to and from a specifiedhost, with all other hosts and
protocols unaffected and no retained history. Becausein practice,
you might need to add further functionality into the kernel
extensionaccording to your own requirements, you will also be shown
details as to how theextension works and how it can be customised.
You can then decide to do this if youneed to:
Permit packet dropping to and from multiple hosts. Support
different packet drop rates for each of those hosts. Create
historic logs of total packets received, transmitted, and dropped
in both
directions. Implement the packet drop simulator on different
versions of AIX or C compilers
to those used when the software was written.
The kernel extension could be readily developed further to
simulate other networkissues that could give rise to performance
problems, such as packet corruption,packets arriving out of order,
and jitter.
It is recommended the kernel extension is only used on
non-productionsystems.
Background
First let's review some basic terms that are important when
considering networkperformance, bandwidth, latency, and
congestion.
Bandwidth is the capacity of the network, that is the rate at
which data can be pushedacross it. This is usually measured in
Megabits per second (Mbps), where oneMegabit is 10^6 bits. It can
be thought of as the width of the network pipeline.
Latency is the time it takes for one piece of data to traverse
the network. This isusually measured in milliseconds. It can be
thought of as the length of the networkpipeline.
The difference between bandwidth and latency is illustrated in
Figure 1.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 3 of 26
Figure 1. Latency and bandwidth and the network pipeline
Congestion is the effect of heavy usage of the network. It can
result in delaysin propagating data, loss of packets resulting in
packet retransmission, and aninability to make connections to the
network. Packet loss is usually measured as apercentage. TCP can
normally handle losses up to 0.1% with little direct impact. If
thelosses increase above this, the effect can be more severe.
Network latency, bandwidth, and congestion can have a serious
impact onapplication performance. Consider, for example, the
situation where a front-endapplication on one system makes frequent
calls across the network to anothersystem which hosts a database
used by the application. The traffic across thenetwork to the
database might consist of many bursts of small data, or might
consistof fewer sets of relatively large data transfers. Either
way, the ability of the network totransfer this data reliably and
speedily can be a significant factor in the application'sperceived
end-to-end performance by a customer.
Application development teams often have access to isolated,
high-speed networksthat can be much more lightly used than the
environment into which an applicationis to be deployed. This can
mean the effect of poor network performance on theapplication may
never be observed within the environment in which it is
developedand tested. This can, in turn, mean that the performance
observed within acustomer's environment might be very different to
that observed in the environmentfrom which it is tested.
This article focuses on the effect of dropped packets on a
particular application'sperformance. Due to the fact that there can
be very high numbers of packetstransferred between a front-end and
a back-end system and the patterns in whichthese application
packets are transmitted are very specific to the nature of the
front-end and back-end applications, the simplest approach to
understand the impact onthe application is to simulate the dropping
of packets.
Packets can be dropped when transferring data between systems
for two keyreasons:
Heavy network utilization and resulting congestion
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 4 of 26
Faulty network hardware or connectors
TCP is designed to be able to react when packets are dropped on
a network. Whena packet is successfully delivered to its
destination, the destination system sendsan acknowledgement message
back to the source system. If this acknowledgementis not received
within a certain interval, that may be either because the
destinationsystem never received the packet or because the packet
containing theacknowledgement was itself lost. In either case, if
the acknowledgement is notreceived by the source system in the
given time, the source system assumes that thedestination system
never received the message and retransmits it. It is easy to
seethat if the performance of the network is poor, packets are lost
in the first place, andthe increased load from these retransmit
messages is only increasing the load on thenetwork further, meaning
that more packets will be lost. This behaviour can result invery
quickly creating a critical situation on the network.
There are some freely available sophisticated software products
that can help tosimulate different characteristics of varying
network performance. These include:
WANem: Runs from Knoppix Linux based CD ipfilter: Public domain
software but not fully ported to AIX dummynet (FreeBSD) ALTQ
(FreeBSD)
WANem runs from a Knoppix-based CD and can simulate a large
range of differentnetwork characteristics. It does offer the
ability to drop a random number of packetswith filtering options
but would need to run on a separate system in between thesource and
target system.This may not be convenient, for example if the source
andtarget systems are installed on dedicated high-speed
networks.Building the kernelextension and utilities
Although ipfilter looked very promising, it had only been
partially ported to AIX 5.3and it became evident that it would be
quicker to write a bespoke utility providing theminimal
functionality required rather than porting the whole of ipfilter to
both AIX 6.1and 7.1.
dummynet is a component of the FreeBSD operating system and has
alsobeen ported to other platforms including Linux and Microsoft
Windows. Thedisadvantage of using this though is that to simulate
packet drops for packetsoriginating from an AIX host, an
intermediate FreeBSD/Linux host has to beconfigured and set up with
dummynet.
ALTQ is an alternate queueing framework that helps to provide
bandwidth control,mostly on BSD routers.
There are also other packages commercially available such as
LANforge ICE butthese were ruled out for cost reasons. LANforge
also requires dedicated hardware.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 5 of 26
How to build and use the kernel extension as
suppliedRequirementsThe packet drop simulator was built and tested
in the following environments:
AIX V7.1.4.0 with gcc V4.7.2-1 AIX V6.1.2.0 with gcc V4.2.0 AIX
V6.1.4.0 with gcc V4.2.0
The kernel extension and associated C programs were compiled
using the gcccompiler. It should be straightforward to use an AIX C
compiler instead if required,although some of the compilation flags
will need to be changed.
Note that it will be necessary to install the AIX
bos.adt.syscalls file set to build thekernel extension. This will
be available on your AIX installation media. The file set willnot
be required on systems were the kernel extension is only to be
deployed as longas the AIX level is identical to the build system.
If there are any differences in AIXlevels between the two systems,
the kernel extension should be built on each.
The kernel extension, the kctrl utility to load and activate it
and the C example controland monitor are all built with the AIX
make facility. A makefile to build all of thesecomponents is
included in the downloadable zip file.
PrecautionsTwo important factors should be kept in mind before
using this utility:
1) Care should be exercised when developing and testing any
kernel extension. Itis common for errors to result in system
failure. A dedicated test system is ideal,but avoid using systems
that are also used for other purposes or by other users. Ifthe
system is located in a remote data centre, you should also ensure
you have themeans for it to be rebooted when required if it does
not reboot automatically.
2) Avoid the use of flood pings to test the utility on a shared
network as this mayseverely impact other users.
Building the kernel extension and utilitiesFirst make sure you
have met the requirements on your development system asoutlined
above.
Next, extract the downloadable attachment into a working
directory of your choice onthe target AIX system.
The download does not include pre-built binaries so as to avoid
the risk of runningit on incompatible operating system levels. The
software will therefore need to bebuilt once it has been extracted.
To this, change directory to the working directory youcreated above
and then run the make commands as follows:
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 6 of 26
# make gcc -Wall -maix64 -ffreestanding -msoft-float -o
pdrop_kernex64.o -c pdrop_kernex.c ld -b64 -o pdrop_kernex64
pdrop_kernex64.o -e pdrop_init -bI: /home/jerry/kernext/kernex.exp
-bI:/home /jerry/kernext/netinet.exp
-bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys rm -f
pdrop_kernex ar -X64 -r -v pdrop_kernex pdrop_kernex64 ar: Creating
an archive file pdrop_kernex. a - pdrop_kernex64 gcc -Wall -maix32
-o pdrop_ccm32 .c -Xlinker -bI:
/home/jerry/kernext/pdrop_syscall.exp gcc -Wall -maix64 -o
pdrop_ccm64 pdrop_ccm.c -Xlinker -bI:
/home/jerry/kernext/pdrop_syscall.exp gcc -maix64 -o kctrl kctrl.c
gcc -Wall -I/usr/java6/include -I. -shared -fPIC pdrop_jni.c -o
libpdrop_jni.so -Xlinker
-bI:/home/jerry/kernext/pdrop_syscall.expTarget "all" is up to
date.#
After the make command is run successfully, all C components of
the packet droputility required to use it will be ready for
use.
Note that the Java control and monitor application need to be
built separately asdescribed in the Control and monitor
applications section
Loading and activating the kernel extension
One of the binaries that will have been build in the proceeding
step is the kctrlprogram. This permits the kernel extension to be
loaded, activated, and unloaded. Toload and activate the kernel
extension, proceed as follows:
# ./kctrl /home/jerry/kernext/pdrop_kernex64
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndlExtension Successfully loaded, kmid is 1353523200
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndi Extension Initialized
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)nde#
At this point, the kernel extension will be loaded and active,
but will not drop anypackets to a target host until either the C or
Java control and monitor utilities areused.
Initiating packet dropping and monitoring
Having loaded and initialized the kernel extension, you can now
see how to use thesupplied C utility, pdrop_ccm, to control and
monitor the setting of packets to a targethost.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 7 of 26
pdrop_ccm is invoked with arguments that specify the location
where the kernelextension was built, the target host name, and the
drop rate.
# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64Usage:
./pdrop_ccm64 #
You can use this, for example, with the host name fred and a
drop rate of 0.5% asfollows:
# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64 fred
0.5Kernel extenstion is loaded.Setting drop ip to: x.y.z.lDrop
rate: 0.50.
Total In In dropped PC in dropped Total Out Out dropped PC out
dropped 0 0 0.00 0 0 0.0010120 57 0.56 10174 53 0.5222224 134 0.60
22340 116 0.52#
In this example, shortly after we started pdrop_ccm we started
sending IP traffic to thetarget host. The pdrop_ccm utility will
keep running until it is interrupted with a ctrl+cand will print an
update line every 10 seconds.
Please note that the first command line parameter, the kernel
extension name, mustbe specified exactly as it was when loaded with
kctrl.
The drop percentage argument is optional. If it is not supplied,
pdrop_ccm will displaythe current drop rate only and will not
re-set it.
A Java-based control and monitor utility is also provided and
this is described in theControl and monitor applications
section.
That is all there is to get started with the kernel
extension.
The rest of this article presents more details on how the kernel
extension works andhow it and the control and monitor applications
can be developed and customized forindividual use.
How the kernel extension works
The AIX kernel extension is written in the C programming
language. The kernelfunction exposes access to it through a set of
system calls. The C and Java controland monitor applications
utilize these system calls to drive the kernel extensionto drop
packets and to and from a particular host at a given rate. For the
Javaenvironment, a JNI wrapper class provides access to the kernel
extension's systemcalls from Java.
The kernel extension itself consists of:
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 8 of 26
A set of system calls which control the dropping of packets and
the collection ofpacket statistics
Two functions which hook into the kernel's networking layers.
One of these is forinbound traffic and the other for outbound.
These functions are called whenevera packet is received or is to be
dispatched to control the dropping of packets.
The system calls are accessed by the user-level code to set the
remote systemto which incoming and outgoing packets should be
dropped and what percent ofpackets should be dropped. These system
calls also contain functions to retrievecounters defining how many
packets have been dropped in both inbound andoutbound directions
and also a function to reset the counters.
For the C environment, these system calls are used directly in
the example controland monitor application pdrop_ccm.c, which we
used in the Initiating packet droppingand monitoring section above.
For the Java environment, the system calls can beused using JNI.
The JNI wrapper can then be used by a bespoke Java applicationto
control and monitor the dropping of packets. Again, a sample Java
application isprovided as a downloaded resource with this article.
This is called pdrop_jcm.java.Refer to the Java control and monitor
section for further details on the example Javautility.
Figure 2 illustrates the different components to the kernel
extension and the flow ofdata through it. The following two
sections discusses the flow of packet flow throughthe extension,
both outbound data to be sent to the target node and inbound
datareceived from it.
Outbound traffic flow
First, consider the case of outbound traffic, where an
application under test submitsdata to be transmitted across the
network. Refer to point A in the Figure 2. Theapplication submits a
set of data to the kernel for sending to the remote node onthe
network. The TCP subsystem breaks this down into a number of
packets whichare normally then sent to the appropriate network
device driver for dispatch acrossthe network. When using the kernel
extension, however, the hook outbound_fw isset in the IP layer
which results in the kernel extension's outbound filter
functionpdrop_outbound_filter being called for every packet being
dispatched. Thisretrieves the IP header from the packet and matches
it against the target nodefor which packets are to be dropped. If
the node does not match, the function,ip_output_post_fw is called
which routes the packet to the appropriate interface onthe host. If
the node does match, then a random decision whether to pass the
packetis made, which is weighted according to the number of packets
to be dropped. Ifthis decision is that the packet should be sent,
the ip_output_post_fw function iscalled and the packet is passed.
Otherwise, the mbuf holding the data is freed andthe outbound
filter returns and the packet is not sent.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 9 of 26
Inbound traffic flow
For inbound traffic, a corresponding operation applies when data
is received fromthe network by the device driver. Refer to point B
in Figure 2. The device driverforwards the packet of data to the
TCP/IP subsystem. This now calls the inboundfilter which again has
been set by the kernel extension by assigning the inboundhook
inbound_fw to the pdrop_inbound_filter function. This function
makes thedecision whether to pass the packet or not. If the packet
is not from the targetnode for which packets are being dropped, it
is automatically passed by callingthe ipintr_noqueue_post_fw
function. If it is from the target node, then again arandom
decision is made whether to pass the packet according to the
requiredpercentage of packets that are to be dropped. If the
decision is to send the packet,the ipintr_noqueue_post_fw function
is called and the packet is passed. Otherwise,the mbuf holding the
data is freed and the inbound filter returns and the packet is
notpassed up to the higher layers.
The AIX IP filtering hook and kernel services described above
are explained in theIBM AIX 7.1 information center.
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 10 of 26
Figure 2. The packet drop kernel extension
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 11 of 26
Inbound filterThis is the inbound filter function:
/*** This function is the filter for incoming traffic*/void
pdrop_inbound_filter(ifp, m, args)struct ifnet *ifp;struct mbuf
*m;inbound_fw_args_t *args;{ struct ip *ip_in;
if (PDROP_TRUE == amDropping) { ip_in = mtod(m, struct ip
*);
if ( (ip_in->ip_p == IPPROTO_TCP) && (dropip.s_addr
== ip_in->ip_src.s_addr)) { fetch_and_addlp((atomic_l)
&total_in, 1); if (dropMod != 0) { if ( (pdrop_random() %
dropMod) == 1) { fetch_and_addlp((atomic_l) &nin_dropped, 1);
if (m != NULL) { m_freem(m); } return; } }
}}ipintr_noqueue_post_fw(ifp, m, args);return;}
The arguments to the filter function include an mbuf containing
the received data.The mtod macro is used to convert the pointer to
the mbuf into a pointer to thereceived data, which is addressed as
an IP header. A counter of the total number ofreceived packets from
the target address is maintained.
The function then compares the IP address selected for dropping
to the sourceaddress specified in the received packet. If these
match, then the pdrop_random()function is called, which uses the
modulus function to select the requiredpercentage of packets for
dropping. If the packet is to be dropped, then an internalcounter
of dropped packets is incremented. If it is not to be dropped, then
theipint_noqueue_post_fw function is called to pass the packet to
TCP for processing.
The fetch_and_addlp system functions are called to increment the
counts of thetotal number of received and dropped packets. These
ensure that the counts aremaintained atomically so as to be thread
safe.
You can see from this example that it is possible to specify
which protocol is tofiltered. In this case, we are only dropping
packets for the TCP protocol. For testing,
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 12 of 26
it is a good idea to make it use another protocol, such as
Internet Control MessageProtocol (ICMP) so then the packet dropper
can be used with commands, such asping. To do that, the IPPROTO_TCP
constant in the test above would be changed toIPPROTO_ICMP.
Note: amDropping, dropip, and dropMod in the above extract are
defined as globalvariables.
Outbound filter
This is the outbound filter function:
/*** This function is the filter for outbound network
traffic*/int pdrop_outbound_filter(ifp, m, args)struct ifnet
*ifp;struct mbuf *m;outbound_fw_args_t *args;{ struct ip
*ip_out;
if (PDROP_TRUE==amDropping) { ip_out = mtod(m, struct ip *);
if ( (ip_out->ip_p == IPPROTO_TCP) && (dropip.s_addr
== ip_out->ip_dst.s_addr)) { fetch_and_addlp((atomic_l)
&total_out, 1); if (dropMod != 0) { if ( (pdrop_random() %
dropMod ) == 1) { fetch_and_addlp((atomic_l) &nout_dropped, 1);
if (m != NULL) { m_freem(m); } return 0; } } }}
ip_output_post_fw(ifp, m, args);return 0;}
This works very similar to the inbound filter. Counters are
maintained for the totalnumber of packets destined for the target
IP address and for the total number ofoutbound packets dropped.
The pointer to the mbuf is freed if the packet is to be
dropped.
Note: amDropping, dropip, and dropMod in the above extract are
defined as globalvariables.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 13 of 26
Randomizing the packets to be dropped
It is important for the kernel extension to select packets at
random for dropping. Ifsay, packets were dropped after a set number
of packets had been sent, this wouldnot necessarily be a true
emulation of what happens on a heavily used network.Further, it is
possible that the software under test may behave differently when
itmisses packets at regular and irregular intervals.
The standard C library rand() call to generate random numbers
cannot be used ina kernel extension. This is because such functions
are not safe to use in a the re-entrant kernel environment. If you
attempt to use this function, the system might failwhen it is
called from the kernel extension.
A simple function to mimic a call to random was therefore used.
This generates along random number based on the following
inputs:
The seconds field from the current time The nano-seconds field
from the current time The number of calls to the random function
The number of processor ticks since system boot
This is the source of the random function that is used in the
kernel extension.
long pdrop_random(){ static long calls = 0; struct timestruc_t
ts; long x = 0;
curtime(&ts);
// lbolt is the number of clock ticks since boot, see HZ in
/usr/include/sys /m_param.h for number of ticks/sec
fetch_and_addlp((atomic_l) &x, (long) ts.tv_nsec + (long)
ts.tv_sec + (long) lbolt + calls++);
return x;}
This is not of course a true random function but is a good
enough mechanism forassuring that packets are dropped in a
sufficiently irregular manner. As the callsvariable is static, it
is incremented atomically with the fetch_and_addlp
kernelservice.
System calls provided in the packet drop kernel extension
In this section, you are shown details of the various system
calls provided in thekernel extension. These will be of interest if
you need to write your own control andmonitor applications or
customize the kernel extension.
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 14 of 26
Setting the target address: int pdrop_set_drop_address(struct
in_addr*block_addr)
This function takes a pointer to an in_addr struct as an
argument. The method maybe called from the application level from a
string-based host name as follows:
int setDropAddress(char *hostname){ int status = 0; struct
in_addr block_addr;
if (1 == inet_aton(hostname, &block_addr)) { if (PDROP_TRUE
== pdrop_set_drop_address(&block_addr)) status = 1; } return
status;}
Retrieving the target address: int pdrop_get_drop_address()
This function returns the target address as a 32-bit unsigned
integer.
Setting the rate at which packets are dropped: extern
voidpdrop_setDropMod(long m);
This function takes a long input value, which represents how
often packets should bedropped. The way this is worked out is that
the kernel extension generates a randomnumber and then takes a
modulus of that random number with the long value. If theresult is
one, then the packet is dropped, otherwise it is passed.
The call is designed in this way as it is not possible to
perform floating pointoperations within the kernel extension.
Here is a simple way of calling pdop_setDropMod() from the
application level basedon a double percentage value:
void setDropPC(double d){ long x = 0; x = 100.0 /d;
pdrop_setDropMod(x);}
So, for example, if it was required to drop 0.1% of packets,
then d would be 0.1 andthe long value passed to pdrop_setDropMod
would be 1000.
Retrieving the rate at which packets are dropped: extern
longpdrop_getDropMod();
This function returns the long representation of how often
packets should bedropped, which is expressed in the same way as a
call to pdrop_setDropMod();
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 15 of 26
Activate the dropping of packets: void pdrop_startDropping()
When the kernel extension is started, packets are not dropped
until thepdrop_startDropping() function is called. You should call
this function after setting thetarget address and the rate at which
packets should be dropped.
Stop the dropping of packets: void pdrop_stopDropping()
This function stops the dropping of packets. The functions
pdrop_startDropping()and pdrop_stopDropping() can be called as
required to enable and disable packetloss.
Query whether packets are being dropped: int
pdrop_amDropping()
This function returns a 1 if the kernel extension is currently
dropping packets and a 0it is not.
Retrieving kernel extension counters: void pdrop_getstats(struct
pdrop_stats*p)
This function returns a structure which contains details of:
The total number of packets received from the target address The
total number of packets received from the target address which
were
dropped The total number of packets sent to the target address
The total number of packets sent to the target address which were
dropped.
The definition of the structure used is in the header file,
kernext_pdrop.h.
Here is an example of how the counters might be received from
the application level:
pdrop_stats_t j;pdrop_getstats(&j);
printf("Total in: %d.\n", j.total_in);printf("In dropped:
%d.\n", j.nin_dropped;);printf("Total out: %d.\n",
j.total_out);printf("Out dropped: %d.\n", j.nout_dropped);
Resetting the internal kernel extension counters: void
pdrop_reset_counters()
This function is used to reset the counters returned by
pdrop_getstats to zero.
Configuring the kernel extension
The How to build and use the kernel extension as supplied
section providedinformation about how to build the kernel extension
and its utilities. This sectionprovides:
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 16 of 26
More details on the build process Information on AIX 6.1 and AIX
7.1 build issues Details about exporting the system calls provided
in the kernel extension to the
application level Details about loading and activating the
kernel extension Information about the use of system logging to
record when the extension is
loaded and unloaded
AIX 6.1 and AIX 7.1 build issuesBeginning with AIX 6.1, the AIX
operating system simplified its kernel environmentby providing only
the 64-bit kernel. AIX 6.1 and AIX 7.1 maintain application
binarycompatibility with previous AIX versions as specified above,
but device drivers andkernel extensions that are only 32-bit cannot
be built on AIX 6.1 or AIX 7.1.
As this article presents the kernel extension as suitable for
AIX 6.1 and AIX 7.1, it hasbeen built in the 64-bit mode.
The build processA Makefile is included with the compressed file
provided with this article. This can beused to build the kernel
extension and application layer programs associated with it.Note
that this Makefile has been written for use with the AIX make
utility and it willneed to be modified if you wish to use gnu
make.
The kernel extension itself is built with the following
commands:
gcc -maix64 -ffreestanding -msoft-float -o pdrop_kernex64.o -c
pdrop_kernex.c
ld -b64 -o pdrop_kernex64 pdrop_kernex64.o -e pdrop_init
-bI:/usr/lib/kernex.exp -bI:/usr/lib/netinet.exp
-bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys
As noted earlier, the kernel extension is built as a 64-bit
binary. The -ffreestandingand -msoft-float options are used to
prevent the use of floating point instructionsto manipulate certain
data structures. This was required on AIX 7.1, but was notnecessary
on AIX 6.1.
The ld command specifies the entry point into the kernel
extension, which in thiscase is the pdrop_init() function. This
function is called when the kernel extension isactivated.
The ld command also refers to the files kernex.exp and
netinet.exp. These files areprovided with the bos.adt.syscalls file
set. This file set may not be installed on yourtest system in which
case you will need to ask the system administrator to install
it.
Note that it is necessary to make one small modification to the
/usr/include/sys/socketvar.h header file for gcc to compile the
kernel extension satisfactorily. Youshould retain a safe copy of
the file and locate the following line:
extern struct free_sock_hash_bucket free_sock_hash_table[];
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 17 of 26
This should be changed to:
extern struct free_sock_hash_bucket * free_sock_hash_table;
The kernel extension makes this #define in the source:
#define _MSGQSUPPORT 1
This is necessary to prevent the fd_select() system call being
made which is notavailable to the kernel extension. Using #define
changes the fd_select() call to theoriginal select() call, which is
available to the kernel extension.
Exporting the provided system callsThe pdrop_syscall.exp file
defines the kernel extension system calls that areexported to the
application level. The contents of this file are as follows:
#!/unixpdrop_set_drop_address syscall3264pdrop_get_drop_address
syscall3264pdrop_setDropMod syscall3264pdrop_getDropMod
syscall3264pdrop_startDropping syscall3264pdrop_stopDropping
syscall3264pdrop_amDropping syscall3264pdrop_getstats
syscall3264pdrop_reset_counters syscall3264
If you need to customize the kernel extension to include further
system calls, youwill need to amend this file. The syscall3264
identifier at the end of each line makesthe system calls available
to both 32-bit and 64-bit processes. The flag may also beset to
syscall32 to support calls from 32-bit processes only or syscall64
for 64-bitprocesses only. If the flag is not set for the correct
target process environment, theprocess will fail with a
segmentation fault when the system call is made. For furtherdetails
on how to set this identifier refer to the topic, Exporting Kernel
Services andSystem Calls.
Use of kctrl to control loading of the kernel extensionThe
kernel extension is loaded and unloaded with the kctrl program
provided asdescribed earlier. The program should be invoked with
the full path name of thekernel extension. It then interactively
accepts the following commands:
q checks whether the kernel extension has been loaded l loads
the kernel extenstion I initializes the kernel extension t
terminates the kernel extension u unloads the kernel exension e
exits the utility
Here is an example where kctrl is used to query, load,
initialize, terminate, unload,and quit the utility:
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 18 of 26
# ./kctrl /home/jerry/kernext/pdrop_kernex
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndq Extension is not loaded
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndlExtension Successfully loaded, kmid is 1353052160
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndi Extension Initialized
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)ndq Extension is loaded, with kmid 1353052160
Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or
(e)nde#
You can also check whether the kernel extension has been loaded
by using the AIXgenkex command, for example:
# genkex | grep -i kernf1000000c02be000 2000
/home/jerry/kernext/kernext_hello#
In this example, the kctrl executable works with interactive
user input. It would bestraightforward to modify kctrl to take the
command as another argument on thecommand line and thereby make it
easy to incorporate within system startup andshutdown scripts.
However, this is not generally recommended given the nature ofthe
utility.
Logging in the kernel extension
The kernel extension uses the syslogd daemon to record when the
extension isloaded and unloaded. This can be useful for debugging
or auditing purposes. Toenable this logging:
Ensure that logging is enabled in /etc/syslog.conf. For example,
the followingline can be appended to this file:
*.debug /var/log/syslog.out rotate size 100k files 4
Ensure that the log file enabled in /etc/syslog.conf exists.
Refresh the syslogd subsystem (refresh -s syslogd)
Use with other IP protocols
Although the extension is primarily considered for TCP/IP, it
would be simple tochange it to work with other IP-based network
protocols. This was discussed in theHow the kernel extension works
section.
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 19 of 26
Control and monitor applicationsTwo example applications that
can use the system calls exposed by the kernelextension are
provided. One is for the C environment and the other for the
Javaenvironment.
These applications show you how to write your own custom
applications to controlpacket dropping. You can decide to do this
if you need to write your own automatedtest framework to simulate
different network conditions.
It should be noted that only the applications that collect
statistics directly fromthe kernel extension can provide meaningful
statistics on packets that have beendropped. Operating system
provided utilities that report on dropped packets will notinclude
details of packets that have been dropped through the kernel
extension. Thisis because the packets are dropped by the extension
before they are passed throughthe TCP/IP subsystem for dispatch or
delivery.
C control and monitorThe test application, pdrop_ccm.c, is
provided in the downloads section of this article.This is invoked
with a target host name and drop percentage on the command line.The
application passes details of the target host and the drop rate to
the kernelextension and then monitors the inbound and outbound
packets to the target systemevery 10 seconds.
The packet drop system calls can be invoked from both 32-bit and
64-bit applications.The makefile provided demonstrates this by
building both 32-bit and 64-bitversions of the control and monitor
application. These are called pdrop_ccm32 andpdrop_ccm64
respectively.
pdrop_ccm shows the total number of input and output packets,
the number of inputand output packets that were dropped, and the
numbers of dropped packets as apercentage. It takes the name of the
kernel extension, the name of the target hostas arguments and an
optional value specifying the target drop rate. If this
thirdparameter is not specified on the command line, no change to
the drop rate is made.Here is an example of the application in
use:
# ./pdrop_ccm /home/jerry/kernext/pdrop_kernex64 fred 1.0
Extension is loaded, with kmid 1353052160Official name is: fred IP
addresses: 9.20.XXX.YYYTarget IP address: 9.20.XXX.YYY.
Total In In dropped PC in dropped Total Out Out dropped PC out
dropped0 0 0.00 0 0 0.000 0 0.00 0 0 0.0013138 138 1.05 13283 144
1.0829158 275 0.94 29460 302 1.0343738 423 0.97 44206 467 1.0658320
564 0.97 58952 632 1.0772653 721 0.99 73437 784 1.0787227 872 1.00
88153 926 1.05#
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 20 of 26
Here, we can see that the host to which packets are to be
dropped is fred and 1.0%of the packets are to be dropped. Status
updates on total and dropped packets arethen displayed at 10 second
intervals.
The test application calls the pdrop_reset_counters() method to
reset the kernelextension counters. After 10 seconds, you can see
that network activity to the targetsystem is started and the packet
statistics are displayed.
Note that the example controls a single target system only. If
you run the pdrop_ccmcommand again and specify a different target
host name, packets to and from theoriginal host will no longer be
dropped.
Java control and monitorA sample Java application,
PDrop_jcm.java, is also provided, which demonstrateshow to use the
packet drop simulator from the Java environment.
PDrop_jcm.java uses JNI to access the C environment for
controlling and monitoringthe kernel extension. The shared library
libpdrop_jni.so provides native methodswhich can be called from the
Java environment. This shared library is built as part ofthe build
process described earlier from the source, pdrop_jni.c.
The JNI wrapper provides the following native methods at the
Java level which mapto the functions in the shared library:
public native boolean isExtensionLoaded(String extName); public
native boolean setDropAddress(String hostname); private native
String getDropAddress(); public native boolean amDropping(); public
native int setDropping(boolean doDrop); public native void
resetCounters(); public native void setDropHostName(); public
native void setDropPC(float d); public native float getDropPC();
public native long getTotalIn(); public native long getTotalOut();
public native long getInDropped(); public native long
getOutDropped();
When these native methods are invoked, the corresponding
function in the sharedlibrary will be called and the result will be
returned to the Java environment.
To build and use this test Java application, first, make sure
that you have set thePATH to the Java SDK environment correctly,
for example:
export PATH=/usr/java6/bin:${PATH}
Next, set the LIBPATH environment variable to reference the
directory where thekernel extension is located, so the shared
library libpdrop_jni.so can be resolved:
export LIBPATH=/home/jerry/kernext:/usr/lib
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 21 of 26
Now compile the PDrop_jcm.java application, for example:
javac -d . PDrop_jcm.java
Optionally, if you need to regenerate the JNI header file, run
the following command.It mightnot be necessary to run unless you
have customized the kernel extension andchanged or added to the
native methods:
javah -d . PDrop_jcm
If you have regenerated the header, you will need to rebuild the
shared library usingthe process described earlier.
The Java application is then run with the path to the kernel
extension, target hostname, and optional new drop rate as
arguments, as in the C example above:
# java PDrop_jcm /home/jerry/kernext/pdrop_kernex64 fred 1.0Ext
name: /home/jerry/kernext/pdrop_kernex64. Extension is loaded, with
kmid 1353052160Drop address is: 9.20.XXX.YYYDrop PC: 1.0Dropping
enabledTotal In In dropped PC in dropped Total Out Out dropped PC
out dropped0 0 0.0 0 0 0.00 0 0.0 0 0 0.013189 145 1.09 13307 118
0.8828515 295 1.03 28782 267 0.9244306 461 1.04 44701 395 0.8859216
609 1.02 59753 537 0.8972979 756 1.03 73688 709 0.9688606 910 1.02
89455 849 0.94102550 1054 1.02 103567 1017 0.98104522 1071 1.02
105551 1029 0.97104522 1071 1.02 105551 1029 0.97
In this example, you can see that when monitoring started there
were zero packetssent or received to or from the target host, but
there was then network activity thatcaused packets to be dropped in
both directions. In this example, the drop rate hadbeen configured
at 1%.
Measuring non-simulated packet loss
It was mentioned earlier that operating system utilities should
not be used to monitorpackets dropped through the use of the kernel
extension. However, these will berequired when you need to access
packets that are actually dropped across thenetwork. This section
gives some useful hints and tips for using these utilities.
The ping utility displays the packet loss on the ICMP packets
being sent across thenetwork. You can see the packet loss
statistics in the above example, where it iszero. However, this
measurement is only based on these packets being transferredfrom
and to the ping utility and do not measure any other packets
traversing thenetwork.
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 22 of 26
On some systems, ping also supports an option to write ICMP
packets as fast aspossible onto the network and again gives you the
loss statistics at the end. This is aso called flood ping. You
should use this with caution as it is likely to impact
generalnetwork performance while running. It should be run only for
a few seconds and onlyunder test conditions.
The netstat utility can also be used to see how many packets
have been dropped ona network. Here is an example output from
netstat -D on AIX.
Using netstat to report packet drops (AIX):
# netstat -D
Source Ipkts Opkts Idrops
Odrops-------------------------------------------------------------------------------ent_dev0
26820670 16079610 0 0
---------------------------------------------------------------Devices
Total 26820670 16079610 0
0-------------------------------------------------------------------------------ent_dd0
26820670 16079610 0 0
---------------------------------------------------------------Drivers
Total 26820670 16079610 0
0-------------------------------------------------------------------------------ent_dmx0
26820664 N/A 6 N/A
---------------------------------------------------------------Demuxer
Total 26820664 N/A 6
N/A-------------------------------------------------------------------------------IP
50335154 44013816 1110616 147099IPv6 1473 1473 0 0TCP 43110957
40986657 6235 0UDP 6105902 2045877 5022268 0
---------------------------------------------------------------Protocols
Total 99552013 87046350 6139119
147099-------------------------------------------------------------------------------en_if0
26820664 16079528 0 0lo_if0 27970975 27975366 4652 0
---------------------------------------------------------------Net
IF Total 54791639 44054894 4652
0-------------------------------------------------------------------------------NFS/RPC
Client 23 N/A 0 N/ANFS/RPC Server 0 N/A 0 N/ANFS Client 14949 N/A 8
N/ANFS Server 0 N/A 0 N/A
---------------------------------------------------------------NFS/RPC
Total N/A 14977 8
0-------------------------------------------------------------------------------(Note:
N/A -> Not Applicable)#
On AIX, the netstat -Zs -p tcp command can be used to reset the
protocolstatistics before running the promotion activity.
If packet drops are consistently in excess of 0.1%, then you
should raise this withyour network administrator.
Retransmitted packets can be seen using the following
command:
$ netstat -s -p tcp | grep retrans
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 23 of 26
Statistics of interest from the output of this command are:
Packets sent Data packets Data packets retransmitted Packets
received Completely duplicate packets Retransmit timeouts
Conclusion
This article provided reference material and instructions to
build, use, and customizea simple utility to simulate dropped TCP
packets on AIX. Such a utility is invaluablewhen writing cross
network software to model how it will behave under non-idealnetwork
conditions.
The tool may be adapted as required and can easily be enhanced
to supportsimulation of other network issues that can give rise to
performance problems, suchas packet corruption, packets arriving
out of order, and jitter.
Acknowledgements
The author would like to thank the following people for helping
to review this article:
Kavitha Baratakke, IBM Systems and Technology Group, Systems
SolutionDevelopment, IBM USA.
Joshua Carr, WebSphere Service Registry and Performance, IBM UK.
Sukesh Chulliyote, Sr Quality Assurance Engineer, Oracle
Corporation Nigel Griffiths, IBM Sales and Distribution, Systems
and Technology Group
Sales, IBM UK. Paul Harris, WebSphere MQ Performance, IBM
UK.
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 24 of 26
Downloads
Description Name Size Downloadmethod
Packet drop source aix_pdrop.tar.gz 8 KB HTTP
Information about download methods
-
ibm.com/developerWorks/ developerWorks
Simulating dropped TCP/IP packets on IBM AIX Page 25 of 26
Resources ipfilter a packet filtering tool written by Darren
Reed. IBM developerWorks article on developing AIX kernel extension
Follow JNI developerWorks article
-
developerWorks ibm.com/developerWorks/
Simulating dropped TCP/IP packets on IBM AIX Page 26 of 26
About the author
Jerry Stevens
Jerry Stevens leads the WebSphere Service Registry and
Repository(WSRR) Performance team at the IBM Software Lab in
Hursley Park,United Kingdom. He joined IBM UK Global Services in
1997 as an IBMAIX Consultant, then transferred to WebSphere MQ
Development andPerformance Analysis in 2001, and to WSRR
Development in 2009.He took on the WSRR Performance team role in
2011. Prior to joiningIBM, Jerry worked for Shell UK as a Senior
Systems Engineer, wherehe undertook a range of technical
consultancy and development rolesand worked with a variety of open
systems platforms and architectures.He graduated from Exeter
University with an Honours Degree inMathematics.
Copyright IBM Corporation
2013(www.ibm.com/legal/copytrade.shtml)Trademarks(www.ibm.com/developerworks/ibm/trademarks/)
Table of ContentsIntroductionBackgroundHow to build and use the
kernel extension as suppliedRequirementsPrecautionsBuilding the
kernel extension and utilitiesLoading and activating the kernel
extensionInitiating packet dropping and monitoring
How the kernel extension worksOutbound traffic flowInbound
traffic flowInbound filterOutbound filterRandomizing the packets to
be droppedSystem calls provided in the packet drop kernel
extension
Configuring the kernel extensionAIX 6.1 and AIX 7.1 build
issuesThe build processExporting the provided system callsUse of
kctrl to control loading of the kernel extensionLogging in the
kernel extensionUse with other IP protocols
Control and monitor applicationsC control and monitorJava
control and monitor
Measuring non-simulated packet
lossConclusionAcknowledgementsDownloadsResourcesAbout the
author