Automated SIP Interoperability Testing Archana Rao and Henning Schulzrinne Department of Computer Science Columbia University

Automated SIP Interoperability TestingAutomated SIP Interoperability Testing

Archana Rao and Henning SchulzrinneDepartment of Computer Science

Columbia University

http://www.cs.columbia.edu/IRT/voip-testbed/

SIP 2009 (Paris, January 2009)

OverviewOverview

The problem of SIP interoperability

Far from perfect

Damages “brand”, harms customers, encourages single-vendor deployments

Testbed Overview

Architecture

Components

Experiments

Interoperability study

End-client characteristics

Summary

InteroperabilityInteroperability

Hi Henning,

You may remember me from IETFs past -- I haven't attended any in some time because I couldn't find any really

interesting projects. I'm finally getting into SIP. I've got Speakeasy VoIP service, two sipphone accounts, a Cisco

7960 and a copy of x-ten on my Mac. And I still can't make it work. Voice flows in one direction only. I'm not even

behind a NAT or firewall -- both machines have global addresses, with no port translations or firewalls.

I've been working with Internet protocols for over 20 years. I've implemented and contributed to them. And if *I*

can't figure out how to make this stuff work, how is the average grandmother expected to do so?

SIP is unbelievably complex, with extraordinarily confusing terms. There must be half a dozen different

"names" -- Display Name, User Name, Authorization User Name, etc -- and a dozen "proxies". Even the word

"domain" is overloaded a half dozen different ways. This is ridiculous!

Sorry. I just had to get this off my chest. Regards, Phil Karn.

Is it a real problem ?

Reference: http://www1.ietf.org/mail-archive/web/rai/current/msg00082.html

Excerpts from an email posted on IETF RAI mailing list –

Interoperability issuesInteroperability issues

Why do we have interoperability problems?

SIP is complex

More than 150 RFCs and 500 active Internet Drafts

Efforts like – “draft-ieft-sip-hitchhikers-guide” help to a certain extent, but not sufficient

SIP has no proposed architecture/profile

SIP usage in variety of environments and domains make it impossible

Standardization from IETF is different from that of ISO/ITU

SIP is continually evolving

Non trivial for implementers to follow the life-cycle from IDs to RFCs

Walled gardens, non-standard environments

Vendors need to make products that work in their customer’s environment

Poor implementations, intentional non-interoperability etc.

Improving interoperabilityImproving interoperability

Current efforts to combat SIP interoperability issues

SIP Interoperability (SIPit) events

Week long gatherings of SIP implementers to test interoperability

Coordinated by SIP Forum

SIP Forum

SIP Connect 1.0 (and 1.1) outsourced enterprise VoIP

IETF Basic Level of Interoperability for SIP Services (BLISS)

Focus on resolving interpretability issues in SIP features

Line sharing, call parking, automated call handling and call queuing

University of New Hampshire – InterOperability Laboratory

Vendor-neutral lab dedicated to testing data networking technologies

20 different testing programs, each costing about $20,000 per year

InteroperabilityInteroperability

How do we go about it?

1. Identify the basic real-world scenarios for SIP registration and session establishment

2. Use our VoIP testbed to configure a variety of SIP infrastructure

3. Study the behavior of SIP end-clients and servers for interoperability

So, isn’t the problem solved? These forums focus on details of advanced SIP features

Basic-level of interoperability between innumerable SIP devices in the market

Can I take any SIP phone and make a VoIP call, through any SIP service

provider?

Studying interoperabilityStudying interoperability

1. Define a minimum set of call flows constituting basic interoperability

RFC 3665, RFC 3666, RFC 4317

2. Use a VoIP testbed to realize a variety of real-world SIP infrastructure setups

Columbia VoIP testbed

3. Study the behavior of SIP end-clients and servers in each of these scenarios

Interoperability matrices

Categorization of common issues Lack of clarity in the specification

Implementation of an older specification

Incomplete implementation of the specification

Incorrect implementation of the specification

Failure against robustness tests

Columbia VoIP testbedColumbia VoIP testbed

What? VoIP infrastructure for experimentation, analysis, testing, prototyping and deployment of

SIP/VoIP components in a variety of environments.

Part of a multi-university research project supported by NSF

University of North Texas, Purdue University and University of California, Davis

Why? There’re no telecom testbeds available for research & experimentation

VoIP implementations & experiments have lacked scientific rigor

Continuously emerging standards

Need to develop repeatable, accurate test frameworks

Realistic VoIP experiments require a distributed testbed

your equipment here :-)

http://www.cs.columbia.edu/IRT/voip-testbed/

http://www.cs.columbia.edu/IRT/voip-testbed/

Columbia VoIP testbed: serversColumbia VoIP testbed: servers

5 SIP servers

Asterisk, Pbxnsip,

Sipd, SER, OpenSER

3 Platforms Microsoft Windows XP

Linux Fedora Core 6

Sun Solaris X

3-way Connectivity the Internet

VPN

PSTN

Columbia VoIP testbed: UAsColumbia VoIP testbed: UAs

20+ SIP hard-phones, soft-phones, WI-FI phones with different capabilities

More details – www.columbia.edu/~ahr2114/VoIPtestbed/end-clients

NSF VoIP testbedNSF VoIP testbed

TheTheInternetInternet

UNIVERSITY-A UNIVERSITY-B

PSTNPSTNNetworkNetwork

UNIVERSITY-CUNIVERSITY-D

Image courtesy: http://secnet.csci.unt.edu/7.15.funding_application.ppt

Columbia VoIP testbed experimentsColumbia VoIP testbed experiments

Planned or ongoing

Interoperability study

Can any two SIP devices talk to each other?

SIP end-client characteristics

Signaling and codec features in SIP devices

Robustness

VoIP with/despite NAT

Performance issues

Signaling delay, voice mouth-ear delay, packet loss resilience

Servers – Scalability, maximum capacity

Security

Separate testbed for DOS and other attacks

work in progress

Interoperability study: call flowsInteroperability study: call flows

Registration (RFC 3665) Successful new registration

Unsuccessful registration

Cancellation of registration

Update of contact list

Session Establishment (RFC 3665) Successful session establishment

Session establishment through two proxies

Session establishment with multiple proxy

authentication

Successful session with proxy failure

Unsuccessful no answer

Unsuccessful busy

Unsuccessful no response from user agent

Unsuccessful temporarily unavailable

Codec Negotiation (RFC 4317) Audio and DTMF

Audio, video and DTMF

Audio and video codec reordering

Interoperability study: matrixInteroperability study: matrix

Interoperability Matrix

Test case #1 - Successful new registration

Asterisk 3Com sipd SER

Eyebeam 1.5 Yes Yes Yes Yes

Wengophone Yes No No Yes

Polycom PVX Yes No No Yes

Express Talk 3.0 Yes Yes Yes Yes

Cisco 7940 Yes Yes Yes Yes

Grandstream GXV 3000 Yes Yes Yes Yes

Snom 320 Yes Yes Yes Yes

Polycom VSX 7000e Yes No No Yes

Interoperability study: auth nameInteroperability study: auth name

Issue 1 – Use of different formats for authentication name

Authorization: Digest username=“user1”,realm=“proxy01.sipphone.com”,nonce=“xxx”,

uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5

Authorization: Digest username=“sip:[email protected]”, realm=“proxy01.sipphone.

com”,nonce=“xxx”,uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5

Implication

Registration and call setup failure, if both formats are not supported by the UAs

Instances

Polycom PVX/VSX, Wengophone don’t register with 3Com / sipd proxies

Workaround

Use Asterisk between Polycom PVX and 3Com proxy

Interoperability study: rportInteroperability study: rport

Implication

Incorrect/incomplete signaling resulting in

multiple retransmissions and failed transaction

Instances

Calls to Cisco 7940 via Sipphone proxy are

always unsuccessful

Calls from Polycom VSX via 3com or sipd

proxies are always unsuccessful

Workaround

Nothing

Issue 2 – Behavior in the absence of rport parameter

SIP server UAC

INVITE (Dest port:5060)

180 Ringing (Src port:12345)

200 OK (Src port:12345)

ACK (Dest port:12345)

ACK (Dest port:12345)

200 OK (Src port:12345)

Interoperability Study: codecsInteroperability Study: codecs

Implication – Audio/Video failure (mostly unidirectional, but bidirectional sometimes)

Problem Instances

Audio failure when Cisco calls voicemail service of sipphone provider

Video failure when Polycom PVX calls Xlite via Asterisk

Issue 3 – Codec Negotiation

[Offer]v=0o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67s= SIP Callt=0 0m=audio 31030 RTP/AVP 0 8 18 101c=IN IP4 128.59.17.67a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:18 G729/0a=fmtp:18 annexb=no a=rtpmap:101 telephone-event/8000a=fmtp:101 0-15a=sendrecv

[Answer]v=0o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67s= SIP Callt=0 0m=audio 31030 RTP/AVP 18 0 8 101c=IN IP4 128.59.17.67a=rtpmap:18 G729/0a=fmtp:18 annexb=noa=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000a=fmtp:101 0-15a=sendrecv

UA characteristics: featuresUA characteristics: features

Basic

Configuration, number of lines, call hold, call forwarding, DND, NAT support

Advanced

Encryption, symmetric RTP, conferencing, audio/video codecs, PoE

Audio quality – silence suppression, echo cancellation, comfort noise

Transport, signaling and support protocols – TCP, TLS, HTTP, DHCP, DNS, TFTP, NTP

More details at – http://www.columbia.edu/~ahr2114/VoIPTestbed/EndClients.htm

UA characteristics: robustnessUA characteristics: robustness

SIP Torture Tests (RFC4475)

A set of SIP messages aimed at stressing the SIP parser, based on experiences at SIPit.

PROTOS

Test suite with 4500+ INVITE requests, developed by University of Oulu

Robustness

SIP end-clients

PROTOS Test-cases

Behavior

WengoPhone General CPU consumption is very high (varies between 50% and 90%), throughout the testing.

SIPc 368, 369 Software crashes (with a message “Hostname variable too long”)

ExpressTalk

General Software crashes, if an INVITE is received, when all the 6 lines are busy.

2363, 2366, 2371, 2374, 2383 – 2406

Software crashes (can’t handle improper values for “Content-length”)

Example Results

UA characteristics: NATsUA characteristics: NATs

SIP and NAT

Problematic for both signaling (SIP) and media transfer (RTP)

Circumventing NAT effect – solutions exist for both end-clients and proxies

Quest to overcome this, could lead to techniques uglier than NAT itself!

NAT test setup

NAT realization - Linux iptables, CLICK router

Studying the behavior in different scenarios

NAT support enabled in both the proxy server and end-client

NAT support enabled in only one of them etc

Aiming to analyze the existing practices – the good, the bad and the ugly!

UA with NATsUA with NATs

SIP devices in NAT

environment

What now?What now?

Summary

Tests indicate that basic-level interoperability is far from perfect

Instances of interoperability failures illustrated in our IETF workshop paper

Achieving Interoperability shouldn’t be left just to vendors

non-interoperability damages SIP brand, causes grief for third parties and harms customers

Going ahead – some ideas

Establish designated interop liaison for each vendor

calling customer support unlikely to be useful

Encourage vendors to publish interoperability reports

in standard format

Provide self-certification tests, supported by a remotely accessible test rig

SummarySummary

Current status

The VoIP testbed & VPN connectivity is operational

Systematic study of real-world interoperability issues

Basic-level interoperability is nowhere near 100%

IETF 70 – The 1st SIP Forum Workshop on SIP Interoperability

Evaluating SIP end client characteristics

Testbed plans

Experiments on NAT, performance, security

Make the testbed accessible and useful for engineers & deployments

Create a knowledge repository about SIP devices & tools for interoperability testing