October 2001

Testing, Testing, 1, 2, 3...

BY CHRIS DONNER

Mr. Bush's comment reflects a fundamental misunderstanding of the testing process, and I don't think he is alone in this misunderstanding. Many people see testing as an end, as some hurdle to be overcome, after which it's all smooth sailing. I remember in college how students (myself included) were more concerned with test results than with retaining or utilizing what we had learned. People would grumble horribly upon learning that a test was to be comprehensive, requiring knowledge of material all the way back to the beginning of the term, rather than specific only to the last three or four weeks of classes.

Testing is not an end. Instead, it is a process. It is a form of assessment, but the act of passing a test is not really the desired result. There is a pride and sense of accomplishment that comes with passing a test, but hopefully there is more.

The Bush quote stands in contrast with the saying attributed to Lao Tzu: "Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime." This quote makes no mention of testing, but there is a test here. Inherently, Lao Tzu assesses two results and decides that it is better to teach someone to do something for themselves than to do it for them. The testing is less apparent than in the Bush quote, but the end result is much more satisfying.

So what does all of this have to do with testing capacity or interoperability in the network? On the surface very little, but look deeper and you'll see that the Lao Tzu quote symbolizes an approach that focuses more on the actual delivery of a service rather than simply passing a test and moving on. You might paraphrase it as: "Give the PSTN a packet and you see what happens to that packet; teach the PSTN to handle packets and you see convergence." Or something along those lines.

In other words, while testing is crucial, it is the actual service being delivered that counts. The closer your testing strategy comes to assuring that the service is being delivered as promised, not only when being tested but all the time, the more effective your testing strategy is. The more likely it is that you are passing the test of consistent user satisfaction, and that at the end of the day, Johnny in fact can read.

[ Return To The October 2001 Table Of Contents ]

• Consultronics Intros Single-Ended DMT Testing
• Empirix Announces Call Patrol Monitor
• Ameritec Releases FeatureCall Millennium
• KPN Belgium Selects Agilent OSS Solution
• Tekelec Announces Breakthrough Diagnostic Solution
• InCert Joins Forte For Java Framework Extension Program
• NetIQ Introduces New WebTrends Product Lineup
• RADCOM, SHUNRA To Develop, Promote Integrated Emulation Solution
• Anritsu Intros Jitter Analyzer For 2.5/10 Gbps
• GL Enhances Voice Quality Assessment Solution

Consultronics Intros Single-Ended DMT Testing
Consultronics is offering a single-ended measurement method to determine the up- and downstream connection rates a local loop can support. The portable CableSHARK performs single-ended rate prediction tests for DMT-based ADSL. The user connects a single CableSHARK to one end of the line; the other end needs only an open or short circuit. The CableSHARK determines the cable's frequency response, then determines how many bits each ADSL DMT carrier can transport. Results are based on ANSI T1.413 standards.
www.consultronics.com, 800-227-3345

Empirix Announces Call Patrol Monitor
Empirix announced its Call Patrol contact center performance monitor. Call Patrol is a hosted service that monitors overall system performance from the PSTN all the way to backend databases, and interacts with the system just as a customer would, by dialing in, stepping through IVR prompts, and determining if this results in the appropriate call flow. A Web interface into Call Patrol allows for both user configuration and access to performance data and reports for assessing current and historical performance. By monitoring your systems and carrier performance, Call Patrol detects outages and slowdowns and resolves them more quickly -- improving your customers' quality of experience and lowering additional call charge and agent costs. Call Patrol monitors response times, prompt accuracy, and error conditions and, when appropriate, sends out immediate alerts indicating in which system the problem was encountered and where in the call flow it occurred.
www.empirix.com, 781-993-8500

Ameritec Releases FeatureCall Millennium
Ameritec has released FeatureCall Millennium, the company's fourth-generation GUI for the control and management of its Crescendo, AM2 Niagara, and AM2S Squirt family of network load generators. FeatureCall Millennium is an "Explorer"-modeled GUI that enables the operator to manipulate protocols, scripts, call programs, test sets, units, and system domains in the same manner as they would any file on their PC. These virtual files can be shared with any Ameritec call generator on a LAN, or they can be e-mailed.
www.ameritec.com, 626-915-5441

KPN Belgium Selects Agilent OSS Solution
Agilent Technologies announced that KPN Belgium, an affiliate of KPN, one of the largest telecommunications companies in the Netherlands, has chosen Agilent's NETeXPERT-based operations support system (OSS) solution to automate service assurance for customer-responsive, flow-through operations for its expanding broadband services and networks. KPN Belgium will use Agilent's unified OSS solution to enhance network performance and reliability for its new digital subscriber line (DSL) and intelligent network (IN) services. By collecting and correlating real-time network fault and performance information, this solution also enables KPN Belgium to accurately show network availability, reduce mean-time-to-repair (MTTR) and better fulfill service level agreements (SLAs).
www.agilent.com, 650-752-5000

Tekelec Announces Breakthrough Diagnostic Solution
Tekelec announced a breakthrough technology that enables companies to demonstrate real-world IP functionality on systems and devices under test. The Tekelec IP Pass-though solution allows companies to demonstrate Internet and Web-based products and services in the absence of many 3G network elements, including the subscriber terminal. The IP Pass-through is a solution, enabled by the Tekelec MGTS [Message Generator Traffic Simulator] simulation capability, which allows Web browsing and other IP applications to pass through a simulated wireless terminal and through the device under test or a simulated network element to the Internet.
www.tekelec.com, 800-TEKELEC

InCert Joins Forte For Java Framework Extension Program
InCert Software Corp. has joined the growing number of tool vendors building extensions to Sun Microsystems' Forte for JavaT integrated development environment (IDE). The combination of the two products gives Sun customers the tools they need to rapidly resolve application faults and failures. InCert's unique approach to diagnosing the root cause of software bugs in JavaT language applications -- in contrast to traditional Java language debuggers -- allows programmers to identify and remedy bugs in minutes versus hours. By allowing developers and release engineers to enable TraceBack directly from the Forte for Java IDE, diagnosis and recovery from problems encountered in both laboratory and run-time environments will be faster and more efficient than when employing traditional debugging tools.
www.incert.com, 877-3INCERT

NetIQ Introduces New WebTrends Product Lineup
NetIQ Corporation announced a new WebTrends product lineup that provides more targeted product offerings for Web analytics customers. The new product lineup will focus on four distinct product categories, based not only on the type of organization --from small/medium businesses to enterprises and service providers -- but also on the complexity of an organization's site and its reporting needs. The four WebTrends product categories are: WebTrends Analysis; WebTrends Reporting Center; WebTrends Visitor Relationship Management (VRM) Solutions; and WebTrends Live, a service for businesses that prefer the convenience of a hosted Web analytics service.
www.netiq.com, 408-856-3000

RADCOM, SHUNRA To Develop, Promote Integrated Emulation Solution
RADCOM, a leading network test and quality management solutions provider, announced the signing of a cooperation agreement with SHUNRA Software Ltd., a leading developer of WAN emulation technologies. The two companies have joined forces to incorporate SHUNRA's Internet simulation technology with RADCOM's integrated VoIP system performance solution. The deal will combine the strengths of SHUNRA's WAN emulation solutions with the cross-measurement capabilities of RADCOM's VoIP Performer solution. This will deliver one of the most comprehensive VoIP performance measurement system currently available to convergence service providers and developers.
www.radcom-inc.com, 1-800-RADCOM-4
www.shunra.com, 201-634-8787

Anritsu Intros Jitter Analyzer For 2.5/10 Gbps
Anritsu Company introduced the MP1580A, which at 320mm x 100mm x 350mm they claim is the world's smallest jitter analyzer measuring jitter characteristics at 2.5 and 10 Gbps. Additionally, the MP1580A can generate variable types of TDEV masks at 10 Gbps when conducting wander measurements. Six high-speed jitter and wander measurements can be made automatically with the MP1580A: jitter tolerance, jitter sweep, jitter transfer, frequency sweep, wander sweep, and wander. A number of other measurements can also be made.
www.us.anritsu.com, 800-ANRITSU

GL Enhances Voice Quality Assessment Solution
GL Communications has announced that their Voice Quality Assessment software now uses PESQ (Perceptual Evaluation of Speech Quality per ITU Rec. P.862). PESQ provides objective measurement of subjective listening tests on telephony systems. The application is offered for use on GL's Ultra T1 (or E1) PCI boards and laptop analyzers. Other application software includes BERT testing, record/playback, DTMF/MF/MFC-R2 detection and generation, signaling bits recording, and a variety of protocol analyses (Frame Relay, HDLC, SS7, and ISDN) applications, and a suite of applications designed for automated and manual echo canceller testing, specifically per G.168.
www.gl.com, 301-670-4784 x114

[ Return To The October 2001 Table Of Contents ]

BY BOB MASSAD

IP networks suffer from a variety of impairments. Typical impairments, such as delay, jitter, and packet loss, critical VoIP parameters, are time varying. They can and do occur at unpredictable times during some time interval and even during a single call. The time-variability stems most notably from network load at various points in time. The load is affected by many factors including the number of simultaneous users, differing application types and accompanying resource requirements, bandwidth reservations, packet priorities, and even the level of link over-subscription employed by service providers. These factors are relative to the finite amount of bandwidth available for use.

VoIP call quality will vary along with the occurrence of impairments, including during the course of a call. In order to provide or receive subscribed quality of service levels, without over-provisioning or over-subscribing, QoS monitors must consider time-variability to provide an accurate assessment of call quality. Averages, counts, and rates are insufficient, and they may actually mask the problem.

In a perceptually-oriented application such as VoIP, it is also important to accurately assess the end-user/listener's perception of call quality, which is affected by the time-variability of the impairments.

How "Bursty" Is Your Packet Loss?
Packet loss appears as the most damaging impairment for VoIP call quality. Jitter buffers do a great job of removing jitter, though at the cost of delay and packet loss. The buffer queuing process increases delay, which is particularly deleterious to two-way conversational quality, causing awkwardness or "double-talk." Packet loss, and thereby actual voice energy loss, is increased by excessive delay, i.e., delay outside the jitter buffer threshold. A significant point is that packets are not only dropped by the transporting network, but also by the jitter-relieving buffer. To get a true picture of packet loss, a monitor must see the packet stream post the jitter buffer process.

Packet loss often occurs as a result of congestion or rerouting in the network, and hence lost packets are likely to occur in bursts. The degree of "burstiness" has a major impact on voice quality. Packet loss concealment algorithms used by VoIP implementations can disguise isolated lost packets but are much less effective when many packets are lost, as in a burst.

Figure 1 shows actual recorded packet loss information from the Internet. What's striking about the figure is the consistency of packet loss throughout the interval and the number of occurrences of loss bursts involving six consecutive packets.

Figure 1

Recorded packet loss information from the Internet1

Figure 2 shows the results of subjective tests conducted by AT&T on the effects of packet loss burstiness. A listening panel rated audio files impaired by both random and burst packet loss for differing packet loss rates. At very low packet loss rates, burst packet loss can sound better than random loss -- as the gaps between loss events become large. But at loss rates above 3%, it is clear that burst packet loss has a significantly greater effect on voice quality than random loss.

Figure 2

Subjective quality for burst vs. random packet loss (G.711 PLC)2

Clearly, from figure 2, any method of monitoring call-quality that does not consider the packet loss bursts but randomizes or averages packet loss is mischaracterizing call quality, and by up to forty percent. More importantly, not considering loss bursts can often give the impression of good quality when the quality level is actually poor.

Human Memory Effects: "Recency"
Time-variability also has considerable affect on the user perception of call quality. During subjective testing researchers found that the call quality reported by a listener depended on the time delay between a significant burst of noise and the end of the call. This is believed to be due to the "recency effect" which relates to a human tendency to remember the most recent events, and possibly to short-term auditory memory decay.

AT&T found that moving a burst of noise from the beginning to the end of a 60-second call changed the reported MOS score from 3.82 to 3.18, essentially a twenty percent swing. The percentage is greater when considering that the typical value range is 2.5 to 4.0.

Summary
Service providers want to increase call durations, retain customers, and improve user satisfaction. To do so, there is a need for accurate metrics. Accurate metrics are those that objectively model the effects of these time-varying-impairments, especially packet loss, and incorporate end-user perception. If providers wait until their customers are complaining about voice quality, then it is too late. If providers are over-provisioning or subscribers over-subscribing, then it is too costly. In either case, failure is close by.

Bob Massad is vice president marketing and strategic relationships for Telchemy.

1 - University of Massachusetts.
2 - AT&T contribution to T1A.7, May 1999.

[ Return To The October 2001 Table Of Contents ]