SIP testing is pretty new to me.  In fact, 3 months ago, I didn’t know what SIP was and I knew little of VOIP.  Today, I work at a company who’s business is phone routing, virtual PBX, fax routing, VOIP, SIP, etc.

While I was able to set up the Front End web app automation, the development leads wanted me to also start building out a framework to test the SIP calls and the PBX software (FreeSwitch.)

Problem:

Create an automated framework that can be used to load test and functional test FreeSwitch.

Analysis:

I looked at a lot of opensource solutions, I was hoping for FS_Spec and some other ruby based solutions to work. But I had nothing but issues with them. In fact most software in this realm hasn’t been updated since ’09 or so.  In my research I did find one Open Source tool that was highly rated all around:
SIPp.

SIPp even comes pre-installed on the security distro of Linux (BackTrack.)  SIPp is pretty powerful, but it has a few drawbacks:

  • It’s not geared to functional tests. SIPp is really a load engine.
  • The reporting isn’t great.
  • Can’t run a suite of tests out of the box
  • Doesn’t have built in features to test simultaneous UAC and UAS (sender/receiver) – it expects a person to have multiple terminal sessions running, and one running a UAC and the other a UAS.

The above issues, broke down to too much human intervention.  What was desired, was something similar to the front end automation: you run a functional test, it either passes or fails, and the results are gathered for the test suite.

Solution:

I started with SIPp as the solution for doing the bulk of the tests, and then worked to build a framework around it to run the tests in an automated fashion.

What I came up with was Cucumber running SIPp functional and Load scripts.
The end result is, I just run:
cucumber
It runs all the tests (i.e. a few seconds later, it’s driving traffic to my desk phone… Desk phone is ringing… yay.) Test fails/passes are tallied and the results are output to screen and file.

In the end, Cucumber is optional.  It’s not a requirement here. For me, I’m only using Cucumber to tally the results.  I may opt to drop it all together as it has a high overhead.  The tests would be faster if just a collection of Ruby/Groovy or Scala scripts.

How it All Works:

To start it off, I had to see how the development servers work (i.e. FreeSwitch.)  So first things first, if you’re going to test SIP, I recommend installing the Virtual PBX you are using in development (Asterix, FreeSwitch, etc.) on a local QA box.

Other tools you’ll want installed on the same box are:
Wireshark
SIPp
SoftPhone for testing (I.e. X-Lite or “telephone”)
Scripting Language (Ruby/Groovy/Python/Scala Scripts)
Optionally: Cucumber
Optionally: Fully Functional Call Control with virtual numbers

I recommend only using OSX or a Linux distro for this set up.  Getting this to work on a PC is too difficult.

You’ll want to spend time with FreeSwitch and WireShark to see  how the packets are sent back and forth… and how to read the exchange.  Then you’ll see how to write the tests.

Writing the SIP tests all boils down to two steps:

  • The main test is going to be written in SIPp.  It’s a command line SIP load tool.  But you can constrain it to only drive one test, for functional testing.
  • Use a Scripting Language (or Cucumber) to run the SIPp tests you wrote.

I detail each step below…

Details: Setting Up FreeSwitch

The easiest way to do this is on OSX or Linux.  Just do a git clone from the FreeSwitch repo… you can follow the instructions here: http://wiki.freeswitch.org/wiki/Installation_and_Tips_for_Mac_OS_X
Make sure you also do the step to make your audio files.

On Mac you’ll prob need to bring in the libjpeg on my mac… to do this i used brew install libjpeg.

Configuration Setup:

This is where it gets a little tricky with FreeSwitch set up.

  1. You’ll need to know or change the default password… this is found in the vars.xml file  in /conf inside of the Freeswitch directory.  The instructions to do this are actually in the comments of the vars.xml file.
  2. Next, you’ll want to change  /conf/sip_profiles/sip_profiles/internal.xml to update the inbounx-acl is localnet.auto
  3. As a tester, you’re probably going to be  using an Internal Profile (meaning your machine’s internal IP) for testing.  Developers will probably not use that set up. For testing though, you probably will.  When pointing to your Internal/local IP, it will bind that to a specific profile called “internal.”  By default that folder is empty.  What I did, to make it quick and easy was this step:  I copied the files in the Default profile to the Internal Profile:  /conf/directory/default/* to /conf/sip_profiles/internal/*
  4. What you just copied over are actual user accounts and extensions.  They need to be modified…. Edit them individually and change the reference of “default” to “internal” and save them out.

Start Up FreeSwitch

Ok lets start it up, go to the freeswitch bin directory (i.e. /usr/local/freeswitch/bin) and type ./freeswitch

Once it’s up, type: sofia status
If everything worked out, you should see 4 rows in a table display.  One will be an Alias, and it references the Internal Profile.  This Alias should have your local IP.

You can tab for methods/commands… like this, type sofia, then hit TAB.  you’ll see all the available options.  For example: sofia global siptrace on  is a very useful command.

Register with your local FreeSwitch

You can now launch a softphone (like X-Lite) and set it up to talk to login as a default user you copied over (i.e. 1000) and use the default password (what’s in the vars.xml file) and then for domain set your local IP.

If everything is set up correctly, it should REGISTER on the FreeSwitch.

If you have a second softphone (i.e. “telephone”), you can also register as a different user (i.e. 1001) and then call the other user (i.e. 1000).

Once you make a call you should start seeing the FreeSwitch server display a lot of activity.  If you  have: sofia global siptrace on running, it will categorize the events.  You should see events like: INVITE, ACK, BYE.

Details: Using Wireshark

Wireshark Installation Overview

Wireshark is incredibly useful here.  If you are installing Wireshark on a MAC, you’ll need to set up X11… on Mountain Lion, you’ll need to use XQuartz.  Once installed, you can then bind Wireshark to XQuartz/X11 – I wont go into all the setup details here, but you’ll need to restart your Mac for the changes to take effect (or logout/login.)
When Wireshark is up and running, you’ll want to listen to the interface: lo0 for  your local traffic.

What can Wireshark show me?

What can’t it show you…. it shows everything.  It not only records all the packets, this tool also has a whole Telephony menu.  From there you can build VOIP diagrams of the call flows. This is very useful in seeing what is sent and expected back for each test.  Wireshark can also copy the RTP stream!  That means it can hear the audio  you send, and verify the audio sent (say a recorded wav) is what’s received!

Using Wireshark to Generate a Flow Diagram

Go ahead and launch XQuartz, then Wireshark.
Now, click on the interface lo0 and start recording.
Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)
Make a call from one softphone to the other
Stop the Wireshark recording.
At first you’ll see a dump of all packets captured on that interface.
Now, go to the Telephony menu, and click “VOIP.”  That will load only the VOIP related packets.
Click the button, Select All.
Now click Make Flow Diagram.
From that window, save it out.  In OSX the initial display usually doesn’t render well. but once you save it out, it looks great.
This flow diagram will show you everything going on in a call, and removes all the uneeded data, making this more human readable.

Using Wireshark to listen/verify the audio of a Call

Go ahead and launch XQuartz, then Wireshark.
Now, click on the interface lo0 and start recording.
Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)
Make a call from one softphone to the other
Stop the Wireshark recording.
Similar to before, click the Telephony menu.
Click the RTP sub menu and the “Show Streams” option.
You’ll see packets here on each line item. click through on one, and click “Analyze.”
You’ll get the audio of the call/transmission.

Using Wireshark Programmatically

My goal is to use it in automation testing.  To do this, I’ll be calling the Wireshark utilities from the command line and getting back results in the command line interface.

Details: SIPp

Now lets look at SIPp.  If you’re new to SIP, you can think of SIPp as a command line version of JMeter.  It was designed to drive lots of traffic (hundreds or thousands of calls a min) to a PBX.
However, SIPp can be set up to run just one call through a functional scenario.
To install SIPp, just follow the instructions over at: http://sipp.sourceforge.net/

Pre-Installed Tests

SIPp has built in tests.  These include UAC, UAS, UAC_PCAP, REGEX, and more.  The whole list is off their main site.  If you don’t know what UAC or UAS is…. you should probably read up on SIP.  But to just basically summarize it, SIP is a P2P system.  At any given time, one user is both the server and client.  They are sending data and receiving.  so UAC and UAS is a client server scenario.
If you just want to see SIPp work, you can type:
./sipp uac [your local ip]
There’s no Params, so it will run it’s default load for the UAC test.
If you want to see what the built in UAC test is actually doing, just do this:
./sipp -sd uac >> [filename]

Working with Parameters and SIPp

There’s some sites out there with useful cheat sheets for SIPp… here’s one such site:
That site also has some tests they wrote, which you can download and see how they work.  They also have tests that utilize CSV files which contain a set of users you want in your load.
For my tests, I use a lot of functional tests, so I don’t want to slam 10,000 phone calls against the server. Instead I want to run one call, one time, to do something specific:
./sipp -s [phone number configured on call control to redirect to my desk phone] -m 1 -l 1 -recv_timeout 6000 -sf [path to my test xml file] [IP of our integration / uat freeswitch]
So here’s what those parameters are saying:
Run SIPP against the QA Integration/UAT env. and dial that number configured in Call Control (-s means service, which could be a extension, phone number, etc.) -m tells it to run once -l tells it to run only one test at a time.  -recv_timeout 6000 says to run this test for only 6 seconds.  Finally -sf is the path to my test…. and of course i end it with the IP of the FS box.

Writing your Own SIPp tests

I wont go into too much detail here.  I’ll save it for a future blog.  But to start with, you can take an existing built in test, and export it using the -sd option I mention above, or you can review some tests other people are writing out there and have put on github or elsewhere.

Automating All This

Now to automate this.
Initially I used straight Ruby for this… but then I decided having Cucumber run the tests, gives me the added benefit of compiling test results. I’m on the fence if that’s necessary at this point or not.
Basically, I found it too hard to use an existing API with what we use here in house as our PBX.  I tried SIPr, FS_Switch and a variety of other tools. Most of these haven’t been touched in many years and have known bugs flagged against them.  Rather then fight with that, I choose to use SIPp.
SIPp is an industry standard, but it doesn’t have an open API.  To automate this, I use SIPp as is, and build the shell around it.
It’s really very simple:
Create a script to drive all the SIPp tests.
Assert via REGEX the output for expectations.
Mark pass / fail.
So I’m basically doing the core work in SIPp.  I write out my test, with the parameters to call it effectively.
Then I use Cucumber (you could use straight Ruby or Groovy or Scala, Python, etc.) to do this:
Given /^a SIP call is made to another SIP account$/ do
   @calling = %x{/Users/brainwarner/sipp-3.3/sipp -s 1000 -m 1 -l 1 -recv_timeout 6000 -sf /Users/brianwarner/sipp-3.3/calling.xml 10.98.1.1}
end
Then /^a SIP 180 ringing should return$/ do
   assert @calling.include?(“180”)
end
The above is a simple set of two steps in a step definition file.  Basically it’s saying ok, I’m going to execute sipp with my sipp test, against the specific IP of my UAT server and I’m using those parameters to make sure it’s just one test, run once and ends after 6 seconds.
Then I’m calling a Ruby method “include” which is like a REGEX, to search the output that’s in that class variable @calling for the specific ID I expect for calling.
Why not use REGEX? well every regex I tried, passed even when it failed. Initially i would search the output for a invalid value (i.e. 777iwin!), and it would always pass, no matter how I wrote the regex!  I ended up using this method because it actually works well.
Simple. But it works.

Future Steps In Automation

So this is the Framework.
You could easily see how that simple test could be transformed to something more impressive:
%x{….} calling one UAS user session
%x{…} calling another session in UAC test mode… and then have them talk to each other.
Since we’re calling 3rd party command line utilities, you can also run the command line wireshark! meaning at test time you could use Wireshark to sniff packets, or verify RTP audio…. now that’s cool.

9 thoughts on “Automating SIP Testing with SIPp, FreeSwitch and Cucumber”

  1. Hi
    I wanted to automate my Sipp Test cases in a similar way as you have done. Can you suggest me how to do the following?

    Say my test case is the following:
    Inivte–>
    180ringing < --
    200 Ok < --
    ACK ->

    How do I check if any of the expected messages didnt arrive? If so, I would like to write to a file, that in a certain iteration it failed to get correct response

    1. Hi John, sorry for the delay in replying… If you use SIPP, then sipp will verify that you got all the expected messages. This is based on the SIPP Scenario file, which you would construct just like you described. It’s an XML file that defines the invite, and what to expect as a response 180, 183, etc. If anything is missed, it will fail.

      Since it’s been awhile since you posted, let me know if you still need help. I have a few examples of SIPP scenario files I can send you.

    2. BTW I have a much better method now for a fully automated test harness… I use SIPP and wrote an app in Grails to handle all the test runs. It actually monitors our production environment. Every 10min it kicks off a job that calls about 30 different numbers and validates each one terminated correctly. I have some guides on this here in my blog.

  2. Hi Brian

    Thanks for the writeup!

    I am in a similar situation. I am new to VOIP and have to setup an automated test environment for FreeSwitch.
    Would you be able to help me with some pointers to:
    – VOIP basics
    – FreeSwitch modules
    – Test environment around FreeSwitch
    – Automating this test environment

    Any inputs will be of great help.

    Thank you.
    Sid

    1. Hi Sid,

      End to End
      well I worked with the director of technology here and we sat down and thought it through from an end to end test. In other words, we exercised Freeswitch through the entire sip call. I tried various sip clients and stuck with SIPP. using SIPP I would do a command line call to one of our sip proxies, which works in hand with FreeSwitch. I had to work with an engineer so that we figured out the exact SIPP XML to send to terminate the call correctly (i.e. we expect 100, 200OK, etc.)

      The trick then was, to make sure we know the call terminates correctly. We would send a call to a proxy (which talked with FS) and then back to our test env where it terminated (test env. -> sip proxy[FS behind the scenes here] -> test env.) Our choice was to stand up a termination system on the test machine (early on i used SIPP UAS (the sipp server to terminate incoming calls) later on I used Freeswitch on the test machine to terminate the calls (sipp on test env. -> prod proxy[FS behind scenes] -> FS on test env to term. call.) A fake DNIS was set up in the system so that it redirected back to my test server. So if my test server is ip: 111.111.111.111 I would send the fake dnis back like 5554446666@111.111.111.111… basically I’m sending a call to a sip proxy, that passes it through the system and ultimately sends it back to my test box, where a Server is waiting (i.e. a local Freeswitch install.) The local freeswitch had a dial plan set up so that it would accept this fake dnis. Now we had our first route.

      This route tests FreeSwitch as well as the proxy. I set up an automation of all the proxies and iterated over them.

      CARRIER TESTS
      The future tests were with carriers… We set up unique numbers for each carrier and ran it in a similar fashion as the proxy tests. We send a call out some carrier, which then has a DNIS configured to use a spec carrier and come back to my test box… That way we test incoming calls.

      Outbound carriers were similarly tested as above.

      AUTOMATION
      Once you can run these tests manually, they can be scripted into an automation suite… and the results saved to a database. Once you have results in a database, you can stand up a web page to display the results… and then you have a production monitoring system of your VOIP system.

      My choice of automation was to have a web application stood up that would run a sipp command to our flows mentioned above. SIPP would return an exit code (0, 1 or 255) depending if the call terminated without error, with errors, or timed out. My web framework made use of a chron type job that ran the SIPP calls every X min. Each result was saved to a database. In my case, I used the Grails framework. But any would work… Django, Rails, etc. So I have Grails set up a bunch of chron jobs that kicked off methods that executed SIPP commands. These SIPP commands returned a value of 0, 1 or 255. These values were stored. I created a presentation layer that was a GSP page that pulled the results from the database and would show the DNIS | Proxy | Carrier tested and the result.

      In case of a failure, we had business logic to email special teams. I kept a log of the failures in a table in the database, so that when a call failed in the past and then started passing again, I would know to send a Fixed email to the special teams.

      It’s hard to go into a lot of detail here, but that’s the gist of it the automation. You can create a db table with all your boxes, carriers, etc. you want to test, and just iterate over the db results… that way you can update more carriers or proxies without a code update.

      MY METHODOLOGY
      You’ll notice that I didn’t choose to test specific API’s of Freeswitch in isolation. Instead I chose to use an end to end test that completed a full call flow. For my needs, that worked out well. We wanted a monitor of user experiences.

      If you have specific Freeswitch API’s you need to test, I unfortunately haven’t ventured into those waters so I have no concrete advice. What I would do is research what API’s you need and how to access them … then try a scripting language to open a connection to FS to that API and see what response I get and flush the script out till I get the result I’m looking for.

      I hope that helps a bit.

    2. AUDIO QUALITY TESTS
      One more thing… I didn’t get into the audio quality tests in my previous comment… What I posted before was just about call termination. For codec and audio quality… that got more complex. I made use of a c algorithm called PESQ. If you search for it, you’ll find it’s download. On my test server I compiled PESQ and set up a test… how PESQ works is that it gives a value from 1 to 5 of how one audio file is similar to another. So if I had a clean audio file called phone_call.wav and I tested it against g711_phone_call.wav (which might be the same audio file that came through a sip call using the G711 codec.)

      In this case the framework was altered a bit… I used a script that made use of tshark (terminal wireshark) to record packets right before I make the sipp call. Sipp calls the proxy, goes to FS/call control, set up to play an audio file back… Let’s say the audio file is a woman reading numbers: “10, 22, 5, 33” The sipp ends, and my script would stop the packet capture using tshark – outputting a pcap file… then using Bash or python (i used both methods), I would rebuild the audio from the RTP stream in the pcap. Then my script used sox (linux tool to convert audio files) to convert the raw audio we got from the pcap to a wav… and finally the script ran PESQ to compare our original audio file of the woman reading those numbers, to the recorded audio file from the packet capture on the call… We might get a number like 3.5 The value is slightly arbitrary… but you’ll see when there is static on the line or packet loss, that the numeric value will drop. When the audio file sounds clean the value is higher.

      I saved those values to a database and used a similar automation web app to keep and display the results. The whole process may sound complex, but it really isn’t… if you search my site for the term PESQ, you should see a few walk throughs (one with BASH and one with Python) to do this.

      SECURITY
      Finally one last thing I would recommend is to build out security test for your proxies/fs. Check exploit-db.com and search it for “Freeswitch” or your proxy technologies. See what exploits are out in the wild and try and run them against your FS in your test environments.

      I recommend using Kali Linux for security testing, and install it into a Virtual machine in your lab/laptop/desktop… then manually run tests ad hoc to make sure your systems are not vulnerable to exploit…. if you can, try and get the Offensive-Security.com training for Penetration Testing with Kali Linux… its good and one of the machines in the lab is an Asterisk SipServer, that you are taught to exploit. You get to see firsthand how something like that can be exploited. From that you can extend to Freeswitch.

      Again.. I hope this helps!

  3. Hi Brian,
    Thank you for the article. Your articles on freeswitch and SIPp have been very useful. I have been trying to do performance testing on the freeswitch based system. I have tested SIPp uac mode and i am able to call freeswitch through SIPp. For my setup i would like to use SIPp as the UAS. I am having issues setting up SIP PROFILE on freeswitch so I can originate call which lands on SIPp. Your help is appreciated. Thank you in advance.

    1. Yes I have almost the same setup here, although I have Freeswitch handle the termination (as the server.)

      You’re doing a full load test of FS… so why not use sipp to directly hit the FreeSwitch? I.e. the FreeSwitch has a dialplan with a fake dnis like 9591919912@[your freeswitch ip]. then you could use sipp to drive load to that fake dnis that’s handled by freeswitch. All the while you could monitor CPU/Mem on the FS box for performance overhad metrics.

      SIPp is a single threaded application, and it tends to hold that thread on a core of the CPU.  To get higher volume you may have to invoke SIPp on multiple threads or machines to the target FreeSwitch.  You can monitor which core / cpu that SIPp is running from using htop (if on Linux.)

      We also do testing where we setup audio playback from the SIPp client, using PCAP audio files.  This allows us to test the overhead performance on transcoding.  Like PCMU to OPUS, etc. That’s really where performance seems to hit with FS (IMO.)  FS can handle a lot of simultaneous calls w/o much overhead problems, but passing audio through you’ll start to see performance hits… and once you get into transcoding, and you might see the server performance take an even bigger CPU hit.

      Anyway, thanks for reaching out. I really appreciate your feedback and comments.

Leave a Reply

Your email address will not be published. Required fields are marked *