Worldwide Radio Broadcaster - Network Performance Improvements
Our client, a major worldwide Radio broadcaster was looking to improve performance on their networks and especially in their Audio over IP Network (AoIP). As preparation for launching new channels for their UK audiences continued, they had begun to experience performance problems within their key audio application. Achieving desired performance levels was being held back by a number of interfering network and application issues. These were having a disproportionate effect and in particular were driving some packet loss.
The Challenge
The client's IT and network staff had identified a number of issues that they felt might be behind the packet loss problem. However both IT and the network department were fully engaged in bringing new radio channels and their infrastructure online.
The requirement was to assist the client's staff in tracking down and identifying the source of problems. Identifying the likely impact of these problems, repairing or remediating those thought to have the greatest impact. Going as far as possible in eradicating errors and creating a clean operational environment within the network and systems supporting the applications.
The Solution
Layer3 began by adding an extra layer of telemetry to provide a deeper understanding of current operation. We installed a copy of our CustardWare network reconnaissance application which allowed us to monitor specific areas and to look at detailed operation such as packet drops, network protocol functions and QoS queuing parameters.
This allowed us to clear some of the minor issues that were producing error messages and to clean up operation so the behaviour of other failing systems was more easily visible. We could then see that there were issues with the re-assembly of fragmented UDP packets in their central VMWare stack, and a specialised multicast-based application was performing poorly in the face of the installed QoS Policies.
We were thus able to suggest mitigation strategies to the on-site support teams which they successfully implemented to deal with the issues.
Layer3 provided a friendly and workable solution that cost less than £2,500 per month. This could provide small teams to tightly focus on fast problem solving.
Issues Resolved
Analysis of Custardware graphs of QoS performance coupled with detailed network packet captures allowed us to identify the following areas of concern
- QoS tuning needed to be improved - especially to mitigate the bad behaviour of an Audio over IP control application which responds badly when handling large (fragmented) control packets
- Improper IGMP Querier behaviour by two Switches.
- Aberrant behaviour of some network device drivers on Windows PCs leading to occasional all-zeros MACs Addresses, all-zeros source IP Addresses, incorrectly DSCP 0 tagged audio traffic, and intermittent stealing of the default routers' MAC address by broken edge devices.
- Consistency of Syslog/NTP setup.
- Port Chanel load-balancing.
Outcomes
- The broadcaster was able to successfully roll out their additional channels without any upgrade of network equipment.
- Launch of channels was on time there would have been a loss of £850,000 in lost advertising revenue if the launch had fallen behind schedule.
- Technical staff gained a deeper understanding of network, systems, protocols and applications.
- The analysis and remediation work cleared a number of minor problems that were impacting day to day work for some key employees.
- Increased stability and uptime of key applications reduced support load on support teams.
- The business gained confidence in the technology and systems in use.
How Can We Help?
In essence we simply maximise uptime by identifying problems and poor performance, then work towards removing problems and applying permanent fixes.So no matter what challenges you face, we can help you overcome them!
Please contact us for a free initial discussion. Call 0203 805 7795 or email sales@layer3.co.uk