Archive for the ‘end-to-end (E2E)’ Category

The Need for IPTV End to End Solution Integration – Learned the Hard Way

Thursday, June 25th, 2009

End-to-end solution integration is not a luxury. In their eagerness to capture innovators, both the service providers that led the early IPTV service launches and the early adopters learned this the hard way. Now, more than a decade later, the same types of deployment and operational issues still exist. However, given the renewed industry focus on end user quality of experience that covers not only received video quality, but ease of use and provisioning and support, operators now realize they must build quality end-to-end integration into the deployment business case.

Introduction
It has been more than a decade since service providers first tried bringing IPTV services to their customers. But despite advances in technology and tools, many of the issues that derailed those initial efforts continue to pose challenges. That’s because the delivery of reliable, revenue-generating IPTV services is an inherently complex undertaking. It spans a variety of deployment and operational requirements—from video quality and ease of use to provisioning and support. Further, the end-user environment has changed since those early attempts: Broadband penetration now approaches 50% of U.S. households, and equipment like encoders, set-top boxes and DVD- and HD-based screens are more sophisticated. Providing a viable commercial IPTV service means addressing all of these issues in a comprehensive manner that’s based on lessons learned the hard way: from realworld IPTV deployments.

That’s where end-to-end (E2E) solution integration (SI) comes in. E2E SI makes use of proven, repeatable processes and methodologies to ensure the smooth implementation, operation and maintenance of IPTV services. In short, it addresses every part of the IPTV project—from the consultation and planning stages through rollout and beyond. E2E SI can be thought of as a “golden thread” that elegantly ties together all deployment lifecycle phases, weaving project work packages and workflows together for maximum efficiency and making IPTV deployments and enhancements predictable, reliable and successful.

However, because E2E SI is so wide-ranging, it typically requires the help of a third party. While many vertical solution vendors can address specific components of IPTV integration, only professional integrators with experience and expertise in every aspect of IPTV can give service providers the thorough, all-encompassing help that they need. Such a partner can draw on knowledge gained through hard-won, first-hand experience in IPTV implementation to make the right decisions every step of the way. This is a vital consideration given the competitiveness of today’s market, where E2E SI isn’t a luxury—but a necessity.

Click here to enlarge Figure 1


The Growth of Solution Integration

The importance of E2E SI can’t be underestimated, as evidenced by the problems encountered by early would-be providers of IPTV. From 1995 to 2001, few if any IPTV projects involved outside solution integrators. That’s because operators drew on their experience with older technologies in delivering a single service over a single network; they felt their in-house teams, familiar with infrastructure, could simply select best-of-breed products for each piece of the service, connect the components, and begin delivery of IPTV to their customers. Third-party vendors were sometimes brought in to help with specific products, but for the most part there was no such thing as an integrator that could oversee all aspects of the project end-to-end.

While many operators were nonetheless able to roll out IPTV services by their planned launch dates, few achieved meaningful success. In most cases, these failures could be traced to post launch problems with usability and quality of service. The problem was that operators typically approached IPTV as just another telco project—not a consumer entertainment service.

Consequently, they encountered significant difficulties. For one thing, they were unable to isolate, mitigate and rectify various unexpected issues, like video freezes. For another, they discovered that fully resolving these issues might take as long as a month or more. Further, they realized they had not implemented any efficient mechanism for provisioning and continued rollout. Predictably, subscribers terminated their services, sometimes after just a few days. Having learned these lessons the hard way, operators realized that E2E SI would be essential in ensuring successful IPTV deployments. That led to a rapid maturation of the video-related SI industry: Today, more than 90% of video services projects involving cable, IPTV and satellite include some form of SI, ranging from a specifically focused consultancy to the more common situation in which E2E SI is led by an established integrator.

Click here to enlarge Figure 2


What to Look for in a Solution Integrator

In today’s market, video service deployment is typically one part of a triple- or quadruple-play service introduction strategy. Consequently, it requires a solution integrator that has experience and expertise across a wide range of relevant technologies, applications, platforms and processes.

Unfortunately, not all solution integrators can deliver this critical combination of knowledge and skills. Sometimes, even when an operator puts its faith in a third-party integrator, the service deployment is unsuccessful; usually the problems can be traced to such things as improper service definition, insufficient architectural design, inadequate guidance in equipment selection, and the inability to tie all the components together via a “golden thread” approach that will ensure quality, usability and continued performance.

Clearly, that makes the evaluation of a prospective E2E solution integrator a vital undertaking. Following are the key questions that operators should keep in mind when considering a solution integrator:

  • Does the solution integrator have extensive experience in video service deployments worldwide, and can it draw on this experience and the lessons learned to the fullest extent?
  • Does it have professional services project managers, video experts, system engineers and all the additional personnel resources needed in taking on an SI project?
  • Can it smoothly incorporate third-party management into the project?
  • Does it implement and follow a proven SI methodology that it uses worldwide, one that covers all relevant processes and procedures, aligns with the service provider’s business structure, ensures maximum efficiency, and helps address unexpected issues?
  • Can it manage and mitigate the risk associated with complex, multi-component projects?
  • Does it have the ability to perform lab validation via its own infrastructure and dedicated validation personnel.
  • Can it bring video services solutions to commercial launch rapidly without sacrificing the end-user’s required quality of experience?

It’s essential that operators seek out solution integrators that can deliver on each of these items. Choosing such a partner not only greatly improves the likelihood of successful IPTV deployment; it also helps the service provider free up resources to focus on other key projects.

Solution Integration: A Real-World Scenario

The importance of the preceding considerations can be clearly seen in the following actual IPTV deployment. The operator in this case seemed to be moving smoothly toward a successful launch: pre-commercial tests were promising; personnel were proficient in the use of the middleware, head-end gear, network infrastructure equipment, and CPE already on site; and more than 50 consultants from major telecommunications and networking companies were on hand to provide both technical and business consulting services.

Yet, what seemed on the surface to be a sure thing was anything but. The operator learned when commercial deployment got under way that there was no single point of communication; its in-house personnel and vendor consultants were not working in a synchronized manner. Thus, there was no assurance that the complex layers of network infrastructure, middleware and other applications and devices would work together seamlessly.

What the operator didn’t have at this vital stage was what it needed most: Full transparency into every aspect of the project, so that if something went wrong it had a way to fix it and move forward.

So the operator decided to bring in a solution integrator. After making a high-level analysis of the project status, the players, and the technical solutions, the solution integrator identified a number of issues that needed attention. The operator then asked the integrator to assume overall management in bringing the solution to market. At that point the solution integrator took the following actions:

  • It reorganized and focused the project organization, including the definition of all work packages and staffing them with relevant, qualified people from the operator, the various vendors, and its own consultants
  • It re-scoped project work packages and obtained operator sign-off
  • It consolidated and issued a new project plan.

The new methodology ensured that all project work packages were identified and structured under the project organization. Each work package was then placed within a lifecycle as defined by the overall methodology. A snapshot of some of the work packages within the lifecycle defined by the SI methodology is shown in Figure 3.

This was the first time the operator had a clear picture of the magnitude of the required actions. The solution integrator also provided the operator with an analysis of all the consolidated project information. In addition, it created a detailed statement of what was feasible within the required timeframe, providing clear visibility into the process as well as the solution blocks and how they interconnected and correlated. The solution integrator also described the risks associated with deviating from the plan, adding to the operator’s understanding of the project’s trade-offs.

Click here to enlarge Figure 3


The operator was then able to make decisions based on the following proposed options: raise the project risk and keep the existing scope, or reduce the scope and the associated requirements. The operator opted for the latter, so as to reduce the risk that various project elements would not work together and undermine the stability of the E2E platform.

The solution integrator then applied the methodology to each work package and guided the efforts of the work package teams. The result was that none of the potential problems the solution integrator discovered surfaced during the implementation—and the deployment was a success.

Learning from Experience

While the preceding scenario illustrates some of the issues that can impact an IPTV rollout, solution integrators with extensive experience on numerous IPTV projects can draw on a wide range of lessons learned and apply them to the project at hand. With this wealth of knowledge at its disposal, an established solution integrator can offer valuable advice to operators as they embark on their IPTV service initiative.

Top line Advice

1. IPTV rollout incorporates both change management and technical deployment, so matching people and processes with appropriate technical skills is a must.

2. The number of people needed by the solution integrator and the operator will differ greatly for each phase of the project.

3. Service definition is a key element of solution design, the CapEx budget, and overall business case.

4. IPTV is by definition a highly complex program initiative, thus requiring very stringent change management discipline to be successful.

5. Some project timelines cannot be compressed, so sufficient time for completion has to be allocated. This often means that the entire organization will have to adhere to new organizational and operational processes.

6. IPTV is not just another data application but a network and service transformation that other services will leverage.

Deeper Into the Details

Solution integrators have also through experience learned just how different IPTV is from other service implementations. These are the IPTV-specific issues that they can educate operators about

1) IPTV SOLUTION DEPLOYMENT REQUIRES A COMPLETELY DIFFERENT SET OF CAPABILITIES, WHICH USUALLY SHOW UP AS KNOWLEDGE GAPS IN THE OPERATOR’S INTERNAL ORGANIZATION

Telecom has expanded its horizons beyond bandwidth and pipes and now offers numerous products and services that are marketed using various pricing schemes, packages and discounts. But IPTV adds another dimension. It’s all about the user experience and it requires the ability to market not just TV channels and movies, but also one-time attractions like sporting events or rock concerts. Content and programming are key concerns; operators may thus want to consider hiring personnel from the TV industry who have experience in this area, and who know how to negotiate with content providers and form long-term relationships.

2) FOCUS ON THE USER EXPERIENCE, NOT OBJECTIVE MEASUREMENTS

Pinpointing packet loss is all well and good, but how are viewers seeing things on their end? If images are freezing, they won’t care how many packets have dropped — they just want their programming delivered without delay. That makes “quality of experience” the key measurement. Operators should thus be prepared to perform a pre-qualification of the line, followed by pre-provisioning. If they go on-site, the operator’s installers must be trained to guide customers through the new service offering and be able to respond in a professional and friendly way to customer requests and questions. In short, consistency, reliability, quality, and ease of use are some of the key consumer factors.

3) TAKE A NEW APPROACH TO NETWORK PLANNING

With IPTV, traffic peaks don’t follow traditional seasonal trends (like spikes during end-of-the-year holiday season), but rather appear during specific events like sports championships, season finales or blockbuster concerts. Network planning should thus reflect these realities.

4) ADJUST TO THE REQUIREMENTS OF DELIVERING MULTIPLE TYPES OF TRAFFIC SIMULTANEOUSLY

Online broadcasting involves different types of traffic traveling to the consumer in parallel, each with its own set of specific set of requirements. These include linear channels over multicast, voice-on-demand over unicast, SVOD and related granting mechanisms, pay-per-view, text and overlay graphics for notification and/or advertisement, interactive applications, and required back-end applications.

In some ways, though, these recommendations only scratch the surface of IPTV-specific issues. Operators must also have a well-tuned OSS/BSS in place. They need to address familiar matters like network security and not-so-familiar ones like digital rights management. They have to tailor the help-desk — both in terms of technology and personnel training — to reflect the specific IPTV service issues that might arise.

And because IPTV isn’t just an add-on service, operators need an organization dedicated to the service, especially with regards to content management. This might include a content acquisition team, a video operations team, and possibly a video editing team to handle ratings, censorship, and other matters. Audio specialists will be needed to fine-tune quality. Installers will be needed for home deployments. And marketing communications specialists will be needed to run dedicated marketing campaigns. Further, the electronic programming guide must be clear and error-free—and easily updatable to reflect changes. Likewise, the user interface needs to be reliable and easy to use. Finally, operators will need to have a plan for implementing future capabilities when TV, mobile phones and the Internet all come together. These are the kind of detailed recommendations that only a solution integrator — with extensive experience gained through lessons learned the hard way—is able to deliver.

How the Solution Integrator Works with the Operator

True E2E SI is a comprehensive, all-encompassing undertaking that involves close cooperation between the integrator and the operator as they tackle complex processes and technical challenges. Following is a list of important considerations — from pre-launch to post-deployment — for operators as they engage a solution integrator to help with their IPTV implementations.

Executive Buy-In

The support of top management may seem like an obvious requirement, but sometimes it is lacking. Without it, the IPTV implementation can be undermined by endless discussion and inaction. Upfront buy-in by the top decision-making executives is a necessity, so that the solution integrator can proceed with the overall mission of helping the operator roll out its IPTV service.

Coordination, Communication, and Continuity

It is the role of the solution integrator to coordinate the efforts of all personnel and functions to ensure that timeframes are adhered to and goals are met. This applies to all of the operator’s diverse internal population: marketing, product management, engineering, operations, core transport, DSL, help-desk, roll-out, IS, finance, testing, head-end, quality and legal. On some occasions it might be appropriate to bring all these functions together in one room. This could be especially helpful at the beginning of the project — so as to properly define roles, deliverables, dependencies and requirements—and during the weeks leading up to a launch, so as to deal with the last outstanding issues.

Obviously, building a cohesive working group also means speaking the same language and understanding the local culture. It is the solution integrator’s responsibility to provide subject matter and technology experts who know how to communicate with and train the operator’s local personnel.

Further, because even a basic IPTV rollout can take nine to 12 months, continuity across functions and personnel is essential. Over this long a period, not everything can be recorded—and many of the important details are in the heads of key personnel. Therefore, continuity is key.

Close Supplier Relationship

The solution integrator must also establish a close working relationship with suppliers. For primary suppliers—such as middleware vendors—daily calls and weekly or monthly face-to-face meetings are advisable; they will provide opportunity to share updates on roadmaps, delays, and other issues. One of the advantages of working with a large solution integrator is that its experience and purchasing power can be leveraged to optimally meet the operator’s requirements.

Well-Structured Project Workflows

Project workflows that are well thought out and carefully crafted allow implementation teams to focus on the many parts of the rollout. Work packages constructed around specific workflows require experts and personnel continuity from the start to the end of the project. The work packages must contain clear deliverables and deadlines.

Standardized Solutions

Unless the operator has a compelling reason to adopt a fully customized solution, a solution integrator should advise starting with a standard, well-understood solution feature set. The technical, organizational and marketing challenges are already big enough without adding another layer of complexity.

However, the solution integrator should help the operator understand that service definition is not necessarily final at launch. Budgeting and resource planning should reflect the potential need to meet new requirements as service features and market conditions evolve. In general, an iterative approach to implementation works best; it is better to move in small steps and evolve gradually to meet market demands.

Network and Solution Essentials

A solution integrator should at the project’s outset help the operator understand network requirements for supporting new IPTV services. IPTV makes for considerable capacity and infrastructure demands, which if unmet undercut the customer’s quality of experience. Solution integrators and operators need to learn upfront whether the network is robust enough to keep delays, freezes and other performance glitches from occurring. Flagging load and performance testing and measurement upfront is thus essential. Because this is very difficult to execute in the field, it is advisable to perform network load and performance tests in a protected lab environment. This allows references to be developed that can be compared to the field situation.

Video Monitoring Tools

Ensuring end-to-end video quality requires operators and solution integrators to implement monitoring tools and determine where in the network to deploy them. In IPTV streams, there are several places where problems can arise, so correlating issues through a comprehensive test and measurement system is essential. Many operators have already embarked on the development of a dedicated video operations center function to gain control over end-to-end video quality.

Rapid Hardware and Software Evolution

Because of the rapid evolution of IPTV technology, a fixed-price contract spanning several years is not the right approach. Today’s vendors sometimes can issue new hardware generations every six months. Requirements and applications change on frequent basis. Established solution integrators can help operators take an iterative approach in dealing with (and leveraging) rapidly changing hardware and software.

Customer Training Prior to User Acceptance Testing

The solution integrator should ensure that operator personnel have sufficient solution training before the implementation moves into the user acceptance testing phase. IPTV is an organic and complex technology, and it may pose special challenges for operator personnel more accustomed to working with conventional networks.

Operational and Support Needs

After the launch, a solution integrator can work with the operator to operate, manage, and maintain the solution. The concept of a “network release” becomes critical when considering how to plan and deploy upgrades based on multiple end-to-end solution components and dependencies. This means consolidating relevant and needed enhancements, patches or new applications that can be deployed at a single time.

The key to successful E2E SI is the definition of focused project workflows that ensure that issues usually encountered after the first launch are in fact anticipated, identified and resolved at the very beginning of the project cycle. Orchestrating these project workflows so that they support one another mutually throughout the entire project is essential

Conclusion

It has been more than a decade since the first major attempts to implement IPTV met with failure. But even today, many IPTV projects run into the same problems that doomed those early efforts. Complexity, lack of coordination and the inability to smoothly integrate processes and procedures keep operators from capitalizing on the opportunities IPTV services present.

End-to-end solution integration (E2E SI), however, can help operators achieve their IPTV goals. A comprehensive, all-encompassing approach to service deployment, E2E SI covers every stage of the IPTV project, from planning and design, through rollout, and beyond. As such it can be seen as a “golden thread” linking all aspects of the IPTV implementation together, so that processes, technology and personnel are in sync with one another, thus ensuring successful deployment.

In short, E2E SI is a necessity. But it’s also a demanding undertaking that many operators cannot accomplish on their own. They need the help of established, third-party solution integrators that — with hard-won experience and extensive expertise—can leverage a proven methodology for applying the lessons they’ve learned to the project at hand. With a trusted partner as their solution integrator, operators can be sure their E2E SI efforts will result in an IPTV service solution that helps them gain new customers, increase revenues and continue to compete in a challenging market.