Huawei Technologies, the largest telecommunications equipment maker in the world, has joined NTAF as of January 2013!

By any measure, Huawei is a leader in networking. Huawei has over 140,000 employees, of whom 46% are in research and development (and certainly many of these employees are dedicated to test automation). Huawei’s revenues exceed $20 billion annually, and over $3 billion of that revenue is re-invested in R&D. Its products are deployed in more than 140 countries around the world, and 45 of the world’s top service providers use Huawei’s equipment in their networks.

The numbers above speak to Huawei’s size and reach in the networking industry, as well as its commitment to innovation in a very competitive market. Moreover, Huawei’s joining NTAF demonstrates that it is also committed to the quality that testing ensures, and to the efficiency that automation enables.

NTAF and its members look forward to cooperating with Huawei as we define the future of network test automation. Welcome!

This blog comes from Todd Law, vice-president of NTAF and senior product manager at Spirent Communications.

NGN Service Delivery Conference

It looks like the NGN Delivery Solutions event last week in London week was a success. This event, endorsed by the MEF and NTAF, brought together key service providers and industry experts to address the challenges of “growing demands for network capacity and secure data application delivery”.

It was good to see strong representation from NTAF at the event. NTAF member companies Spirent and Ixia were both associate sponsors of the event, and had booths in the exhibition. NTAF’s president, Keith Kidd (Verizon/Director), and NTAF’s marketing committee chair, Ameya Barve (Spirent/Product Marketing Manager) both gave presentations (see previous blog entry). One of the goals of attending the conference was to make the leading service providers in the EMEA region aware of NTAF and the work that we do – I believe this was successfully met.

Ameya reports that the quality of speakers at the conference was really high. Keith indicates that he was able to connect with a number of like-minded people who agreed on the need for standards in this space. It sounds like the NTAF presentations generated quite a lot of questions and discussions.

Finally, Ameya also participated in a panel discussion on strategies for test automation, which is an interesting topic and extremely relevant to NTAF. This topic is so important, in fact, that I will dedicate a separate blog entry to it in the coming weeks – stay tuned!

[Todd Law is vice-president of NTAF, and a senior product manager at Spirent Communications.]

Follow NTAF on Twitter @NTAForum
Join the NTAF group on LinkedIn

Looking forward to the NGN Delivery Conference!

I’m looking forward to the NGN Delivery Solutions 2012 event which will be held November 27-28 in London. Since Network Automation is one of the themes of this year’s event, it’s definitely a good fit for NTAF. In addition to co-sponsoring the event, along with the MEF, NTAF will be well represented in person by its President, Keith Kidd, and by its Marketing Vice Chair, Ameya Barve, both of whom will be speakers.

Keith will deliver opening remarks for the conference where he will introduce NTAF. He also be the speaker for the first session, where he will be addressing the added demand on CSP networks, and explaining the requirement for self-aware and optimized networks. He will also provide an update on NTAF and its standards. This should be interesting since the current standards being worked on have only been seen so far by NTAF members. Also on the first day, Ameya, will provide an update on ITO (Infrastructure Test Optimization) and its relevance to carriers and service providers. Finally, both will be part of a panel on devising and implementing an automated testing strategy, which is again very relevant to NTAF.

The event as a whole has an impressive line-up of speakers from key service providers and labs around the world. While each speaker’s experience is certainly unique, one can also detect emerging common themes, including automation. It will be great for NTAF’s representatives to connect with the individuals who are tasked with conquering these challenges, and show how NTAF’s efforts are aligned with their efforts.

[Todd Law is vice-president of NTAF, and a senior product manager at Spirent Communications]

Follow NTAF on Twitter @NTAForum
Follow the NGN Deliver Conference on Twitter #NGND @iirtransmission

Watch this space for NTAF’s impressions and feedback on this conference!

NTAF’s Annual Face-to-Face Meeting

NTAF had the most ever attendees at this year’s annual face-to-face meeting! Besides the record number of attendees, it was also impressive to see how many flew in from great distances to be there in person. I think the prizes for dedication would have to go to Torbjorn Nyman from Ericsson who flew all the way to the Bay Area from Sweden, during his vacation, followed closely by Kenneth Green from Ixia who flew in from Melbourne, Australia and Lars Friedrich from JDSU, flying in from Munich, just for the event.

The Technical Committee used the opportunity to meet in person to advance the Inventory and Resource specifications. Eric Miller of Spirent, and Chair of the NTAF Technical Committee tells me he is greatly encouraged by the progress made in locking down how attributes will be categorized and expressed in the inventory specification. Demonstrations of implementations based on those draft specifications were also given by Ixia, Spirent and Verizon, showing how the theory can be put into practice.

I was also really impressed by the stepped up participation from NTAF members Brocade and Cisco, with both companies demonstrating true leadership in multiple parts of NTAF. This overall increase in participation shows how the enthusiasm for NTAF is growing, backed by the clearly demonstrated value of the specifications, and implementations based on those specs. Next step: show that value to the world at large.

Finally, “NTAF Champion” awards were given out to Raj Kalvendi and Masi Mohammed from Cisco for their effort in promoting NTAF within their company, as well as to Brian Bonnett Verizon for his contribution in the technical committee. The NTAF President’s Award was given to Eric Miller for his leadership of the Technical Committee. All of these awards reflect the huge contribution made by these individuals.

NTAF’s Annual Face-to-Face Meeting – This Week!

NTAF’s annual face-to-face meeting is being held this week at the Spirent-Sunnyvale site. I’m looking forward to meeting NTAF colleagues in person as well as many exciting things in our packed agenda.

One of the highlights should be the show case demonstrations by members of the technical committee, which will show implementations of the inventory and resource specifications. Both specs are currently both in draft form, but are clearly marching to maturity and ratification as we can see numerous NTAF members’ actual implementations.

A further highlight will be the keynote speeches, to be given by Biju Nair from Brocade, and Pradeep Kathail from Cisco. Both companies are taking a very active role in NTAF, and we look forward to hearing their perspectives.

Finally, I think one of the “hidden gems” in the agenda will be the session, on Tuesday at 12 noon, detailing how various labs around the world are using NTAF specs as the basis for their lab design. It will be good to see how NTAF is getting traction at numerous companies, including several which are not even NTAF members (so far :-) ).

Juniper Networks Joins NTAF!

Juniper Networks, Incorporated, has formally submitted its application to join the Network Test Automation Forum! This is great news for NTAF to have such a major player in the networking space join the NTAF discussions, and is a huge endorsement of NTAF’s mission to create standards for network test automation. With over $4B in revenue and 9,000 employees in 70 countries, Juniper certainly wields huge influence in the networking industry.

But it’s not just about might – networking is as much about innovation. Juniper has consistently developed and productized some of industry’s most ground-breaking and disruptive innovations across every aspect of networking technology. I know Juniper will likewise bring that same spirit of innovation to NTAF.

Membership in NTAF will also benefit Juniper. Past experience has demonstrated that NTAF membership directly translates into benefits to its member organizations such as improved efficiency and optimized alignment with customers and suppliers. It’s a win-win-win situation.

NTAF extends a warm welcome to Juniper and looks forward to collaboration.

This blog comes from Todd Law, vice-president of NTAF and senior product manager at Spirent Communications.

Gale Technologies Joins NTAF!

 

Gale Technologies has joined the Network Test Automation Forum (NTAF).  NTAF extends a warm welcome to Gale and looks forward its active participation in the organization.

Gale Technologies provides advanced software solutions to automate, orchestrate and optimize resources.  As a pioneer of provisioning and workflow automation across networking, server, storage, and virtualization technologies, Gale enables the automated and self-service provisioning of dynamic lab, data center and cloud environments.  Gale brings 14+ years of experience in resource optimization and will support NTAF in its efforts to set standards for test automation.

Gale Techologies is headquartered in Santa Clara, California, in the heart of Silicon Valley. Fore more information, go to www.galetechnologies.com.

What is Automation?

Hello, this is Todd Law again, vice-president of NTAF and senior product manager at Spirent Communications. Last time I wrote about the various reasons people choose to automate, but this time I wanted to ask an even more fundamental question, namely, what is automation?

This question may seem strange to ask since clearly most of us have some feeling and understanding for what automation is. But if you think about it for more than a few seconds, the answer to the question is not necessarily so clear.

Perhaps let me start by stating what automation is not.

Automation is not the writing scripts or programs. There is a common perception in the networking space where automation is often equated with using the API of a product. Most test tools do indeed provide both a GUI and an API, and both are valid means to using those tools. However, we should not confuse the means for automation with automation itself. It is, in fact, possible to automate using a GUI interface, and also possible to drive many test tools by firing off Tcl commands one by one. With that important distinction being made, I do of course accept that automation is frequently achieved by writing scripts that utilize test tools’ APIs.

So having separated the means by which automation can be achieved from the idea itself, let’s now attempt to state what automation is. If you look it up in a dictionary, you will find a somewhat self-referential definition that says “automation is the act or process of getting a machine to be more automatic”. If you look up “automatic”, you will find something like, “capable of operating without control or intervention”.

If we glue those two definitions together to come up with a single, unified definition (and taking some liberties to simplify things a little) we could then define automation to be “the process of getting a machine to be more capable of operating on its own”.

It is worth noting that the definition uses the word ‘more’ which is a relative term. I like this, because it reflects the reality of automation in that the goalposts keep moving. For example, in the network test space, traffic generators started off as fairly simple frame blasters (which were themselves an extension of even more basic bit blasters). Gradually over time, however, traffic generators have added the capabilities to emulate a plethora of protocols, which are, in a sense, a form of automation, in that packetized responses are sent automatically from the tester to the device under test in response to the stimuli of specific incoming packets. Such emulations hold up one side of a protocol conversation with a counterpart in the device under test. Emulations have been a natural evolution of test tools as the ways in which the Internet is used has exploded over the past decade. Emulations provide the ability to test higher-up layers in the 7-layer OSI model of networking, and the various services that depend on underlying infrastructure.

Supporting multiple emulations is now the norm, and the expectations for automation continue to rise. Routers, and the tools used to test them, now more resemble high-end computers than simple hardware designed primarily for the narrow-purpose of forwarding packets. Routers are now capable of doing so many things that the number of test cases usually reaches into the thousands. Vendors and customers need to apply ever more automation simply to validate the vast scope of functionality, scale and performance, not to mention a continuing cascade of new features with each software release.

So how can network testing be more automatic than it is now?

Well, automation could be applied to many more parts of the network test puzzle: reservation, scheduling, inventory, topology, execution, inventory, reporting and authoring. All of these items are the stated priorities for NTAF. Moreover, since these problems frequently involve the multiple test tools as well as the devices under test, there is a clear need for standards in each of these areas to achieve the efficiencies possible via automation. Creating such standards, and driving their implementation, is the primary objective of NTAF itself. These are the problems we are working on, or will be working on, in the NTAF technical committee.

Why Automate?

Why Automate?

My name is Todd Law, and I am the Vice-President of NTAF, as well as a senior product manager at Spirent Communications.  I’m happy to have this chance to communicate through the NTAF blog.

Most of the blogs posted to date in this space have been primarily focused on NTAF’s formation and organizational progress in getting new members or standards written.  In this entry, I wanted to take a step backwards and talk about something slightly more fundamental, namely, the reasons for automating in the first place.  Put another way, what are the problems that NTAF is attempting to solve?

Of course, the “motherhood and apple pie” reasons to automate are the same classic reasons as many other business motivations:  save money, increase efficiency, increase quality, reduce time to market, and so on.  But those high-level reasons often mask some of the other underlying reasons why test engineers choose to automate.  To shed more light on this topic, I’ve compiled a list of those underlying reasons of why test engineers go down the automation path.

  • To go beyond functionality that is in a GUI – Automation is often (but not always) equated with using the API of a product.  And of course, if you write a script or a program, you can get it to do many things besides drive a piece of test equipment; for example, you can have a script also control the device under test (DUT), or have it send you an e-mail with the results.
  • To achieve a level of scale that would be difficult or impossible manually – when tests must scale up to thousands of ports or thousands of emulated protocol sessions, the configuration can become tedious and prone to error, or simply too slow.
  • To do time-dependent testing – Some test cases require an orchestration of events that are best handled via automation.  For example, in the access network, a test case might require that a number of PPP sessions come up before traffic is sent over those sessions.   Automating makes such tests repeatable.
  • To see trends over the long term – following from the previous reason, for results to mean something from run to run, the test must be applied in a consistent way.   The classic example is a regression test bed, where test managers want to see that existing functionality is not broken when a new software release comes out.  Or conversely, managers want to see where bugs are clustered with a view to identifying root causes.
  • To optimize investment in equipment – automation can be used to connect multiple traffic generators to multiple devices under test in various configurations using programmable layer 1 switches.
  • To achieve re-use – a test case written once can be re-used over and over again.  If it’s written well with a properly designed environment, it can be used in a different context.  In contrast, a script written for a very specific purpose for a very specific environment is essentially a one-time investment that gets thrown away quickly.
  • To achieve a consistent workflow – if an automation team is working with many different tools, efficiency can be gained by providing a consistent structure to automation components.  A typical strategy is to wrap up native API commands as commands which, to some degree, achieve some consistency (but unlikely full inter-operability) across vendors
  • To consolidate low-level functionality – some APIs can be very granular and low level in nature, which can mean that many steps are required to achieve basic tasks.  As with the previous point, wrappers around native API commands can be used to achieve basic tasks, reducing, for example, 20 lines of low-level code down to a single line of code.
  • To insulate against change – Most APIs do not change from release to release, because most vendors have learned that customers have invested a great deal in scripts that are expected to continue to work well into the future.  However, tool vendors new to automation may still be learning this lesson…
  • To insulate against variations in vendor implementation – The last point was about intra-vendor variation; this point is about inter-vendor variation.  In reality, all test tools are quite different, and there is only partial overlap in functionality.  For that part which is common, in some cases, automation can be used to increase re-use.
  • To create solutions consisting of multiple products working together – in many test cases it requires more than one test tool to provide complete testing.  For example, one tool may excel at (or be cheaper at) providing background traffic, while another tool may be excel at simulating complex user behaviour with realistic applications.  Automation can be used to get the two tools to work together to provide the best of both worlds.
  • To integrate into a bigger system – tools are often part of a bigger ecosystem that includes other components.  So not just other test tools or devices under test, but also test management and reporting systems, version control software, bug-tracking software, and inventory management systems are all part of the environment

As we can see, automation is motivated by many different underlying reasons!   No wonder it is a topic where confusion can reign and discussions quickly come to cross-purposes.  But these are the problems that NTAF is solving – or has already solved in part!

Proving Interoperability Throughout the Process

Automation standards groups, in the testing industry and beyond, pop up all the time. In each case there is a group of founding members that have a mission to make life easier for users of their products or solutions. It was no different when we came together to form NTAF in 2010. But NTAF is radically different from other groups, and the evidence was laid out for all to see during our first “plugfest.” My name is Kris Raney and I am the director of software development at BreakingPoint Systems, a founding member of NTAF. Today we are going to look at how far NTAF has already come in developing and promoting interoperability standards for the network test automation industry.

Proving Interoperability Throughout the Process

There are four key aspects of the NTAF effort that I think will make it a huge win, and they show why BreakingPoint works closely with the group:

  1. It moves the point of automation from a programming API to a network-based API.
  2. It makes use of the existing XMPP protocol, with XML-based messages.
  3. It requires the interfaces provided by tools to be self-describing.
  4. It’s being developed using a “show me how it works” approach, not a theoretical debate.

This last point is a defining difference for NTAF and why we don’t simply meet about these things and debate implementations. Instead, NTAF takes a proactive approach and puts these things consistently to the test.

Just a few months ago several of the NTAF members (BreakingPoint, Fanfare, Ixia, Spirent, and Verizon) put our initial draft specifications to the test. During the beginning of the week, each company chose the implementation language that made sense for them, and there were several—including TCL, Java, python, and an implementation that would work for any Windows CLR language. Then test tools that were located in two separate cities in California, two separate cities in Texas, and a server in the cloud started to interact. All of them were communicating through a variety of corporate firewalls, VPNs, and network security.

By the end of the very first day, these tools were functionally talking together, and by the time the demo came on Friday, they were working quite well together. The demonstration clearly showed multiple different combinations of cross-vendor interaction. Performing this live implementation of the specification not only showed us we were on the right track, but, more important, it also identified areas that needed improvement—and that’s the great thing about having made real implementations. The NTAF 1.0 specification isn’t being made in a theoretical vacuum, and these things will be fixed.

Very soon we will officially publish the NTAF 1.0 specification. Iterations of this specification have already been put to the test, as I mentioned, to show how NTAF is moving the test automation industry forward. And with the introduction of 1.0 we will continue to put our work to the test and proactively work together to produce a specification that is strong in its vision and in its capabilities.