pyVmomi: preparing for 5.5.0-2014.1.1 our first bug fix release.

I've been heads down this week, so a short update this time.

Last week, we released pyVmomi 5.5.0_2014.1 to pypi. This week we started work on preparing RPM and Debian packages which will allow pyVmomi to be included with Linux distributions important for OpenStack work.

Having released 2014.1 I opened a milestone for 2014.1.1 as a bug fix release and identified a few smaller quick turn around tasks. We have one known bug and a few minor improvements and changes that we'll have to get in place. (Being right up against VMworld has slowed down a few tasks, but we're still moving rapidly.)

Notably the 5.5.0-2014.1.1 release will:

  • Fix a bug with sending date-time stamps to the server
  • Tweak the version number standard to fit better with pypi and RPM.
  • Standardize packaging for the pyVmomi library
  • Improve release documentation and processes for versions.
I got a bit side-tracked this week as I investigated more of the pyVmomi internals and noted that the XML documents are non-deterministically assembled. This means from one run to the next the SOAP message coming out of pyVmomi can be different in certain trivial ways.

I noted that...

  • The order of xmlns="*" attributes inside tags can occur in random order
  • The amount of whitespace in documents can vary in some situations
This meant that naive string-based comparisons done in vcrpy's built-in body request matcher can't actually compare XML document contents for logical consistency. I wrote the start of an XML based comparison system but after burning more than three days on the problem I have to stop and wonder if this is worth the effort and if it wouldn't be simpler to just figure out why pyVmomi creates random ordered XML and then fix that instead.

In the next week-or-so, we should be able to deliver a quick-turn around bug fix release with the improvements I've listed here. We're doing well for this now that we're getting more attention from last week's release. The 2014.2 is tentatively scheduled for November 15th this year.

More on these topics next week...


pyVmomi version 5.5.0-2014.1 is available.

Earlier today with help from other VMware employees and #python IRC users, I pushed the buttons necessary to release pyVmomi version 5.5.0-2014.1 to the general public. This is the first open source community built and released version of the vSphere Management API library since VMware open sourced the project in December.

This release featured contributions from 13 contributors, 10 of which were completely unaffiliated with VMware. With the exception of myself the most active contributors are not at all affiliated with VMware.

Special Thanks to...

Thanks as well to William Lam for championing our project as well.

This release marks an important step for pyVmomi, most notably the ability to use pyVmomi with vcrpy means we can do much more in-depth and thorough bug reporting for 3rd party developers. It also means that the focus of development can move from infrastructure to feature parity.

More on that next time...


On the topic of Goals and Objectives for pyVmomi

As long as I'm working on the pyVmomi project, its goal will be to become the single best way to talk to vSphere technologies. This is a broad and hard to measure thing, and with luck as VMware technologies evolve, it's also a moving target. Once we arrive at one location there will be a new version to sail toward. In order to steer accurately, we must set a course by some kind of metaphorical guiding star, and this is what objectives are meant to be.

While the goal is the destination we are sailing our metaphorical ship toward, our near-term objectives are the metaphorical stars we choose to help guide us toward that goal. Just as sailing a ship may require course corrections, which implies choosing different stars; so too might managing a project require course corrections which implies choosing different near term objectives. When we course-correct ourselves, our projects, our teams, or our careers, we are changing the objectives much as a ship navigator might choose a different guiding star at different times during a ships journey toward a distant destination.

In common parlance, the terms goal and objective are virtually synonymous but it makes sense to make a distinction between them when we're talking about conducting the daily business of building software. Being "the best" is kind of hard to measure, it implies quality, adaptiveness, and nimbleness in the library as well as a breadth of uses for the library. This requires us to choose some guiding stars with wisdom.

How do you design a library for these attributes? How do you pick the correct guiding stars?

A little philosophy

A little philosophy is necessary to guide how we go about setting our objectives to achieve our goals. Just a little philosophy is necessary, too much and we spend our days navel gazing, too little and we flail about wildly wondering why we work so hard and accomplish so little. Regular periods of slack followed by pressure are best for this. Creativity can only come when you are free to explore but productivity only really truly solidifies itself under threat of deadline.

So what are some philosophies I could hold in this case?

A few contrary points of view

I might think that audience size is the best measure of project success. That might mean I would have less priority on other aspects of the software and more on things that make the software appeal to a wide audience. In other words, a bug's priority only matters as a measure of the number of new users it can net.

Examples of projects guided by this principle to the exclusion of others might include things like PHP. The design choices in PHP were early on driven by getting broad adoption and rapidly securing mind-share. Very little time was spent thinking about the long-term impact of design decisions. The result is PHP is the most successful programming language (in its niche) of all time and one of the most hated.

I might choose to believe that 'feature count' was a better measure for project success. If I believed that cranking out code was the way to 'get stuff done', then I would probably believe that anything that got in your way of 'getting stuff done' was a waste of time. That would probably mean I would be after the largest volume of code in the shortest amount of time. More code means more function right?

The problem with this is feature creep. If you want to keep a light nimble software project that can respond quickly to changes in its environment a small modular project is best. You keep the project itself small or suffer the consequences. There's usually an 80/20 rule for features and following it can result in faster release cycles.

After years of working on software systems big and small, with tiny audiences and accidental audiences of millions, I've come to believe a few things.

In Generalities

 I feel that the competition between Xbox versus Playstation in the marketplace is a good case-study of these philosophical disagreements in action. The results have mostly played out in the marketplace for us all to study. If we take a lesson from this history we might be able to improve our own state of practice.

In 2012 it was hard to tell if there was a marketplace winner in the video game console market. The three top video game consoles of the time period had traded top position on various metrics repeatedly but by Q2 2014 there is a clear winner in market sales (only time will tell if this is a permanent state of affairs).

Sony had always invested in a complete engineering package for its consoles and frequently talked about 10 year plans with its ecosystem. Ironically, this same business strategy had failed them before. When it came to BetaMax versus VHS this same strategy of 'technical superiority' did not pay off, however, and that's a cautionary tale. The entire engineering process matters.

When building a system you have to take into account all the pertinent forces shaping its market share. These include multiple product audiences, shareholders, and customers, as well as multiple engineering considerations about how the product functions. Not the least of which includes the process by which you create the product itself.

Engineering Objectives

Audience size matters, feature count matters, and perceived quality matters. Each affects the other and each affects the total impact of your software project. Minmaxing on only one dimension or another doesn't necessarily equate to a lasting victory. So we need to find ways to incorporate all these elements into our daily choices. That's what setting objectives is all about.

Over the years, I've been thoroughly impressed at how products generated by very bad engineering can sometimes capture and dominate markets when very good engineering fails. I believe the problem comes from improperly balancing objectives. A single dimension of engineering and design has been maximized at the expense of balancing concerns. It's far too easy to make an easy to measure metric, set it as an objective, and steer the metaphorical ship by a star that has nothing to do with the goal. Such engineering produces something that is arguably beautiful yet broken.

Broken Strategy

For example, a typical strategy used to solve quality issues in software systems is to increase test coverage. Coverage is an easy number to measure. It makes nice charts and gives a wonderful feeling of progress to developers. It's also a trap.

Merely increasing code coverage does not universally improve the code in all its dimensions. In fact, improperly applied test coverage can create tightly coupled systems that are worse suited. This is perhaps the starkest lesson you can learn about successfully reaching a 100% code-coverage goal: you can end up with more technical debt not less. (I could call out certain open source projects here but I won't for brevity's sake.)

If no metric can measure this concept of tight coupling to balance the code coverage metric, then, merely measuring code coverage percentages pushes the software design in the wrong direction. Your team will optimize for coverage at the expense of other attributes. One of those attributes can actually be code quality (in that fixing a simple bug can take an inordinately long time) and flexibility (in the sense that your code can lose the ability to respond to new requirements).

I have come to believe Test Driven Development just as code coverage can also become a trap. Improperly applied, it similarly optimizes systems for unit tests which may not reflect the real design forces that the unit of code is under. These circumstances the code developed can end up very far from the intended destination just as high code coverage numbers can degrade actual quality of a software system.

Actively Compensating

Agile methodologies were intended as a tool to compensate for this dis-joint between the steering stars of objectives and the actual destination. The ability to course correct is vital. That means one set of objectives are perfect for a certain season while the same objectives might be completely wrong for another season.

To effectively use these tools (agile or otherwise) you can't fly by instrument. You need to get a feel for the real market and engineering forces at play in building your software product. These are things that require a sense of taste and refined aesthetics. You don't get these from a text book, you get them from experience.

My experience has taught me that you actually don't want to write more code you actually want to write less. You want to accomplish as much as possible while writing as little code as necessary without falling into code golf. That means that the most effective programmer may have some of the worst numbers on your leader board. Negative lines of code might be more productive than positive, fewer commits may be more profound than more. The mark of good engineering is doing a great deal with very little and that's what we strive for in software engineering.

From Philosophy to Concrete Objective

In the case of pyVmomi, we have no open sourced tests that shipped with version 5.5.0 as released from VMware's core API team. (Note: there are tests but they are fenced off from public contributors and this is a problem when it comes to getting quality contributions from the general population.) With no unit tests available it is almost impossible for a contributor to independently determine if their change positively or negatively impacts the project's goals. Some over-steer in the area of code coverage would be forgivable.

I also want to avoid solidifying library internals around dynamic package and class creation as well as internal aspects of the SOAP parser and its details. This puts me in an awkward position because the simplest most naive way to fence off units and begin busting-gut on test coverage would also force the tests to tightly couple onto the classes currently defined in what long-time pyVmomi developers refer to as the pyVmomi infrastructure.

Separation of Concerns

The fact that there is even a term 'pyVmomi infrastructure' means that there is an aspect of the library that some people need to talk about separately from some other aspect of the library. That indicates a conflation of separate concerns. This particular point in itself would be a lovely talking point for a whole different article on how software engineering becomes social engineering after a certain point. To become a highly used, trusted, and distributed library; pyVmomi should disambiguate these concerns. But, I digress.

Application of Philosophy as Strategy

I spent no less than three weeks developing the test strategy for pyVmomi that will allow us to test without boxing in the library itself. The strategy leans heavily on fixture based testing and specifically on the vcrpy library. In our case, the nuance is that our fixture needs to setup a fake socket with all the correct information in it to simulate an interaction with vCenter and/or ESXi without requiring mocked, stubbed, or simulated versions of each.

If we avoid testing directly design elements (things like the XML parser itself), and we avoid testing in isolation concerns that are deemed infrastructure versus not-infrastructure, then we are left with testing the API "surface" as exposed by pyVmomi. The unit tests call on the actual symbols we want to expose to developers and these are the API surfaces as I call them. The outermost exposed interface intended for end consumption.

The shape of these fixture-based tests are virtually identical to targeted samples of the API pyVmomi is binding. Given a large enough volume of use-cases these unit tests with fixtures might eventually encompass a body of official samples. Existing as tests means that these samples will also validate the fitness of any changes against known uses of the library.

This strategy effectively retro-fits tests onto the library without locking in design decisions that may not have had very much thought. It frees us to build use-cases and eventually fearlessly refactor the library since the tests will not tightly couple to implementation specifics and instead the tests couple to interface symbols.

Objectives Accomplished

We want pyVmomi to continue to exist long enough that it can accomplish its goal of being the best library for working with vSphere. To survive, we need the library to have a lifespan beyond Python 2. We need the library to allow contributors to objectively measure the quality and fitness of their own contributions so it attracts enough developers to evolve and spread toward its goal.

So far we've accomplished the following objectives in the up-coming release due to come out in mere days:
  • Python 3 support gives the pyVmomi library time to live and flexibility to grow
  • Fixture based tests give users and contributors confidence to develop while also...
    • avoiding design detail lock-in
    • hiding irrelevant library infrastructure details
    • providing official samples based on actual API units that will not break as the API changes
  • we also established contribution standards

Objectives to Accomplish

While we want pyVmomi community samples to evolve unrestricted and rapidly, it is also the source for requirements for the library. The samples project is separate so that it can welcome all comers with a low barrier to entry. But, it is very important as it will feed the main pyvmomi project in a very vital way. The samples become the requirements for pyVmomi.

The samples and the pyvmomi unit tests need not have a 1-to-1 relationship between sample script and test, but each sample should have a set of corollary unit tests with fixtures that give basic examples and tests for the use case illustrated in the parent sample. That means one sample might inspire a large number of unit tests.

These are some of the high level objectives to reach going forward on pyVmomi:
  • remain highly reliable and worthy of trust
    • cover all major vSphere API use cases in unit tests with fixtures
    • squash all high priority bugs rapidly which implies releasing fixes rapidly
  • reach feature parity with all competing and cooperating API bindings
To reach these new objectives we'll need to do some leg work to find way-points along the way. We may change some of our finer targeted objectives later, but these objectives help us reach the goal of being so good nobody working in this space can ignore us, being so good we deserve the title the best.

More on that in a future post...


Notes on Developing pyVmomi itself.

Most of my development work over the last few weeks has been spent finding more efficient ways to develop pyVmomi itself. The problem with networked libraries is that they sit between metal and user code.
As I've pointed out before, virtually all code written in the modern world fits in this model. There's a 'core system' under the code you are writing... that is to say the code you write orchestrates some other system of libraries, then you test the core system by writing tests that anticipate what the 'user' of your library is going to do. This is basically API development, at least in spirit.

Everyone does API development (somewhat)

The interesting thing is that in today's world virtually all development is API development if you allow for this loose interpretation of what an API is. The 'user' code may end up being a Selenium test or a unit test. The 'user' party may be yourself or your coworkers ... or they maybe the industry at large. If code gets reused then that code is essentially part of the vocabulary of an API.

I advocate that you write one test per use-case to describe each of the types of anticipated use of your API. These tests define the surface of the library and you should have no distinction between the library's tests and it's public API. That means anything private or internal should only be tested indirectly if it exists at all.

Porting pyVmomi to Python 3

In the case of porting pyVmomi from python 2.x to python 3.x (as I covered in this 3 part post) the problem has been finding a way to rapidly regression test the library on python 2.x and python 3.x interpreters. The technique I settled on was using fixtures for testing.

To build these fixtures I alluded to what I call the simulator problem. In this situation, your back-end is so complex that your stubs and/or mocks become so comparably complex that these fake components approach the complexity of the actual system you are substituting for. At this point, your mock has become a simulator. This is a lot of effort and typically non-unique for any given project. That's why you can find multiple simulators for webservers out there. It makes sense to share effort.

In the vSphere space we have VCSIM which can be used to simulate vCenter. In most situations these simulations will be good enough for writing test cases with fixtures. These test cases with fixtures do not obviate the need for integration testing, but they do shorten the feedback loop for an individual developer. This shorter feedback loop is critical.

Simulating for Success

I recorded a short video this week on how to setup VCSIM. My video is by no means definitive, but I hope it helps get you started. In the video I show how my setup works. I chose to alias my VCSIM to the fake DNS name 'vcsa' and I setup the user 'my_user' with the password 'my_password'. I make thes substitutions so that real IP, usernames, or passwords, do not leak into unit tests. The resulting setup helps me explore the API exposed on the wire by vCenter in my unit tests. Once I hit the scenario I want to develop on I metaphorically freeze it in amber by recording a fixture (as recorded in the preceding video).

Release Plans

I plan on releasing officially the next version of pyVmomi (with python 3 support) in the next 2 weeks. The goal was always to release in front of VMworld. That means we'll be flat-out on the release war-path the next few days until we have a solid, stable, and tested release to push out. Fortunately, we are very nearly there. Python 3 support itself would be a big enough win without any additional changes. The ability to record fixtures in tandem with the library is icing on the proverbial cake and it will help us as a community develop pyVmomi more reliably and uniformly.

In summary

The key to successfully navigating the change from python 2.x to python 3.x is testing, testing, and more testing. The *faster* you can get testing feedback (test, change, retest as a feedback loop) the better off you'll be. Waiting for a CI to kick back results is *far too long* to wait. You want all this to happen in a developer's local development environment. The more *automated* and *stand alone* your tests are the broader the total population of possible contributors are. The standalone test means a CI like Travis CI or Jenkins can run your tests without infrastructure. This does not obviate the need for integration testing but it will hopefully let you catch issues earlier.

If you have a python project on Python 2.x and you want to follow pyVmomi into Python 3, I hope these posts have helped! Now that we have a solid way to test for regression on the library we can start work on feature parity. Future development after the next upcoming release will be focused on building up the library's capabilities.

More on that next time...


What every developer should know about testing - part 3

For the pyVmomi folks in a hurry, this video covers the big shift in the project and the code from the video is here. This week I'll finally dig into some detail on fixture-based testing.


If you don't bother reading more from part 1 and part 2 in this series, the one thing to take away is: testing must be stand aloneautomated, and deterministic no matter what you are writing
No matter what you are doing the bulk of your effort should go toward creating ways to write as little code and as few tests as possible yet still cover the domain. This is  a much harder philosophy to follow than 'cover all the lines' but it is much more robust and meaningful. 

Good code coverage should be the outcome of good testing, not the goal. Because testing is hard and getting it wrong is bad you should write tests as sparingly as possible without sacrificing quality. Finally, unit boundaries are API and you should write tests to reflect the unit boundary which is an art in and of itself.

In part 1 I covered why your testing is bad. In part 2 I covered what are stubs, mocks, and fixture testing. In part 3 we'll get specific and cover how to build a fixture and what it represents as a programming technique.

What your code is determines what your test is.

Virtually all software built today is going to fall into this pattern:
core system -> [your code] -> user's code
Virtually everyone who writes software writes it in a space sandwiched between the software you use and the software that uses you. Interesting things happen when you move up stack far enough that 'user's code' becomes actual human interaction or you move down stack far enough that 'core system' means physics.

In most projects, however, code that has to perform actual human interaction means code written for a web app or some kind of GUI. In these special cases you need tools like Selenium or some other 'bot like Sikuli to drive your tests. In that case the far right of my diagram "User's Code" is simulated in that test framework. It's ultimately code at the end of the day and the code you write for test is something you create anticipating how that middle category "your code" is going to be used.

The more familiar case, where you are sitting between an end-developer (sometimes yourself later in the project's lifecycle) and a core-system of libraries that constitute the framework, runtime, or operating system your code is build on is what most unit test philosophies are build around. This is the world that is most comfortable for mock and stub based testing. It's the world of TDD and other methodologies. But, what happens when you start getting close to the metal when the things we need to mock are on the network or some other physical or transient infrastructure?

This is the case where I advocate the use of fixtures. Depending on what's on the other side of your code the right tool to craft a fixture will differ. I've worked in environments where fixtures had to be physical because we were testing micro-code. Most of the time these days I need a network based fixture.

I am a big fan of vcr, vcrpy, and betamax. These are tools for recording HTTP transactions to create a fixture. In generic terms, a testing fixture helps you fast forward a system to get it into the state you need for testing. In our specific purposes the fixtures replace the need for a network and related network servers.

Recording transactions

BTW: The source code for this test is available on github now.

Here's the sample interaction we want to support. It's a simple set of interactions... we login, get a list of objects, loop through them, and put things away.

    def test_basic_container_view(self):
        si = connect.SmartConnect(host='vcsa',
        atexit.register(connect.Disconnect, si)
        content = si.RetrieveContent()
        datacenter_object_view = content.viewManager.CreateContainerView(
            content.rootFolder, [vim.Datacenter], True)
        for datacenter in datacenter_object_view.view:
            datastores = datacenter.datastore

As written this test goes over the network nine times. Without a tool like vcrpy running this test in any automated way would require us to use a whole pile of cloud infrastructure or at least a smartly built simulator. This means doing special setup work to handle edge cases like faults, or large inventories. It requires the construction of an entire mockup of the production scenario we want to test. This could be automated in itself; but then, if the tool to perform such automations is literally what we're writing; how do we develop and test those automations? That's an extremely time consuming and wasteful yak shaving exercise.

Fortunately, in our project a simple one liner can help us remove the need to always have a service or simulator running somewhere for every test.

                      cassette_library_dir=fixtures_path, record_mode='once')

This decorator allows us to record the observed transactions into a YAML file for later consumption. We can't completely remove the need for a simulator or service, but we can remove the need for such beasts during test.

Modifying Recordings

Once you have the capacity to record network events into a fixture you can tamper with these recordings to produce new and unique scenarios that are otherwise hard to reproduce. That means your code can rapidly iterate through development cycles for situations that are really hard to get the core-system code on that remote server into.

You "shave the yak" once to get a VCSIM to a state that represents the scenario that you want. Then you script your interactions anticipating a use case your end-developer would want to exercise. Finally, using vcrpy decorators, you record the HTTP interactions and preserve them so that you can reproduce the use-case in an automated environment.

Once you have that fixture you can develop, regression test, and refactor fearlessly. You can synthesize new fixtures approximating use cases you might never be able to reliably achieve. And that's how you take something very chaotic an turn it into something deterministic.

This is of course predicated on the idea that you have a VCSIM setup for testing, more on that next time...


What every developer should know about testing - part 2

The short version of this post:

pyVmomi contributions from this point forward will be required to follow this pattern. They include a fixture based unit test build around vcrpy.
Note: While these posts are python specific, the techniques I advocate are not. If you happen to be working in a JVM language, then I recommend Betamax. And, if you happen to be on Ruby, then use vcr. The point of any unit test is that it should be stand alone, automated, and deterministic. These posts discuss how to create deterministic and automated tests in situations where people often argue it's impossible to create such tests.


Last week I covered some basics about unit testing. What is a unit? What do you need to test? What don't you? This week we'll dive a bit deeper and cover why a Stub, a Mock, or any other trick may not solve your problems and how we create a unit test where there's arguably no 'unit' only a client.

In this post we'll briefly recap what a unit test is. Then we'll cover stubs and mocks and why they are useful but aren't sufficient. Finally I'll cover fixtures.

If I'm in the mood, I may cover property based testing at some future date ... but that's an intermediate to advanced topic in my book.

Recap: What a Unit Test is, what it isn't

The dividing line between units is 'human hard' because it's not just 'a method is a unit' ... a method very frequently is a unit but then so is a class and there can be hidden classes and methods. I'm advocating a somewhat controversial position based on the back-lash to TDD you'll find around the blogosphere recently. My position in a nutshell is that tests are best defined at 'unit boundaries' which are necessarily abstract borders between software units (and units themselves are a term of art).

What is a good Unit Test?

There is some debate over what makes a good unit test. In general, your tests should be stand alone, automated, and deterministic (and those three words in this context have very precise meanings). This is simple enough when a unit is simple. For example, the canonical roman numeral example is easy enough to unit test.

In the roman numeral example the unit is easily isolated since the concept of a roman numeral does not require the introduction of additional units. This is the most basic tier of unit testing that every developer  must learn.

The next set of techniques are to use Fakes and Stubs, and then later maybe Mocks, and these work as long as system interactions remain relatively simple. But, these are not sufficient tools for your tool box. You have to learn one more trick before you have all the basic tools you'll need for modern development. You need to know how to build fixtures and the resources that go with them.

When to Stub

If your programming language supports interfaces and dependency injection then creating stubs will feel natural to you. If your unit uses several other objects by composition to accomplish it's work, one of the simplest things to do is write a fake version of the objects your unit under test uses.

That means you have to create an object that implements all the methods of the dependent class that your particular test unit will use. The stub will have to cover as many calls as your unit of code uses. If you call on method 'foo' you need to write something that reasonably reproduces the expected output of 'foo'. This is fine when the call number is small, but it can quickly get out of hand. Take for example that someone felt compelled to write an entire fake server in Java.

The fake HTTP server author calls their product a mock server but this is in fact a stub. You have no facility to assert call order, parameters, or the absence of calls.

When to Mock

A mock is not a stub. With a stub you have to provide a stub implementation of all possible call paths and have no facility to later go back and make assertions on call order. If you do, you probably wrote it yourself... and you've probably neglected what your day job really is.

A mock is about coding for expected calls and call order. The problem with doing this with your stub is that you will have to either be organically grow your stub into a mock to get this information (and that's a whole new project worth of complexity) or you'll have to invent a whole system of semaphores and messages to watch for these details.

With mocks, you can assert how a method was called; you can make assessments about call order; you can also assert that there is an absence of calls. This is all very important for you to be able to do in your unit tests. With these tools you can continue to assert that no matter how you evolve your intervening code the units code continues to use it's underlying API within acceptable parameters.

A concrete example is if you are consuming pyVmomi or rbVmomi as your client library bindings, you can mock the calls to the client binding library. If you observe that vim.VirtualMachine's PowerOff method is still called properly even after you refactor your own library then there's no need for a VCSIM to run your tests at the unit testing phase.

Too much of this, however, and you can't make assertions about the code's behavior outside of test. Not to mention that creating mocks and stubs are programming efforts of their own. This can lead to wonderful code coverage numbers, tightly coupled designs, and a frustrating body of work.

Enter the Network

What if you're writing a client library? On the pyVmomi project, that's what we're doing. A highly efficient and specifically tailored API binding in Python for vSphere. How can we unit test something that's intended for use with a networked appliance?


In the case of building tests for networked software, the simulator problem is a common problem. In order to deal with this it's vital to have some tool that can record your interactions then play them back. The ability to deliberately tamper with the interaction recording is vital. There are bound to be states that the complex server or simulator on the other end can't reach.

If you only use a simulator for your tests, then they are by definition not stand-alone. You have to exit the testing context and enter an administration context to build an orchestrate the situation you want to test. This can be costly and it can be a problem very similar to the simulator problem. You will inevitably have to build a complex environment that will harm the determinism of your tests.

Next week, I'll get specific and cover how to build a test with a recording, we'll have to get into details and I'll cover how I use VCSIM to create the basis for complex situations I can't even create in my environment.

More on that next time...


What every developer should know about testing - part 1

The short version of this blog post is that this week and next week on pyVmomi will be spent on radically improved testing code and processes. This testing process will become part of the commit cycle. And, it's about damn time.

Once these measures are in place over the next few days the project will be better able to absorb new commits from interested parties.


Over the last three weeks I've been working on pyVmomi's next release. If you've not been following along, it's a python client for a SOAP server and it's a code base dating back to at least Python 2.3 but it has only been in the wild as OpenSource now since December 2013. The pyVmomi library has a long internal history at VMware and has served many teams well over the years. But, it does have problems.

The OpenSource version of the library has *no* unit testing shipping with it this makes it hard for interested third parties to contribue. It's time to change that. But, it's a client library for a network service. How do you test such a beast?

In this post I will cover what unit testing is and what integration testing is and how this impacts the design choices made on libraries. This discussion is directly applicable to the long-term evolution of a library like pyVmomi and is generally applicable to Software Design. I'm bothering to write all this down because I want everyone to be on the same page and I don't particularly want to repeat myself too much. Over the years I expect to be involved with this project I expect to point at this post frequently.

The Problem

Work on pyVmomi has been rather painful. For much of it, I have had to spend vast amounts of time deep in the debugger. Testing on this library involves building a VCSIM and simulating a vCenter environment. This in turn means the creation of a suitable inventory and potentially setting up a suitable Fault to work with. This is a lot of yak shaving to get to the point you can even start considering doing development work.

The root of the specific problem

The problem in specific is that pyVmomi as a library speaks to a server. No other thing can completely simulate all the inputs, outputs, and exposed states that such a thing achieves except the big complex thing itself. This problem is routinely solved by developers in this entire cloud infrastructure space by spinning up virtual environments to create the scenarios they need.

This is a natural inclination since you have a beautiful hammer, why not nail all the bugs with this beautiful hammer? Virtualization is powerful and has transformed our industry. One day I will be an old man and tell stories of how infrastructure and development worked in the bad old days of Dot-Com Boom but this inclination is an example of the hype-cycle in full detrimental effect.

The problem in general

Because client library code development starts at the integration phase the units that end up defined by the client library programmer are inherently integrations. How do you test integrations? With integration tests. But, how do you do integration testing when the thing you are testing isn't even on your build machine? If it's a server (such as our case) you have to fall back to either a simulator or you have to stand up a whole new of your environment just for testing.

Unsurprisingly, this is fairly standard practice for every step of IaaS and PaaS development. You stand up the universe to author a new function then you retest the whole thing on a fresh copy of the universe. Then, you wash-rinse-repeat for the whole integrated system. It's so easy. It's also so very wrong. Because code that is hard to test (or completely untestable) in isolation is poorly designed. If you're defending the fact that it's tested, you're missing the point.

This isn't just a problem with the one library I'm working on now. I've seen this repeatedly in development environments of every kind at huge shops and tiny shops. You build up a pile-o-software that glues systems together and to test it you build a pile-o-infrastructure that you bring to a pile-o-state so you can validate the right calls and responses.

When you test this way (bringing a whole simulated universe into existence to test your new 'if' statement), invariably something's state gets out of sync and what do you do? You have to test the test environment to validate that you don't have false positives for your failure report, then you have to retest and you re-start the whole process which typically grows into hours. This is, frankly, an extremely expensive way to develop software.

And, for the record I've seen this in JEE, Spring, Grails, Python, Bash, Perl, C, C++, projects on Solaris, Linux, Irix, BSD, and now ESX based environments. This is not a problem unique to those crappy developers on that crappy environment. This is an intrinsic integration development problem that crops up when you routinely write code that takes big complex systems and makes them work together. It's a far too easy trap to fall into and a far too difficult of a pit to climb out of.

Unit Testing?

So the story so-far is that we have a library, maybe that library talks to things "not of this machine". Maybe, it speaks over the wire, talks to other things we can't see or directly control. These are things well outside of anything we could define as our unit of code. So if that's our fundamental unit (because what *else* is something like pyVmomi?) How the heck do we test it?


The term unit is deliberately ambiguous in this context. Did we mean class? Did we mean method? The answer is it depends. Getting the logical border of the unit is hard. It's actually human intelligence hard. It's "why AI do not yet write code" hard. Why is it hard? It's hard in the same what the making beautiful art is hard it's a fuzzy problem that requires aesthetics.

The definition of where a unit is, is hard and simultaneously critical to get right. Define the wrong unit and pay the price. This doesn't mean testing is wrong, it just means testing is a programming-hard problem. Looking for easy answers here just means you don't know what you're doing.
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
        — Brian W. Kernighan and P. J. Plauger in The Elements of Programming Style.

I mock your Mocks!

A simple answer to the problem is to Mock and Stub everything. (Mock objects are not Stubs, and you should know the difference. Edit: this is a whole topic unto itself and I will cover it separately at a later date.) The problem is when you work with a sufficiently complex interactions you will be forced to write sufficiently complex mocks and stubs. You will back your way into the simulator problem I mentioned before. In our specific case this means essentially re-inventing something like VCSIM except all in Python mock objects and that's absurd.

What are you forging, o' library author?

Consider also, where is the unit boundary when it comes to a client library? The library absolutely has a boundary at its public interfaces. If the bit-o-code you just wrote is private why did you write it? The private method is not part of the interface and so therefore it's not part of the library's unit definition. The unit in this context only makes sense as a test-first tested component if it's going to be exposed. By definition a private method isn't exposed so it's an implementation detail and we don't test our programs to make sure implementation details work. We don't test if the 'if' statement works. So where is the detail and where is the interface?

This means your test-first tests should be your surface. To develop a library that you intend on providing to people to interact with you should model sets of expected interactions. Each interaction should then be codified as a set of calls to the library.

Code as little as possible, test as little as possible

Tests are code. The mark of a good programmer is not how much code they write, but how much they can accomplish with how little. If you are following the aesthetic of minimalist code, then this attitude should also follow into your tests. Your tests should cover as many lines as is needed to validate the code and should do this as effectively as possible. Ideally, you should have an efficiency to your tests. No two tests should cover exactly the same unit. Covering a unit multiple times is effectively wasted effort.

This is a much harder philosophy and practice to follow than the lazy 'cover all the lines' strategy. It requires you to understand the functional cases your code can cover. In a rarified ideal world, this might mean you get 100% coverage anyway but the percentage isn't what we care about. You can have 100% code coverage and still have horrible comprehension of what your project even is.

Breaking down your tests to cover all the methods, even the private ones is a horrible idea. If you cover your private methods you will be tying your tests that matter to what (by defining them as private methods) you have decided are implementation details. That equals tight coupling; tight coupling is bad.

How do you test something where it provides very little function other than basic interactions with a service? How do you exercise a library that is arguably mostly private and hidden code?

Introducing Fixtures

The concept of a testing fixture is a very old testing concept. It even predates software as a thing and yet I rarely if ever see a shop using fixtures. The truly sad thing is that for most development languages fixtures are old as the hills. So, WHY are so few projects using them?

A Specific Solution: vcrpy

I reviewed several Python fixture libraries this week and was fairly well impressed with vcrpy for our purposes. The description may mention 'mocking' but function of the library is to provide you with testing fixtures at the socket level. In libraries like pyVmomi we are effectively a skin over a very complex back end web service. This 'skin' nature of ours means that a simple set of library interactions may hide dozens of network conversations.

Manually creating dozens of HTTP interaction mocks to explore a single high-level test can be so painful that you are likely to just not do it. Fortunately tools like vcrpy exist and can record your HTTP traffic. Now you can do the lazy thing and toy with your client-server a bit, record the on the wire interactions, and then later (and more importantly) edit the conversations to represent the larger API cases you want to cover.

With the recorded HTTP fixtures at our disposal we can now work with the binding in much more predictable and controlled ways.

More on that next week... (or skip to the end)