Notes on Developing pyVmomi itself.

Most of my development work over the last few weeks has been spent finding more efficient ways to develop pyVmomi itself. The problem with networked libraries is that they sit between metal and user code.

As I've pointed out before, virtually all code written in the modern world fits in this model. There's a 'core system' under the code you are writing... that is to say the code you write orchestrates some other system of libraries, then you test the core system by writing tests that anticipate what the 'user' of your library is going to do. This is basically API development, at least in spirit.

Everyone does API development (somewhat)

The interesting thing is that in today's world virtually all development is API development if you allow for this loose interpretation of what an API is. The 'user' code may end up being a Selenium test or a unit test. The 'user' party may be yourself or your coworkers ... or they maybe the industry at large. If code gets reused then that code is essentially part of the vocabulary of an API.

I advocate that you write one test per use-case to describe each of the types of anticipated use of your API. These tests define the surface of the library and you should have no distinction between the library's tests and it's public API. That means anything private or internal should only be tested indirectly if it exists at all.

Porting pyVmomi to Python 3

In the case of porting pyVmomi from python 2.x to python 3.x (as I covered in this 3 part post) the problem has been finding a way to rapidly regression test the library on python 2.x and python 3.x interpreters. The technique I settled on was using fixtures for testing.

To build these fixtures I alluded to what I call the simulator problem. In this situation, your back-end is so complex that your stubs and/or mocks become so comparably complex that these fake components approach the complexity of the actual system you are substituting for. At this point, your mock has become a simulator. This is a lot of effort and typically non-unique for any given project. That's why you can find multiple simulators for webservers out there. It makes sense to share effort.

In the vSphere space we have VCSIM which can be used to simulate vCenter. In most situations these simulations will be good enough for writing test cases with fixtures. These test cases with fixtures do not obviate the need for integration testing, but they do shorten the feedback loop for an individual developer. This shorter feedback loop is critical.

Simulating for Success

I recorded a short video this week on how to setup VCSIM. My video is by no means definitive, but I hope it helps get you started. In the video I show how my setup works. I chose to alias my VCSIM to the fake DNS name 'vcsa' and I setup the user 'my_user' with the password 'my_password'. I make thes substitutions so that real IP, usernames, or passwords, do not leak into unit tests. The resulting setup helps me explore the API exposed on the wire by vCenter in my unit tests. Once I hit the scenario I want to develop on I metaphorically freeze it in amber by recording a fixture (as recorded in the preceding video).

Release Plans

I plan on releasing officially the next version of pyVmomi (with python 3 support) in the next 2 weeks. The goal was always to release in front of VMworld. That means we'll be flat-out on the release war-path the next few days until we have a solid, stable, and tested release to push out. Fortunately, we are very nearly there. Python 3 support itself would be a big enough win without any additional changes. The ability to record fixtures in tandem with the library is icing on the proverbial cake and it will help us as a community develop pyVmomi more reliably and uniformly.

In summary

The key to successfully navigating the change from python 2.x to python 3.x is testing, testing, and more testing. The *faster* you can get testing feedback (test, change, retest as a feedback loop) the better off you'll be. Waiting for a CI to kick back results is *far too long* to wait. You want all this to happen in a developer's local development environment. The more *automated* and *stand alone* your tests are the broader the total population of possible contributors are. The standalone test means a CI like Travis CI or Jenkins can run your tests without infrastructure. This does not obviate the need for integration testing but it will hopefully let you catch issues earlier.

If you have a python project on Python 2.x and you want to follow pyVmomi into Python 3, I hope these posts have helped! Now that we have a solid way to test for regression on the library we can start work on feature parity. Future development after the next upcoming release will be focused on building up the library's capabilities.

More on that next time...

What every developer should know about testing - part 3

For the pyVmomi folks in a hurry, this video covers the big shift in the project and the code from the video is here. This week I'll finally dig into some detail on fixture-based testing.

Summary

If you don't bother reading more from part 1 and part 2 in this series, the one thing to take away is: testing must be stand alone, automated, and deterministic no matter what you are writing. 

No matter what you are doing the bulk of your effort should go toward creating ways to write as little code and as few tests as possible yet still cover the domain. This is  a much harder philosophy to follow than 'cover all the lines' but it is much more robust and meaningful. 

Good code coverage should be the outcome of good testing, not the goal. Because testing is hard and getting it wrong is bad you should write tests as sparingly as possible without sacrificing quality. Finally, unit boundaries are API and you should write tests to reflect the unit boundary which is an art in and of itself.

In part 1 I covered why your testing is bad. In part 2 I covered what are stubs, mocks, and fixture testing. In part 3 we'll get specific and cover how to build a fixture and what it represents as a programming technique.

What your code is determines what your test is.

Virtually all software built today is going to fall into this pattern:

Virtually everyone who writes software writes it in a space sandwiched between the software you use and the software that uses you. Interesting things happen when you move up stack far enough that 'user's code' becomes actual human interaction or you move down stack far enough that 'core system' means physics.

In most projects, however, code that has to perform actual human interaction means code written for a web app or some kind of GUI. In these special cases you need tools like Selenium or some other 'bot like Sikuli to drive your tests. In that case the far right of my diagram "User's Code" is simulated in that test framework. It's ultimately code at the end of the day and the code you write for test is something you create anticipating how that middle category "your code" is going to be used.

The more familiar case, where you are sitting between an end-developer (sometimes yourself later in the project's lifecycle) and a core-system of libraries that constitute the framework, runtime, or operating system your code is build on is what most unit test philosophies are build around. This is the world that is most comfortable for mock and stub based testing. It's the world of TDD and other methodologies. But, what happens when you start getting close to the metal when the things we need to mock are on the network or some other physical or transient infrastructure?

This is the case where I advocate the use of fixtures. Depending on what's on the other side of your code the right tool to craft a fixture will differ. I've worked in environments where fixtures had to be physical because we were testing micro-code. Most of the time these days I need a network based fixture.

I am a big fan of vcr, vcrpy, and betamax. These are tools for recording HTTP transactions to create a fixture. In generic terms, a testing fixture helps you fast forward a system to get it into the state you need for testing. In our specific purposes the fixtures replace the need for a network and related network servers.

Recording transactions

BTW: The source code for this test is available on github now.

Here's the sample interaction we want to support. It's a simple set of interactions... we login, get a list of objects, loop through them, and put things away.

    def test_basic_container_view(self):
        si = connect.SmartConnect(host='vcsa',
                                  user='my_user',
                                  pwd='my_password')
        atexit.register(connect.Disconnect, si)
 
        content = si.RetrieveContent()
 
        datacenter_object_view = content.viewManager.CreateContainerView(
            content.rootFolder, [vim.Datacenter], True)
 
        for datacenter in datacenter_object_view.view:
            datastores = datacenter.datastore
            pprint(datastores)
 
        datacenter_object_view.Destroy()

As written this test goes over the network nine times. Without a tool like vcrpy running this test in any automated way would require us to use a whole pile of cloud infrastructure or at least a smartly built simulator. This means doing special setup work to handle edge cases like faults, or large inventories. It requires the construction of an entire mockup of the production scenario we want to test. This could be automated in itself; but then, if the tool to perform such automations is literally what we're writing; how do we develop and test those automations? That's an extremely time consuming and wasteful yak shaving exercise.

Fortunately, in our project a simple one liner can help us remove the need to always have a service or simulator running somewhere for every test.

    @vcr.use_cassette('basic_container_view.yaml',
                      cassette_library_dir=fixtures_path, record_mode='once')

This decorator allows us to record the observed transactions into a YAML file for later consumption. We can't completely remove the need for a simulator or service, but we can remove the need for such beasts during test.

Modifying Recordings

Once you have the capacity to record network events into a fixture you can tamper with these recordings to produce new and unique scenarios that are otherwise hard to reproduce. That means your code can rapidly iterate through development cycles for situations that are really hard to get the core-system code on that remote server into.

You "shave the yak" once to get a VCSIM to a state that represents the scenario that you want. Then you script your interactions anticipating a use case your end-developer would want to exercise. Finally, using vcrpy decorators, you record the HTTP interactions and preserve them so that you can reproduce the use-case in an automated environment.

Once you have that fixture you can develop, regression test, and refactor fearlessly. You can synthesize new fixtures approximating use cases you might never be able to reliably achieve. And that's how you take something very chaotic an turn it into something deterministic.

This is of course predicated on the idea that you have a VCSIM setup for testing, more on that next time...

What every developer should know about testing - part 2

The short version of this post:

pyVmomi contributions from this point forward will be required to follow this pattern. They include a fixture based unit test build around vcrpy.

Note: While these posts are python specific, the techniques I advocate are not. If you happen to be working in a JVM language, then I recommend Betamax. And, if you happen to be on Ruby, then use vcr. The point of any unit test is that it should be stand alone, automated, and deterministic. These posts discuss how to create deterministic and automated tests in situations where people often argue it's impossible to create such tests.

Overview

Last week I covered some basics about unit testing. What is a unit? What do you need to test? What don't you? This week we'll dive a bit deeper and cover why a Stub, a Mock, or any other trick may not solve your problems and how we create a unit test where there's arguably no 'unit' only a client.

In this post we'll briefly recap what a unit test is. Then we'll cover stubs and mocks and why they are useful but aren't sufficient. Finally I'll cover fixtures.

If I'm in the mood, I may cover property based testing at some future date ... but that's an intermediate to advanced topic in my book.

Recap: What a Unit Test is, what it isn't

The dividing line between units is 'human hard' because it's not just 'a method is a unit' ... a method very frequently is a unit but then so is a class and there can be hidden classes and methods. I'm advocating a somewhat controversial position based on the back-lash to TDD you'll find around the blogosphere recently. My position in a nutshell is that tests are best defined at 'unit boundaries' which are necessarily abstract borders between software units (and units themselves are a term of art).

What is a good Unit Test?

There is some debate over what makes a good unit test. In general, your tests should be stand alone, automated, and deterministic (and those three words in this context have very precise meanings). This is simple enough when a unit is simple. For example, the canonical roman numeral example is easy enough to unit test.

In the roman numeral example the unit is easily isolated since the concept of a roman numeral does not require the introduction of additional units. This is the most basic tier of unit testing that every developer  must learn.

The next set of techniques are to use Fakes and Stubs, and then later maybe Mocks, and these work as long as system interactions remain relatively simple. But, these are not sufficient tools for your tool box. You have to learn one more trick before you have all the basic tools you'll need for modern development. You need to know how to build fixtures and the resources that go with them.

When to Stub

If your programming language supports interfaces and dependency injection then creating stubs will feel natural to you. If your unit uses several other objects by composition to accomplish it's work, one of the simplest things to do is write a fake version of the objects your unit under test uses.

That means you have to create an object that implements all the methods of the dependent class that your particular test unit will use. The stub will have to cover as many calls as your unit of code uses. If you call on method 'foo' you need to write something that reasonably reproduces the expected output of 'foo'. This is fine when the call number is small, but it can quickly get out of hand. Take for example that someone felt compelled to write an entire fake server in Java.

The fake HTTP server author calls their product a mock server but this is in fact a stub. You have no facility to assert call order, parameters, or the absence of calls.

When to Mock

A mock is not a stub. With a stub you have to provide a stub implementation of all possible call paths and have no facility to later go back and make assertions on call order. If you do, you probably wrote it yourself... and you've probably neglected what your day job really is.

A mock is about coding for expected calls and call order. The problem with doing this with your stub is that you will have to either be organically grow your stub into a mock to get this information (and that's a whole new project worth of complexity) or you'll have to invent a whole system of semaphores and messages to watch for these details.

With mocks, you can assert how a method was called; you can make assessments about call order; you can also assert that there is an absence of calls. This is all very important for you to be able to do in your unit tests. With these tools you can continue to assert that no matter how you evolve your intervening code the units code continues to use it's underlying API within acceptable parameters.

A concrete example is if you are consuming pyVmomi or rbVmomi as your client library bindings, you can mock the calls to the client binding library. If you observe that vim.VirtualMachine's PowerOff method is still called properly even after you refactor your own library then there's no need for a VCSIM to run your tests at the unit testing phase.

Too much of this, however, and you can't make assertions about the code's behavior outside of test. Not to mention that creating mocks and stubs are programming efforts of their own. This can lead to wonderful code coverage numbers, tightly coupled designs, and a frustrating body of work.

Enter the Network

What if you're writing a client library? On the pyVmomi project, that's what we're doing. A highly efficient and specifically tailored API binding in Python for vSphere. How can we unit test something that's intended for use with a networked appliance?

Fixtures

In the case of building tests for networked software, the simulator problem is a common problem. In order to deal with this it's vital to have some tool that can record your interactions then play them back. The ability to deliberately tamper with the interaction recording is vital. There are bound to be states that the complex server or simulator on the other end can't reach.

If you only use a simulator for your tests, then they are by definition not stand-alone. You have to exit the testing context and enter an administration context to build an orchestrate the situation you want to test. This can be costly and it can be a problem very similar to the simulator problem. You will inevitably have to build a complex environment that will harm the determinism of your tests.

Next week, I'll get specific and cover how to build a test with a recording, we'll have to get into details and I'll cover how I use VCSIM to create the basis for complex situations I can't even create in my environment.


More on that next time...