pyVmomi and VMware’s internal R&D Innovation Offsite for 2015

I've been invited to speak at VMware’s internal R&D Innovation Offsite (RADIO) on the topic of my experiences with pyVmomi. This is a rare honor, and I'm very pleased even just to be invited. For the pyVmomi community this potentially is a big boost to our visibility.

I expect to be sharing the stage with some heavy hitting presenters with big ideas to share in a fast-paced TED Talk like format. So, I'm spending a good deal of effort making sure I make the most of the opportunity.

I only have so many hours in a work week. That extra effort in one area means other areas slow down a bit. For you, that translates into a bit less predictability in when things are going to happen on pyVmomi in the near term. Right now, the plan of action looks something like this:
  • Roll out a beta of v5.5.0.2015.1 by the end of Q2 2015 as a bug fix only release
  • Begin alpha work of v6.0.0.2015.2 for alpha release by the end of Q3 2015
The links above are to the issue tracker for issues targeted to their respective releases and these lists may change in real time as the community works through the release. Naturally, these are aspirational statements at the moment so the plan may have to change as we roll forward. But, I think this flexibility is the strength of our process on this project.

I have also resolved to spend more time mentoring new developers. That kind of work may cut into my actual code output and slow things down a bit when it comes to things that I personally code. The return for that slow down will be that we will grow a larger core group of developers who will be highly skilled and capable of executing projects like this one in the broader community. That's something the world just needs more of in general, so I think it's a good investment.

In a nutshell in 2015, I want to focus on mentoring more developers who can help drive this type of work forward. If you want to help and don't know how to get started, reach out to me and I'll do my best to connect you to the right places depending on your skill level. If you just want to start cutting code, feel free to jump into the samples project and start slinging away.

Either way, welcome to the party... it's just getting started.


retrospective and prospectives

You're busy, I'm busy, so here's a Summary of the post's points.
  • I'm focusing on pyVmomi for a Q2 release (hopefully earlier)
  • I'll be assisting in on boarding rbVmomi contributors from inside VMware in Q2 (I hope, no solid commitments there yet.)
  • Both rbVmomi and pyVmomi will have samples of the vSphere 6.0 API some time in 2015


If we look back at the amazing output we saw in 2014, it was driven by a clear targeted focus. We knew that pyVmomi needed to be future proof. It had to be able to survive into the Python 3 era. A quick survey of the pyVmomi code base reveals that the project has existed since 2008. (I've personally been doing integrations with vSphere on some level since that time without knowledge that pyVmomi existed.) This was a big step for VMware to Open Source the project.

Today pyVmomi is a fork of that internal project. The idea of a vibrant and collaborative Open Source community improving vSphere is a novel idea. It's one that I hope will catch on. That said, 2008 was a long time ago in internet time. The future is coming and it's called vSphere 6.0

Much of this goes for rbVmomi too. I have a mental cycle that I use when approaching a project. When I enter a new project there is always a period of observation and orientation before I make decisions and start acting. When my attention is divided that forces me to lengthen my input phase before I make a decision and then act.

On the pyVmomi project I've come to trust the judgement of a select handful of contributors. If they like the submission I'm more likely to accept it. I've not built relationships like that on rbVmomi. It takes time to form that kind of trust and I'm actively soliciting VMware staff who are deeply experienced Rubyists for help in forming those.


Most VMware followers will know the company had a big launch in February. Some of what I've been doing is coming along side folks who needed a fresh pair of eyes just before release. The launch of vSphere 6.0 is a big step toward a new future for VMware. There are many new features and new ways to integrate, converge, and innovate with these offerings. The new vSphere 6.0 API collectively point the way to how VMware of the future will deliver API, I am eager to hear people's reactions and to help adjust these offerings.

I'm currently juggling pyVmomi, rbVmomi, and a separate academic research project that I made commitments to. I've been talking to folks about delegating some of this work. Each of these projects deserves my full attention right now, but I can only divide myself so much. In the case of pyVmomi and rbVmomi your participation is vital and I view my job in this area as supporting the community's desire to get things done on vSphere.

I'm not the primary mover here, you are. That includes work on getting pyVmomi into the hands of python developers however they manage libraries. That is all work initiated by and driven by folks like you. As I dig deeper into what needs to happen in 2015, I would welcome your participation. If you want to take on one of the fixes that need to happen, let me know via GitHub or via IRC.

I've also gotten an early look at the vSphere 6.0 SDK and I know that there are a few new API features that developers will find immensely useful. There is also a grand SDK vision to be worked out and its early days to be talking about that yet. Remember that pyVmomi is currently a Fling from VMware so it's living in slightly different product support space than most VMware products do.

Project Goals

Both rbVmomi and pyVmomi will have code samples available that will show off the new vSphere 6.0 API and I can talk about that more as we come up on the release. In the mean time I'll be firming up what we're going to deliver in our rapid succession 1st and 2nd releases of 2015.


Projects status for Q4 2014

I'm presently allocated to several other vmware projects that are running up against hard deadlines in January 2015. Because of these resource constraints, I've been forced to remove the time lines from my other projects including the public GitHub facing projects. Both pyVmomi and rbVmomi presently do not have time lines for release anymore. I can't control my velocity on these projects, therefore I can't reliably forecast release time lines.

I will revisit both pyVmomi and rbVmomi in February of 2015 and update the projects respective time lines when I gain clarity on what can be done and how long it will take.

It is important to note that these particular versions of these projects are not officially supported VMware products and do not have purchasable support plans at this time.  I will continue advocating inside VMware for the inclusion of these Open Source projects as part of the over-all product suite and API plans, however, at the moment these are Flings under the VMware labs umbrella.

I have truly relished working on pyVmomi in this very open manner and I'm actively looking for ways to make the best parts of this project parts of every project that I work on. I have already presented to various groups the wonderful lessons we've learned from both pyVmomi and rbVmomi. I feel some of my audiences were truly enlightened by my frank assessments of the lives of both these projects.

That reaction to these experiments encourages me in my pursuit of finding better ways to make impactful software. What every developer should be after is impact, not volume. We should pursue not creating more code, nor more refined code, but creating code that makes a bigger impact with lesser effort. By the term 'impact' I mean various permutations of the words 'useful', 'efficient', 'important', 'delightful', and 'powerful'.

I will post more updates in the first three months of 2015 as I have them. Progress will be quite slow between now and then as I will have to address a number of other duties and deadlines. Happy New Year to you all and I look forward to getting back down to business in 2015.


This week on pyVmomi/rbVmomi

Last week, I took on a high priority task for VMware so my work on pyVmomi and rbVmomi had to move down in priority for me. I'll continue to post regular updates on progress but the velocity will have slowed on both project until I have cleared my higher priority tasks.


  • I have slid out the next release until November 15th
  • I have begun learning how to upstream the github changes into the VMware git repos for rbVmomi
    • we will need to determine the upstreaming process' impact on the public repo
    • I need to get any merge requests peer reviewed by VMware internal folks before proceeding


  • I have triaged 2 new bugs
    • issue 176 is an interesting issue we've seen directly impact integrations like OpenStack, SaltStack, and Ansible. I would like to find a clean answer.
    • issue 180 is a problem I've heard reported before, it has to do with a newer vCenter server speaking to an older ESXi host. The current work-around is to ask the programmer to select a lowest acceptable version number (ie: 5.1) and use that namespace exclusively. The bug reporter has done a good job of breaking down the problem. We need to have a solution that will work robustly and impact client code minimally. EDIT: This issue is only known to impact ESXCLI use at this time
  • I wrote a simple sample for an IRC user today describing how to navigate a virtual machine's hardware devices. The sample is rather straight forward and I was glad to get to write a sample after having neglecting samples for so long
  • I  revisited pyVmomi-tools and started a testing refactor using VCRpy

Thursdays will be my rbVmomi/pyVmomi day and I'll alternate between the projects. This week I spent most of my time on pyVmomi so next week I'll try to focus on rbVmomi.

More next week...


Software Engineering at scale is Social Engineering

This week, I spent time at VMware's HQ in Silicon Valley. I gave a number of talks on my lessons learned with pyVmomi and rbVmomi and gave several calls to action. This week, I'd like to share one of the more profound lessons from the project. It has to do with how tools impact social interactions.

The first social networks were the ones that built software. The Software Development Life Cycle (SDLC) were social actions of developers directed, enabled, and enforced through the software tools that they used. If you take this view of creating software, then you it's natural to view any conflicts between departments or teams of developers as an opportunity to invent and apply new technology.

Let's examine something really interesting and profound about Travis CI that may not even have been intentional. The way that Travis CI shifts responsibilities through a tiny change in how builds are specified. The magic is in that .travis.yaml file and any build system that emulated the .travis.yaml file strategy would exhibit the same magical effects (even without using yaml). To understand the profundity of the change, let's revisit the bad-old-days of Developer Operations for a moment.

On the left, our developer builds some code and designs a build system to build their module. On the right an operations person builds a CI server to match. This process has some issues with it already. First off, the operations person must be told what the developer needs in a CI server. This requires a conversation and frequently its one that the operations person isn't directly incentivized to pay much attention to since this isn't likely to be their primary job.

The Operations person may even feel that their cooperation with the developer is a gift. That comes from the fact that the developer likely isn't in the Operations persons direct line of report and their collaboration is a cross-functional team. What is the Operations person really on the hook for?

Let's assume the previous gift from the Operations person lead to success. There's a working build. Over time the build works and breaks as you would expect. The developer is quite happy to fix their code to repair the build problems. Until one day... the build breaks permanently.

First the developer will try their normal work-around tricks. These won't work. The build environment is operationally broken now because the developer changed a library dependency or some other build criteria. Or, perhaps, our developer is spinning their wheels this effort is pointless, because something in the data center is broken.

What's the problem?

  • if the fault is in the build service we waste time debugging code that isn't broken
  • the developer is helpless to do their job without direct intervention from operations
  • the operations people have no view into when a build is broken because of a service problem
  • Operations may exercise their right to change how the server is running and break the build without notice to the developers

In the most extreme cases, the Operations person is going to be bothered during a crisis. This will result in loss of developer time as they wait on Operations (at best), or it will result in the Developer demanding more services from an already busy operations crew what the Operations person thought was a gift. And, often the timing is terrible. Both parties are blocked from doing their jobs effectively.

I've known at least one developer who would handle this by screaming "This is your job! Do your damn job!" Not only did this not get people to do their "damn jobs" but it also made them defensive viewing development as an enemy to defend against. This developer was not pleasant to be around.

Demanding people do their jobs and threatening their livelihoods doesn't exactly setup a safe place for innovation. Not all of us are high-priced-prostitutes and instead aspire to other motivations. (By the way, if you think I'm talking about you ... I'm not. I'm really really not.) If you find yourself here, start asking how to keep people from feeling like they need pitch-forks and torches.

Let's tear down this conflict. What are the facts in this scenario?

  • The developer specifies the build's operational requirements
    • building software is their job
    • running build servers is not their job
  • The operations person runs the build server
    • running workloads, servers, and Infrastructure is their job
    • deciding how software is built is not their job
Does this about cover it?

With these facts consider the following:
  • The developer 
    • can't build software without asking for permission from operations
  • Operations can't create the build server on their own
    • ops has to stop a dev and ask for direction
This is an issue of Separation of Concerns ... only on social systems, not software systems. The concern of specifying a build in most CI systems rests with a separate configuration. In Jenkins this configuration is commonly run through a control panel. That panel is not version controlled, it's not tested code, yet it is effectively a software artifact.

Enter .travis.yaml

The Travis CI build setup is minimal. Point a Travis job at a repository and branch. The configuration work typically carried by an Operations person (those long configuration forms) is carried by the .travis.yaml file. This file is versioned with the code-base itself. That means, when the developer exercises their right to change their own build, the yaml file goes through version control and we can see if things break because of the change in build specification.

We now have two options as the developer:
  1. modify our build to fit current expectations
  2. request a new build environment that fits our new needs
The .travis.yaml file does not eliminate conflict or eliminate the problem, it frames the issue in a different manner. 

Using a build specification by configuration file model means, the build configuration is now part of the project it describes. We have placed the control and authority over that process in the correct place ... with the person who's responsibility it is to maintain it. So one possible conflict is removed, the developer no longer has to beg and wait on Operations to donate their time to allow the developers to do their jobs.

The other part of this new social reality is, if we build a .travis.yaml file that our CI can't support we know it will fail. What's the conversation we're going to have now? The developer now has highlighted boldly that Operations is prepared to give certain services but not others and to ask for new services is to ask someone to go beyond their normal duties.

Operations persons now equipped with the right back-end tools can see what kinds of build requests are coming in, what their resource consumption and performance profiles look like... and without having to interfere with the developer's jobs they can tune their environment to cope. Build configuration by document now becomes a workload specification that an IaaS or a PaaS operator can judge for themselves whether or not they can service.

The conflict and struggle is still there, but now it's framed properly. We accomplish this through software. And, this ability to shape social interaction through software tools is why Software Engineering with more than a handful of people who know each other personally is really Social Engineering.


This week in pyVmomi & rbVmomi


  • version 1.8.2 is live on rubygems
  • release notes are here


I'm in Silicon Valley next week for an unrelated project so it will likely be a slow week on the *Vmomi projects. If folks would like to meet up I'm trying to organize a hack-night for projects in pyVmomi or rbVmomi while I'm in town. Contact me for details.


This week in pyVmomi & rbVmomi


Not much motion this week. I've spent the majority of the week focused on coming up to speed on rbVmomi. Of note we did have a bug report of sockets getting left open and I'll be spending quality time debugging that next week.

  • The next major revision of pyVmomi is slipped to December now in light of my new work commitments. I will keep github up to date on how these projects are going.
  • Disconnect doesn't close socket is this a pervasive bug or a quirk? TBD I'll carve time from my schedule next week to attempt to reproduce and resolve the bug as either a quirk or create a fix. The fix will go in 2014.2 at minimum but might be bad enough to cut 2014.1.2 if needed.
  • We need some more education and evolution on tasks. I've shelved my work on pyvmomi-tools temporarily while I come up to speed on rbVmomi. But, tasks are one of those areas I feel the raw API is too rich for new developers to grasp right away. Some syntax sugar should help.
  • Currently stalled on my desktop is an effort to introduce an SMS "plugin" (plugin isn't the right word but it's close) which I wanted to introduce through pyvmomi-tools. I'll ask for review inside VMware and from key thought leaders in the python and pyVmomi communities before shipping it to pypi ... more on this as we have time.
  • We changed version conventions yet again. On pyVmomi we wanted the version number to communicate freshness and compatibility. So the year-dot-quarter method was used. Then releases moved to every six months so it's 1st half and 2nd half now. Then we found out `-` means something in pypi so now it's just a `.` character. Finally we have and to indicate 1st half of 2014 bug fix 1 is compatible with vSphere 5.5.0. Part of this is we still don't know precisely how pyVmomi github and pyVmomi vSphere will interact on a code-base level going forward. That's TBD be the vSphere teams in charge of that. The admittedly complex interaction between these two things 


Just coming up to speed on the modern state of Ruby gem management since it's been a number of years since I've had to work with them. I'm also moving cautiously since there's a standing development practice and tradition in place that I need to be mindful of.

  • A prerelease for rbVmomi 1.8.2 is available on rubygems now please test!
  • A regular release for 1.8.2 will happen October 1st barring any negative testing results
  • A 1.8.3 release is tentatively set to got to pre-release October 17th (or so) depending on how well we can manage the process of upstreaming pull requests and this is very tentative. Note: this is a change I had intended to hold the release 2 weeks for myself to gain familiarity first. Instead I like this pre-release, delay for 3rd party testing/review, then release strategy and I'll likely adopt it as my normal operating style from now on.
  • Long term planning will have to happen for a 1.9.1 release but I'll not comment much on that until I know what my long term relationship to the project looks like.
  • I opened a low priority research ticket for [higher performance XML](https://github.com/vmware/rbvmomi/issues/57) which seems to be a minor scalability problem rbVmomi faces. My initial research is in the ticket. It's a help wanted item so feedback is much appreciated.
  • NOTE: any item I tag 'help wanted' is an area I feel a 3rd party can either easily help with (usually also tagged 'low hanging fruit') or an area we can't really move on without outside help.
  • NOTE: any item I tag 'VMware staff assistance needed' is something that I feel a 3rd party should probably avoid working on since the issue is fuzzy and bleeds into VMware internal technical details, decisions, or other issues muddy the ticket such that it's not going to move without someone (usually me) bridging the Open Source and VMware internal engineering communities.

General Status

My Friday posts will probably take this two part pyvmomi then rbvmomi flavor as long as I'm bridging both communities. Sorry if that confuses things but that's a function of Conway's law and at the moment the Open Source to VMware and back communications conduit is basically me for both projects so you have to expect a bit of that.

Thanks for your support! The following freenode IRC channels will at least have my IRC 'bot in them #pyvmomi , #pyvmomi-dev , #rbvmomi , #rbvmomi-dev , #govmomi , #govmomi-dev and I'll eventually see IRC messages in those channels through my 'bots. The -dev channels are for advanced programming topics while the other channels will be open for newblings and will need community support since I'm pretty bad at dealing with neophyte level problems these days.

More on each of these threads as progress is made...