This week in pyVmomi development I've been working on Python 3x support. In my opinion this is the most important move to make in pyVmomi development. While python 2.7 will have a long life ahead of it with support continuing through 2020 most state of practice projects will be moving to python 3 much ahead of that.
As a library, our target audience is the developer working on new products right now so tying the project to Python 2 only means cutting out a majority of our potential adopters. For the general health and longevity of the project we need to make the move.
That means, the first, most invasive, and most volatile change to be made to pyVmomi will be to change the library just enough to allow it to operate both on Python 2 and Python 3. And I'm already in process of making these changes while also writing a test suite for the library.
In the process of writing these changes I've found that there's more to do than initially thought. Python 3 migration was one of the blockers that initially made me pause when I was evaluating the use of pyVmomi in OpenStack. I went from very optimistic, to the realization that there would be some work to do. Now in my current investigation, I've determined that trivial python 2 to 3 automation may not get us all the way there.
I'll be spending the next week or so investigating how to best make this move. And I'll keep you posted on how this is progressing. I'll be looking for the simplest, cheapest, and best fit solution for the problem. That doesn't mean I'm looking for the most elegant solution but just the most effective.
Of primary concern to the pyVmomi SDK is maintaining compatibility with the existing tools and toolchains that already use it. That may mean occasionally making decisions that don't look Pythonic. The solution as a whole is more important than any one technical element. That's what we keep in mind when we design solutions like pyVmomi. How do we deliver something that doesn't just work in its beauty, but also in it's practicality.
Here's a nice video to underscore this kind of thinking. It's a thoughtful analysis of why Betamax lost to VHS and it underscores the role of ecosystem. The total ecosystem a product lives within is important and its really those factors that determine which solution is proven.
On other fronts, the pyvmomi-community-samples project is starting to take shape. This project has our lowest bar to entry. If you were just getting started and wanted to contribute that's the best place for you to go. If you want to write something but don't have an idea what to write, there is a significant backlog on the project ready for takers. Some marked with the 'low hanging fruit' tag are likely suitable for beginners.
If you are contributing to pyvmomi-community-samples then you are making a contribution to an Apache 2 Licensed project. That means you get to keep a by-line on your file if you like, but it really belongs to the community after you submit it and it's under that nice Apache 2 License. In my experience, code you post can end up in the darnedest places so you might as well post to a project that will respect your authorship.
If you are a bit more advanced and want to extend pyVmomi then I've set up pyvmomi-tools. The project's purpose is to act as an incubator while we iron out problems like the Python 2 to 3 issue. I've blogged about the project here. You'll need to be a more advanced developer to really help out with that project and I've been hearing from people interested in talking about which way to go with extending pyVmomi beyond it's core vSphere API. There's lots to do there and we'll need to setup some time to discuss between interested parties new feature sets to include.
If you are an IRC user you can hang out with me on freenode on #pyvmomi and #pyvmomi-dev or as always I'm on twitter.
We're also working on a regression testing suite for pyVmomi targeted for public use by teams interested in doing their own integration testing. This is intended to be a test suite for building a public facing CI for pyVmomi contributors, but also it will be a way for private cloud vendors take and adapt the project and create their own in-house IaaS stress testing services.
More on that another time...
2014-06-27
2014-06-25
pyvmomi-tools and alarm ack & reset
There are going to be use cases like "Ack & Reset vCenter Alarm implementing hidden API method" these are going to be relatively common and yet reasonably outside the definition of an official API binding. To handle this I've created pyvmomi-tools as a project to distribute these kinds of additions to the official bindings.
We'll record feature requests and after a number of these get implemented we'll provide an official release that you can pull down with `pip`. The pyvmomi-tools project will be freer to explore techniques for working with vSphere that might not be officially supportable or might break between releases.
I will want to come up with a system of warnings to let you know when you are using an API that might not survive between vSphere releases or isn't officially released and therefore isn't covered by the very generous backwards compatibility guarantees that the rest of vSphere is.
But... that's another topic...
We'll record feature requests and after a number of these get implemented we'll provide an official release that you can pull down with `pip`. The pyvmomi-tools project will be freer to explore techniques for working with vSphere that might not be officially supportable or might break between releases.
I will want to come up with a system of warnings to let you know when you are using an API that might not survive between vSphere releases or isn't officially released and therefore isn't covered by the very generous backwards compatibility guarantees that the rest of vSphere is.
But... that's another topic...
2014-06-20
Documentation is live
This week I've been dealing with documentation for pyVmomi. I decided to go with the Google Python Documentation standard for this current set of docs because the Google standard is very human readable and renders reasonably well in GitHub. The docs are live on GitHub now.
However, when it comes to the most useful documentation standard for inline documentation the Sphinx standard has too many benefits to ignore. In particular if you want code completion using tools like Eclipse or IntelliJ you need to have Sphinx docs and those have to be on concrete static classes.
So, for actual inline code comments I want to enforce Sphinx's standard with type hinting. These kinds of hints are very valuable for static code analysis tools and you simply can't get that with any other standard at present.
My next set of tasks will be to build up the testing infrastructure around pyVmomi, once that is done we'll be able to leave behind the timid and slow progress stage we've been stuck in. This will have to happen sooner rather than later. The project will officially "launch" at VMworld coming up in August and we'll need to be ready for the additional attention that will bring.
Part of this week's documentation effort has been trying to find ways to present developer documentation and developer guides to make sure that new developers don't get mired down too badly. I've been years off the speakers circuit now and I think it shows. My presentation is a bit choppy but I hope you'll find it useful.
This video that I've prepared walks through the code contribution process from fork to pull request. It assumes you have a previously setup GitHub account. This is the first time I've done one of these in a good long while and I was surprised by how badly the text blurred. I may update this later.
When you submit a pull request it should only have the commits listed in it that you wrote if it doesn't then something went wrong! You'll have to start over by branching from master and cherry-picking your commits. Git has a learning curve but it's worth learning and it's worth spending time with to get things down properly.
If you have a good contribution and you're really stuck, I've already pulled and cherry-picked some key submissions from other people into my local git repos. I try to preserve the code authorship marks on the patches so that you can find your committer credits on the source repository later.
Now that we know the basics of how to contribute, I'll take some time and work on precisely what we're going to work on and when we'll release it.
More on that next week...
However, when it comes to the most useful documentation standard for inline documentation the Sphinx standard has too many benefits to ignore. In particular if you want code completion using tools like Eclipse or IntelliJ you need to have Sphinx docs and those have to be on concrete static classes.
So, for actual inline code comments I want to enforce Sphinx's standard with type hinting. These kinds of hints are very valuable for static code analysis tools and you simply can't get that with any other standard at present.
My next set of tasks will be to build up the testing infrastructure around pyVmomi, once that is done we'll be able to leave behind the timid and slow progress stage we've been stuck in. This will have to happen sooner rather than later. The project will officially "launch" at VMworld coming up in August and we'll need to be ready for the additional attention that will bring.
Part of this week's documentation effort has been trying to find ways to present developer documentation and developer guides to make sure that new developers don't get mired down too badly. I've been years off the speakers circuit now and I think it shows. My presentation is a bit choppy but I hope you'll find it useful.
This video that I've prepared walks through the code contribution process from fork to pull request. It assumes you have a previously setup GitHub account. This is the first time I've done one of these in a good long while and I was surprised by how badly the text blurred. I may update this later.
When you submit a pull request it should only have the commits listed in it that you wrote if it doesn't then something went wrong! You'll have to start over by branching from master and cherry-picking your commits. Git has a learning curve but it's worth learning and it's worth spending time with to get things down properly.
If you have a good contribution and you're really stuck, I've already pulled and cherry-picked some key submissions from other people into my local git repos. I try to preserve the code authorship marks on the patches so that you can find your committer credits on the source repository later.
Now that we know the basics of how to contribute, I'll take some time and work on precisely what we're going to work on and when we'll release it.
More on that next week...
2014-06-13
on the topic of python documentation
I was going to talk about task management and how the pyVmomi API works with that. But, this week was eaten up by documentation work.
If you're not aware there's a number of Python documentation styles out there. In particular the Sphinx style documentation is the most popular in the Python community at large. The problem with Sphinx documentation style is that it's rather dense to read. I plan on hosting these final products on the github project site, so that becomes part of the conversation as well.
Because GitHub does such a good job with markdown files, as part of my documentation work I prepared this markdown version of the documentation. This is a very nice looking set of docs that we can generate procedurally. It's relatively regular and it will be easy for people to edit and maintain. But markdown isn't anywhere near anyone's idea of a documentation standard for Python.
As a matter of human readability I'm very fond of the Google Python Documentation style. It has the benefit of being regular yet very human readable. The problem is it doesn't get you where you need to go 100% of the time. Many tools around Python are built assuming Sphinx style markups.
So... what to do? Plainly I've not decided. I will at minimum need the sphinx version of these documents. With the Sphinx version we should be able to enable IDEs to do code completion and other nifty tricks. But there's a fair bit of work in taking the vSphere HTML documentation and turning it into Sphinx documentation. I'm busy with a tool to finish off conversions to either format.
I've spent the week examining various ways to document the project and how to link these up in people's IDE. I've also been looking at ways to walk people through using the library and I've been busy cataloging new enhancement directions as we move along.
The documentation work makes me very optimistic about switching to static Python classes for a future version of pyVmomi and it might even yield some interesting ways to do a new Java binding using Groovy as a dynamic language base (because Groovy and Python share a lot of language features.)
I've had to drop off here at week's end with about 80% of the solution in hand for the documents.
More next week...
If you're not aware there's a number of Python documentation styles out there. In particular the Sphinx style documentation is the most popular in the Python community at large. The problem with Sphinx documentation style is that it's rather dense to read. I plan on hosting these final products on the github project site, so that becomes part of the conversation as well.
Because GitHub does such a good job with markdown files, as part of my documentation work I prepared this markdown version of the documentation. This is a very nice looking set of docs that we can generate procedurally. It's relatively regular and it will be easy for people to edit and maintain. But markdown isn't anywhere near anyone's idea of a documentation standard for Python.
As a matter of human readability I'm very fond of the Google Python Documentation style. It has the benefit of being regular yet very human readable. The problem is it doesn't get you where you need to go 100% of the time. Many tools around Python are built assuming Sphinx style markups.
So... what to do? Plainly I've not decided. I will at minimum need the sphinx version of these documents. With the Sphinx version we should be able to enable IDEs to do code completion and other nifty tricks. But there's a fair bit of work in taking the vSphere HTML documentation and turning it into Sphinx documentation. I'm busy with a tool to finish off conversions to either format.
I've spent the week examining various ways to document the project and how to link these up in people's IDE. I've also been looking at ways to walk people through using the library and I've been busy cataloging new enhancement directions as we move along.
The documentation work makes me very optimistic about switching to static Python classes for a future version of pyVmomi and it might even yield some interesting ways to do a new Java binding using Groovy as a dynamic language base (because Groovy and Python share a lot of language features.)
I've had to drop off here at week's end with about 80% of the solution in hand for the documents.
More next week...
2014-06-06
pyvmomi-tools: providing library extensions and tools outside of the pyvmomi release cycle
Last week I mentioned how some sample requests on pyvmomi-community-samples were bringing up a few interesting topics. These were issues I've seen elsewhere and desperately wanted to fix.
I've found that new programmers to the vSphere API have a lot of trouble dealing with Tasks, PropertyCollector, PropertyFilter, and Views constructs. These are in particular sticking points I would like to make easier for the programmers that don't want to necessarily deal with those constructs straight off. It is also beneficial to certain projects if the code for handling these kinds of issues is kept independently from the project itself.
In general, the tactic of developing pyvmomi-tools independently of pyvmomi means we get more latitude in the tools' development. If we identify certain tools as promotion worthy some might get promoted up into pyvmomi itself and others might be broken out into supporting libraries. In general, smaller projects are easier to maintain and having a large number of small projects to maintain means its easier to delegate work and assure an individual module is bug free.
However, a large number of small modules is hard to use when you are a developer. Instead, it's much easier to grab a single library. So, to handle that problem when we get there, pyvmomi-tools should morph into a top-level project in time. When you include pyvmomi-tools in your requirements you would be getting a number of smaller libraries with focuses on any number VMware APIs but you would get to interact with a cohesive whole.
That's the vision at any rate. You can compare what using pyVmomi with and without tools is like by looking at my latest samples on power_cycling virtual machines.
I've found that new programmers to the vSphere API have a lot of trouble dealing with Tasks, PropertyCollector, PropertyFilter, and Views constructs. These are in particular sticking points I would like to make easier for the programmers that don't want to necessarily deal with those constructs straight off. It is also beneficial to certain projects if the code for handling these kinds of issues is kept independently from the project itself.
Why not just add these functions directly to pyVmomi?Well ... just how can I do that? The pyVmomi library dynamically generates and loads its class definitions for vim.* and vmodl.* namespaces. I'm currently playing with ways to make this happen statically but that is going to be a big change and I don't want to tackle it until we have more tooling support around the library.
In general, the tactic of developing pyvmomi-tools independently of pyvmomi means we get more latitude in the tools' development. If we identify certain tools as promotion worthy some might get promoted up into pyvmomi itself and others might be broken out into supporting libraries. In general, smaller projects are easier to maintain and having a large number of small projects to maintain means its easier to delegate work and assure an individual module is bug free.
However, a large number of small modules is hard to use when you are a developer. Instead, it's much easier to grab a single library. So, to handle that problem when we get there, pyvmomi-tools should morph into a top-level project in time. When you include pyvmomi-tools in your requirements you would be getting a number of smaller libraries with focuses on any number VMware APIs but you would get to interact with a cohesive whole.
That's the vision at any rate. You can compare what using pyVmomi with and without tools is like by looking at my latest samples on power_cycling virtual machines.
- plain pyvmomi: virtual_machine_power_cycle_and_question.py
- with pyvmomi-tools: virtual_machine_power_cycle.py
The big things to notice? Look at finding a virtual machine, waiting for a task, and responding to something that can cause a task to hang. Naturally it's possible to do all this well without pyvmomi-tools but I hope you'll agree it's nice to have a well conceived helper tool to simplify some of this work.
Building pyvmomi-tools this week, I actually ended up throwing out a lot of lib work I believed would be necessary based on my work with other vSphere bindings... pyVmomi's internals have some pleasantly surprising side-effects and benefits that I'd love to dissect.
More on that next week...
2014-05-30
pyvmomi-community-samples the tags and what they mean
This week on the pyVmomi Community Samples project I created a list of tasks to be accomplished.
The pyVmomi library itself is a very bare binding onto the vSphere Management API. This is by design since the library has to match very closely to the API which only changes with each vSphere release. That means we need to be careful to only alter that library when vSphere gets altered. Breaking out the samples to its own project frees the software to change at a much more dynamic rate and opens up the possibility that we can have a range of collaborators outside VMware contribute to those samples.
Since I've heard a lot of folks are wanting to help out but don't know how, I've created a set of issues marked 'help wanted' at any point if you want to help and don't know how... pick up a task marked 'help wanted' and comment on there that you're working the issue. Once you have a pull request ready reference the issue number in the comment.
Some issues as I notice they are being worked on will get marked 'in progress' to let you know that the issue is taking a bit to work on. I'll be tagging issues with other special tags to make lists to send to VMware staff internally in order to solicit help and support from folks working inside VMware that may not normally spot issues in public.
Currently I happen to be working on issue 40 and it's opened a whole different can of worms. The samples currently have a tools package. This tools package is the first stop on our way to identifying useful tools for general pyvmomi development.
More on that next week...
The pyVmomi library itself is a very bare binding onto the vSphere Management API. This is by design since the library has to match very closely to the API which only changes with each vSphere release. That means we need to be careful to only alter that library when vSphere gets altered. Breaking out the samples to its own project frees the software to change at a much more dynamic rate and opens up the possibility that we can have a range of collaborators outside VMware contribute to those samples.
Since I've heard a lot of folks are wanting to help out but don't know how, I've created a set of issues marked 'help wanted' at any point if you want to help and don't know how... pick up a task marked 'help wanted' and comment on there that you're working the issue. Once you have a pull request ready reference the issue number in the comment.
Some issues as I notice they are being worked on will get marked 'in progress' to let you know that the issue is taking a bit to work on. I'll be tagging issues with other special tags to make lists to send to VMware staff internally in order to solicit help and support from folks working inside VMware that may not normally spot issues in public.
Currently I happen to be working on issue 40 and it's opened a whole different can of worms. The samples currently have a tools package. This tools package is the first stop on our way to identifying useful tools for general pyvmomi development.
More on that next week...
2014-05-23
How pyVmomi is something different
I'm largely responsible for the current set of Java samples vSphere WS Management SDK released with vSphere 5.5 last fall. It was really cool for me, for the first time in a 20 some-odd year career to actually be involved in a shrink-wrapped product roll out. That may be out of fashion these days but it was still cool.
This was also an unusual project for me mainly because I couldn't really talk about it at the time. I've worked most of my career on websites, COTS or GOTS modifications, or OpenSource customizations and that implicitly entails some freedom to talk about things that closed source projects don't. The original plan was that I was going to start writing about vSphere 5.5 once it rolled out, but then OpenStack happened... (more on that in coming weeks).
My job at VMware wasn't just to write samples but also to provide a link between the API developers inside VMware and developers outside VMware. Since the Java SDK was my first big job with the company I did things the way that people at the company were used to, I followed the normal channels. I really wanted to be a good VMware citizen first before I started rocking the proverbial boat.
If you've seen the vSphere Management SDK for vSphere 5.5 then you'll notice I took it in a very JPA-like direction. I wanted the SDK samples to feel like working with any other high level Java code. The theory was that in tools like JPA magic annotations take care of wiring up an Object to a Table. The effect is that the Java programmer gets to focus on data modeling and less on details that don't pertain to their application. I tried to emulate that with the samples... magic annotations wire the POJO getters and setters up to the Command Line Interface (CLI).
The major changes involving the annotations and the framework that supported them were done in just three weeks. However, testing, verifying, and repairing the samples took the rest of nine months with a team of testers. That second part was long, labor intensive, and frankly not very satisfying.
Unfortunately, because this was all closed-source and behind the scenes development I couldn't get feedback from the most critical audience that these changes affected... you the developer outside VMware. Inside VMware these changes have been well received and I gave the new Java SDK maintainer explicit permission to completely nuke the vSphere 5.5 Java samples, annotations, and framework when I shifted to OpenStack work in January 2013. The current maintainer has told me he likes the feel of the new samples and I hope VMware will eventually open source the annotation framework that takes CLI options and wires them to the POJO. The vSphere samples that arrive on the next release will be managed by a new developer and I'm not involved with that new framework anymore.
As I mentioned in early 2013 I started to transition to OpenStack Nova work full time. Again, my typical strategy is to observe a project for one cycle before rocking the proverbial boat. I mostly watched the the Havana release and I more actively participated in the Icehouse release. In all this I felt there were some real problems with how the vSphere Management SDK was getting used. Some of these problems were related directly to how people understood the SDK and others were directly related to the SDK itself.
I asked that we either bring in someone full time to work the pyVmomi library, its community, and its problems or that I be moved to the pyVmomi project full time. Guess which happened?
As of this April, I took over the public facing parts of the pyVmomi project. Part of that has been not rocking the boat for a bit as I observe how the sausage gets made. The other part of has been finally rocking the boat in regards to how samples are made. With pyVmomi I've been asked to write samples again but this time it's different.
The project pyVmomi Community Samples is a really radically different way to do SDK samples for VMware. The project is an Apache 2 Open Source samples project and we're getting samples from non-VMware contributors. This new project structure means that you help write which samples you want to see, you help write them the way you want to see them, and you can use the awesome power of github to send pull requests. The samples become a durable, sharable, and living community effort.
I've tentatively slated a new release of pyVmomi should come out every 6 months. That means releases every December and July. That does mean a bit of boat rocking early on for me... but then... if you'll help me out I think we can get this ship sailing in the right direction.
More next week...
This was also an unusual project for me mainly because I couldn't really talk about it at the time. I've worked most of my career on websites, COTS or GOTS modifications, or OpenSource customizations and that implicitly entails some freedom to talk about things that closed source projects don't. The original plan was that I was going to start writing about vSphere 5.5 once it rolled out, but then OpenStack happened... (more on that in coming weeks).
My job at VMware wasn't just to write samples but also to provide a link between the API developers inside VMware and developers outside VMware. Since the Java SDK was my first big job with the company I did things the way that people at the company were used to, I followed the normal channels. I really wanted to be a good VMware citizen first before I started rocking the proverbial boat.
If you've seen the vSphere Management SDK for vSphere 5.5 then you'll notice I took it in a very JPA-like direction. I wanted the SDK samples to feel like working with any other high level Java code. The theory was that in tools like JPA magic annotations take care of wiring up an Object to a Table. The effect is that the Java programmer gets to focus on data modeling and less on details that don't pertain to their application. I tried to emulate that with the samples... magic annotations wire the POJO getters and setters up to the Command Line Interface (CLI).
The major changes involving the annotations and the framework that supported them were done in just three weeks. However, testing, verifying, and repairing the samples took the rest of nine months with a team of testers. That second part was long, labor intensive, and frankly not very satisfying.
Unfortunately, because this was all closed-source and behind the scenes development I couldn't get feedback from the most critical audience that these changes affected... you the developer outside VMware. Inside VMware these changes have been well received and I gave the new Java SDK maintainer explicit permission to completely nuke the vSphere 5.5 Java samples, annotations, and framework when I shifted to OpenStack work in January 2013. The current maintainer has told me he likes the feel of the new samples and I hope VMware will eventually open source the annotation framework that takes CLI options and wires them to the POJO. The vSphere samples that arrive on the next release will be managed by a new developer and I'm not involved with that new framework anymore.
As I mentioned in early 2013 I started to transition to OpenStack Nova work full time. Again, my typical strategy is to observe a project for one cycle before rocking the proverbial boat. I mostly watched the the Havana release and I more actively participated in the Icehouse release. In all this I felt there were some real problems with how the vSphere Management SDK was getting used. Some of these problems were related directly to how people understood the SDK and others were directly related to the SDK itself.
I asked that we either bring in someone full time to work the pyVmomi library, its community, and its problems or that I be moved to the pyVmomi project full time. Guess which happened?
As of this April, I took over the public facing parts of the pyVmomi project. Part of that has been not rocking the boat for a bit as I observe how the sausage gets made. The other part of has been finally rocking the boat in regards to how samples are made. With pyVmomi I've been asked to write samples again but this time it's different.
The project pyVmomi Community Samples is a really radically different way to do SDK samples for VMware. The project is an Apache 2 Open Source samples project and we're getting samples from non-VMware contributors. This new project structure means that you help write which samples you want to see, you help write them the way you want to see them, and you can use the awesome power of github to send pull requests. The samples become a durable, sharable, and living community effort.
I've tentatively slated a new release of pyVmomi should come out every 6 months. That means releases every December and July. That does mean a bit of boat rocking early on for me... but then... if you'll help me out I think we can get this ship sailing in the right direction.
More next week...
Subscribe to:
Posts (Atom)