Some comments on testing in virtual environments

This post talks about load and performance software testing in a virtual environment. The post also includes a presentation on the topic.

Microsoft and VMware are actively pushing use of virtualization in the data center. For instance, sometime back Microsoft posted that they were dog-fooding Hyper-V for hosting the MSDN and TechNet portals. As server virtualization is adopted in the production center, any lingering concerns around conducting specific performance testing in a virtual environment will be moot! In fact, the same virtual test setup will be used in your production environment!

Speaking of virtual test setups, the presentation talks about testers assessing system performance in not very production-like environments. This is one reason why I lay emphasis on virtual lab automation solutions like VMLogix LabManager. I have written sometime back on what VLA offers over and above a hypervisor. VLA solutions have network fencing, networking, automation and a host of other capabilities that help testers do their job in production like environments.

The presentation also mentions that the team puts in lots of overtime to fix the defects (for testing in a virtual world). With a VLA solution (i.e., when you’ve formalized virtualization as a technology tool to use in your software engineering cycle), users can dramatically reduce the costs associated with manual IT provisioning and bug reproduction.

- Srihari Palangala

Automation in Virtual Labs — What exactly do we mean by it?

This blog has been talking about automation in virtual labs for a while now — so what exactly do we mean by it? Maybe it is time we bring some clarity around this.

Automation to a lot of users of virtual labs is the ability to spin up and tear down virtual machines on the fly (leveraging a central repository/library). This enables IT admins to provide lab users with self-serve capabilities. At VMLogix, we view this level of automation as a great start. However VMLogix LabManager provides a lot more in the name of automation (to IT admins and users alike).

A couple of pictures will speak a thousand words. The first picture below is the VMLogix LabManager high level architecture layout. Like you will see, many virtual hosts are managed by a central VMLogix LabManager server. These virtual hosts are able to bring up and tear down VMs, multi-machine VM configurations etc. on the fly (thanks to the VMLogix VMagent existing on each host — see the second picture). Thats where the definition and implementation of automation stops in the conventional sense.

High level VMLogix LabManager architecture

The second image below is more interesting in the context of automation and VMLogix LabManager. Each virtual host can support multiple VMs (LabManager supports multiple hypervisors including Citrix, Microsoft and VMware) — in each VM, you have the option of running a VMLogix guest agent. We believe that this guest agent provides the next level of automation & management support in virtual labs. IT admins and users can now run several operations inside each of the VMs — for example, installing software, cleaning the machine environment, capturing a movie, executing a command, pausing operations etc. You can see the full list of operations here.

Automation in the virtual hosts and virtual machines

We believe that this “second level of automation” provides users with a lot of flexibility and truly takes away manual repetitive tasks out of their hand. Think of the time you have spent getting a build machine ready, a test machine environment ready, a developer machine ready, …. and the list keeps growing. You can now automate all these operations to get the internals of a VM setup the way you want it. And if our repertoire of operations out of the box is not sufficient, you can build custom operations as well

Now, does this level of virtual lab automation help you in your job?

Today VMLogix LabManager supports leading hypervisors from Citrix, Microsoft and VMware as well. As newer hypervisors from these industry leaders are released (such as Hyper-V), VMLogix is committed to maintaining the hypervisor agnostic stance as a strategic differentiator.

Login here to register to our VMLogix Virtual Portal. If you want to try our product for free for 30 days, register here.

- Srihari Palangala

Next generation test tool from IBM - Rational Quality Manager

We came across an interesting blog (albeit new, but will be the space to watch for RQM development news) that posts about the latest on IBM Rational Quality Manager - which is being touted as the next generation test management tool from IBM. Apparently there will be a migration path for current IBM CQTM users.

The blog also had a post on the integration of test lab management long with IBM RQM.

The development of these tools from IBM is interesting news. VMLogix has a history of providing value added integrations with IBM Rational products. If you are an existing IBM Rational Build Forge or CQTM user and would like to leverage virtual lab automation - check out the VMLogix LabManager integrations and the value add of VLA with these products here and here.

We look forward to providing integrated solutions to market with newer IBM Rational products such as RQM as well.

- Srihari Palangala

Software build and test environments - how complicated is it for you?

I know from experience and from speaking with people in various domains that the software build and test environment setup process can easily get complicated and hard. So what have people written about it? I came across this post from Dana Cline — and was I overwhelmed. Dana has the mix of everything to make his job really difficult — building the base images, installing the database and sample data, multiple DB configs, multiple server platforms, multiple web servers, multiple languages, multiple architectures (32 and 64 bit), multiple client platforms and even multiple product releases supported. Whew! Did he miss anything at all

How complicated is your test and build scenario? Anything over and beyond Dana’s scenario? Would love to hear from you.

If you are anywhere close to how complicated the above scenario is (heck - even 20% as complex!) - you owe it to yourself to check out virtual lab automation products - such as VMware Lab Manager and VMLogix LabManager. In a few days, I’ll post a RoI calculator for you to check out. Can’t wait for it? Just drop me a note.

If you get yourself an account with us at VMLogix and login to our Virtual Portal, you will find an interesting set of resources - and among them a video that demonstrates how you can benefit from VMLogix LabManager automation. This video specifically demonstrates how your life with building/testing on multiple platforms/languages/systems/… and creating a test environment can be rapidly eased out.

- Srihari Palangala

Can Virtual Lab Automation help you treat VMs like Physical Machines?

I came across Eric Siebert’s write up on “When not to treat VMs like physical servers“. It is an informative post and he does make valid points — especially if you are in a situation where you are only running VMs independently on hypervisors.

In a test/dev/support/training or pre-production lab environment where you can have many VMs under the purview of a central lab management system (like VMLogix LabManager), then you do have some additional management functionality to help address some of the points he raises.

• One of the concerns raised is running operations on VMs concurrently (with VMs that are hosted on the same server). With a virtual lab automation solution like VMLogix LabManager, users can leverage automation capabilities to execute scripts/demand scans/commands/re-booting of machines, installing of software etc. in an automated manner. Here is a screenshot of all the automation operations available in VMLogix LabManager. With the automation capability and the ability to schedule jobs on hosts, users can cascade/stagger operations with VMs. This helps control the load on physical hosts.
  
• With a centralized VM management system user and team management is simplified — the lab administrator can control user/team access to lab resources (such as machine templates, configurations etc.). Users/teams can also share lab resources with appropriate permissions (view/edit/delete). This functionality helps in driving security and access control to lab resources. Here is a screenshot:

User and Team Management in LabManager

• Performance monitoring - Again a centralized virtual lab management system helps track VM use of host resources (in addition to other performance monitoring capabilities). Here is a screenshot of VMLogix LabManager monitoring the hosts’ RAM utilization.

Reports in VMLogix LabManager

What do you think? Do these functions help you work with VMs providing a layer of management control as you transition from working with physical to virtual machines?

If you would like to learn more, register for a free evaluation of VMLogix LabManager.

- Srihari Palangala (srihari@vmlogix.com)

Find out why developers and testers will love VMLogix LabManager

Dave Oliver has posted yesterday on “Why Developers and Testers will LOVE VMLogix LabManager More!” (the “More” being in follow up to the VMware Lab Manager product review).

Thank you for thinking about us Dave.

- Srihari Palangala

Interact with the VMLogix LabManager community

I’m very happy to announce that yesterday we added a significant component to the VMLogix Virtual Portal — the LabManager user forum. For those of you unaware, the Virtual Portal should be your one stop resource to learn about virtual lab/test lab automation and VMLogix LabManager (it has several resources, including product videos, documentation, technology videos, a knowledge base etc.). Now the VMLogix LabManager user forum is also available to you to post queries, check responses and learn more about the product.

If you’ve been on the fence re: test lab automation - take the plunge now! Make a request for a free evaluation of the LabManager product, leverage the resources on the portal, interact on the user forum and share your journey experiences with the LabManager community.

Will look forward to seeing all of you there!

- Srihari Palangala

Test Lab Management - Can it help you make a better decision on when to ship software?

I found this page which provides 6 common factors which managers use to decide when to stop software testing and ship software. As quoted on the site, the factors are::

• “Deadlines, e.g. release deadlines, testing deadlines;
• Test cases completed with certain percentage passed;
• Test budget has been depleted;
• Coverage of code, functionality, or requirements reaches a specified point;
• Bug rate falls below a certain level; or
• Beta or alpha testing period ends.”

I beleive most practical software engineering cycles balance time, quality and budget while making a choice of when to ship software. Lets for a moment assume that the list above is a reasonable representation of factors which influence when software is shipped.

Among the reasons above, not all factors are equally good (obviously, right?). I believe some are ‘better’ factors and some are ‘worse’ factors that influence when software is to be shipped. I would categorize as:

The “Bad” factors that primarily influence when to ship software

• Deadlines, e.g., release deadlines, testing deadlines
• Test budget has been depleted

The “Better” factors that primarily influence when to ship software

• Test cases completed with certain percentage passed
• Coverage of code, functionality or requirements reaches a specified point
• Bug rate falls below a certain level
• Beta or alpha testing period ends

How are your ship decisions made currently?

I would venture that if you are making a decision based on one/more of the ‘better’ factors, then you have better control and better management on your software engineering cycle. Unfortunately though, sometimes we have to make a decision based on the ‘bad’ factors.

If your software ship decisions are influenced by the ‘bad’ factors above, then I would encourage you to take a look at the test lab management software like VMLogix LabManager. You will certainly benefit by productivity gains offered by test lab management software. You can register for a free VMLogix LabManager software trial as well.

Even if your decision is based on the ‘better’ factors above, you would benefit by using a test lab management software. You can vastly improve on the threshold levels for test case completion, code/functionality/requirements coverage or bug discovery rates. This has a direct bearing on your software quality.

Where does your software engineering process stand currently? Are you making choices based on the ‘bad’ or ‘better’ factors? Or something else not currently on the list? Would love to hear your thoughts.

- Srihari Palangala

What are IT Test Lab Managers Responsible for?

I looked up Google for people’s expectations and job description of an IT Test Lab Manager. Here are a couple of sources I found:

• Microsoft TechNet - Has a good description on the job expectations of a Test Lab Manager. It is a dated article (2003) - however, I do believe that if you have not invested in a virtual lab automation solution like VMLogix LabManager, a lot of what is listed on the site might still be very true.
• This job post on Yahoo requires a test lab manager/technician to handle — configuration & access control, scheduling lab use, audit trails, configuration & administration of test lab servers etc.
• Another job posting for a test lab manager - describes both strategic and tactical job expectations.

If I were to summarize the expectations, I would put them in two buckets –

• IT Lab planning (e.g., Evaluate and recommend changes & plan the lab to meet organization needs, Plan Budget and costs, Lay down policies & procedures, Working with cross functional teams for lab planning - product management, marketing, support etc.)

• IT Lab Operations (Lab scheduling, Writing test plans, Performing tests, Lab maintenance, Lab documentation, Audit trails, Inventory control, Capacity planning, Configurations and updates on hardware/software, Physical security, Backups, Incident tracking, Access control, etc.)

Most of the IT Lab Operations tasks listed above can be handled by good test lab automation products (freeing up your lab manager’s time to focus on higher value tasks).

Are there other tasks that you find a test/IT lab manager should be responsible for?

I’d encourage you to take a look at VMLogix LabManager and request a product evaluation if you are interested in drastically cutting down time spent on your lab operations (like those mentioned above).

- Srihari Palangala

Virtual Lab Automation Overview Post on the Server Virtualization Blog

The Server Virtualization blog made a post on virtual lab automation titled “Lab management streamlines asset allocation, saves cost” and covers the three leading test lab automation vendors. The post also covers customer’s viewpoints on the products and the benefits that they have got by deploying test lab automation products. A good read if you are somewhat new to test lab automation. I’d also encourage you to register on the VMLogix Virtual Portal - it will help you make a deep dive into the technology and the LabManager product.

Drop me a note at srihari@vmlogix.com if you want to know more.

- Srihari Palangala