Today, we’re thrilled to say that we have the only three test management solutions that support the newly announced JIRA 6 and have been supporting the pre-released versions of JIRA 6 for the last three months. JIRA 6 is Atlassian’s largest release yet, which includes major enhancements across the product; faster navigation and editing, simplified workflow creation and powerful mobile functionality so teams building software can get more done with less effort.

“We are pleased to provide value to JIRA users on project teams of any size to easily integrate test management with JIRA 6,” said Samir Shah, CEO of Zephyr. “Atlassian is a key partner of Zephyr and we strive to enable our customers to derive the greatest value from their JIRA and Zephyr deployments.”

JIRA helps companies improve project management productivity by enabling teams to be more modern, fast, mobile and simple.

Modern: New user interface (UI) redesigned for visual appeal and speed

The UI improvements focus on JIRA’s most widely-used features and functions, so team members can create, edit and work through lists of issues more efficiently and easily. JIRA incorporates Atlassian’s new intuitive, design-centric approach to user experience that has been deployed across Atlassian’s product portfolio. Keeping lock-step with JIRA 6, Zephyr for JIRA v1.7 has been redesigned to use Atlassian’s new Design Guidelines. Check it out!

Fast: Enhanced navigation and editing capabilities to speed use in common scenarios

A new Detail View introduces split screens to easily edit issues without jumping between pages, and a new navigation bar provides quick access to the issues and applications teams need most. Whether triaging a list of issues, planning a sprint, or handling basic tasks, users can quickly optimize JIRA to work in a mode where all important details and information are readily available. Zephyr for JIRA offers a single top navigation bar with bigger fonts and sizes and a cleaner left side project menu.

Mobile: Collaborate anywhere from tablets and mobile handsets

A new mobile interface designed from the ground up to give modern teams quick access to critical information and important activities, anywhere. On-the-go users can easily make quick comments, prioritize or assign issues, use @mentions to bring team members into a discussion, and participate in the full comment threads.

Simple: More than 20 pre-configured workflows

Simple project creation out of the box so teams can get started in just a few clicks with scrum, kanban, or another favorite flavor of agile. In addition, more than 20 best-practice pre-configured workflows that match an industry or approach are available on the Atlassian Marketplace, a one-stop shop for add-ons to Atlassian products.

Zephyr continues to stay lock-step and our solutions are ready for JIRA 6 today! 

Zephyr Enterprise Edition is best suited for departments and enterprises that run multiple projects in parallel with very structured testing cycles, and a lot of automated and/or manual testing. This product has best-in-class integration with both JIRA 6 Download and On-Demand.

Zephyr for JIRA completes end-to-end project management in JIRA 6 by adding testing to the overall planning, development, bug tracking and reporting process. Zephyr for JIRA is best suited for project teams that want to integrate testing directly into their JIRA workflows.

Zephyr Community Edition is free for small project teams with 10 users or less, Community Edition is perfect for those that need to get a handle on testing in their current project or for teams just getting started with test management.

They will discuss Room to Read’s inspiring story, their secrets to scale and success, and the role of corporate social responsibility.

The event takes place from 6-8:30pm, at Atlassian’s Headquarters in San Francisco, located at 1098 Harrison St. You can register on their meetup page.

What is a testing strategy?
A testing strategy basically defines how testing is going to be performed on a particular project. What kind of testing, how, when and where are all questions that need to be answered in a good testing strategy. It is important to understand that the testing strategy belongs to the whole team, not only the QA department (in a whole team approach, everyone is responsible for the quality). Another point worth mentioning is that there is not a best test strategy that can be applied to all projects. All projects are different, with different technologies, different business objectives and of course teams with different skills sets.

Why is important to have a testing strategy?
First, to be able to work sustainable. “Agile Processes promote sustainable development. Teams should be able to maintain a constant pace indefinitely” says one of the principles in the Agile Manifesto. And the Declaration of Interdependence re-states this objective saying that “we increase ROI by making continuous flow of value our focus”. To achieve a sustainable pace a team needs to understand how is going to maintain the cost of change low as the codebase (and the complexity) increases. Secondly, to understand the teams strengths and weaknesses and how these are going to be mitigated. The testing strategy depends on the team’s skill set and it doesn’t make any sense to have an unrealistic strategy.

How to build a test strategy:
The whole team should be responsible for creating this test strategy. When defining one, the team should think about the business objectives, tduration, size and importance of the project, the team that is going to be working on it, the current framework and tools, etc. There are a couple of really useful tools that any team could use when building its strategy.

Agile Testing Quadrants
In the testing quadrants matrix (created by Brian Marick, a tester and signatory of the Agile Manifesto), the X axis is divided between tests that support the product and tests that critique the product and the Y axis is divided between business facing tests and technology facing tests. The 4 resulting quadrants of the matrix look like this:

The matrix is very useful to categorize all the types of tests that need to be performed and how those are going to be performed. It is highly recommended to buy Crispin and Gregory’s book or you can just google some of the presentations available (like this one).

The Test Pyramid
The test pyramid is a concept popularized by Mike Cohn. It looks like this one:

Basically, what the pyramid tries to recommend is the amount of automated testing that should be performed in an Agile process. Therefore, there should be a great deal of unit and components tests, less acceptance tests and even fewer GUI tests (and finally the manual tests that should always exist). The rationale of this pyramid lies in that the tests at the bottom are faster and give feedback immediately while going up takes us to slower and more brittle tests.

Conclusion
Understanding where you are at what you want to accomplish in terms of testing on your project is of utmost importance. Be realistic and use tools/frameworks that help you maintain the cost of change low and support a sustainable pace. Everyone in the team is responsible for quality and therefore everyone should contribute to it!

The new look and feel is modern, fast and easy to use. In most cases, the changes are cosmetic while the functionality remains the same. Some of the key changes that users will notice are:

Single top navigation bar with bigger fonts and sizes and a cleaner left side project menu

Rounded edges and shading around boxes

A more responsive user interface

In addition to the UI makeover, we expanded our compatibility with other Add-ons; GreenHopper, Structure and JJUPIN. You’ll notice some of the productivity improvements we’ve made to make it easier to update status of test executions when all steps have passed or failed, test steps can now be cloned by the Structure add-on and test executions are not deleted when that user is deleted. And like all releases, we made it perform better and fixed bugs.

Download your 30 day free trial to see why 900+ customers have made Zephyr for JIRA among the Top Grossing add-ons in three categories and experience the same productivity gains as customers such as Amazon, BMW, Audi, Michelin, NavTEQ, Lufthansa, Riot Games, ING Bank, Pitney Bowes and American Red Cross.

1. #1 Top Grossing Testing & QA add-on
2. #2 Top Grossing JIRA add-on
3. #2 Overall Top Grossing add-on

How to get it:
Sign up for a 30-day trial or buy it from here. If you already have a previous version, go to your UPM (Plugins Manager in JIRA 5) and look for the upgrade notification. It’s a one-click upgrade process.

Testing has historically been a heavily manual process, but to the benefit of software quality everywhere this is becoming less acceptable to organizations, and less feasible overall. Today’s systems tend to have a great deal of functionality and with that comes more testing. Continuing to add value to these systems while ensuring that the existing value stays intact is a balancing act for management and a challenge for test teams. The desire for constant improvement with the need to keep existing functionality working indefinitely takes test automation from a stretch goal to an essential skill in your Quality Assurance teams.

The reasons for this can be articulated with a simple thought exercise. Consider that we have a new software development effort. We begin this effort by implementing and testing Features A, B, & C. In this first iteration QA only has to test the features that the development team implemented. Next, Features D & E are implemented. Now the development organization is faced with a dilemma. Do we:

1. Try to keep our pace by only testing the new functionality.
2. Hold up everything (including new development) until all new features are tested, and all existing features are regression.
3. Begin to stagger our development and testing cycles by moving to a gated or “over the fence” type of approach?

I can say with confidence that the only sustainable answer is #2. While it sounds extreme, this is the only way for teams to maintain a consistent delivery pace with predictable quality. However desirable, this is difficult in practice. As shown in the table below, while development effort over a short time horizon (say 2-4 weeks) stays fairly consistent, test effort over the same period (including regression testing) continues to expand. The water just continues to pile against the dam. In a QA effort that only performs manual testing, attempting to execute to #2 above is not achievable without either longer iterations or adding more testers to the team with each iteration. This is illustrated in the table below. Note how test effort balloons with each iteration while development effort stays basically the same.

Enter test automation. If your QA teams can offload some of this preexisting test effort to machines to do what they do best (repetitive tasks with known good outcomes) your team can continue to add value indefinitely while the machines will let you know if something goes wrong with old functionality. Combine this test automation with a Continuous Integration system and you discover problems almost as soon as they are introduced.

That’s all well and good in a perfect world, but what about those tests that can’t be automated? In these cases you still have the same classic problem as described above with expanding test effort, but the solution comes in how you manage it. One of the greatest advantages of automated testing, particularly automation that is triggered every time your system changes, is that you get feedback of all that prior knowledge of your system within the context of a specific change and against a specific version of the software.

This is where good configuration management becomes essential. If you want to achieve the kind of visibility you get nearly for free with automated tests and continuous integration with your manual tests you must be able to point to a specific version of your software and say: “tests X, Y, and Z have been run against this, but Q, R, and S haven’t”. Sadly, this also means accepting the reality that every single time your software changes (for me this usually means the MD5 hash has changed, since it’s now not the same system) all prior results have been invalidated.

However, this doesn’t have to be terrible. If you have good decomposition of components and capture precisely which version of the software you’re running a particular manual test against you now have a powerful set of data. Take the results of the automation and the results of the most recent run of all your manual tests and you can perform a risk analysis prior to your next deployment to decide whether or not it is worth it to run any missed/not-run-lately manual tests or just go for it. The choice is yours, but now you’ll have data to back your decision.

Bio:
Matt is a Software Developer at Rough Stone Software. But don’t hold that against him, he has spent his entire career focused on quality and testing, and he strives to help his teams turn out their highest quality work from beginning to end. Read his blog.

Now with the popularity of the Agile, Lean and Continuous Deployment, the developers are required to be more vigilant while coding. They are recommended to follow practices like TDD, ATDD to improve the code quality, reduce defects among other benefits. Since these practices are supposed to reduce the burden on testers, a lot of people are questioning the need for dedicated testers in the software development.

In this article, I will address questions around the need for testers, the value they bring in, and their role in the context of Lean and Agile.

Why do we need to build the quality in?
Let me begin with the Lean Principle #2, Building Quality In. Before jumping to solution mode, let us understand the ways the quality issues crop up and the cost associated with it.  Quality issues originate at the beginning of the project and the top 3 reasons could be grouped into:

1. Customer giving a wrong requirement and realizing it at a later stage
2. The developer writing wrong code due to misunderstanding of requirement
3. Developer understood requirement but coded a wrong logic

Irrespective of whether the mistake is by the customer or the developer, there is an effort involved in testing and rewriting the code. As per the 1:10:100 rule, the failure to identify the defect upfront could cost a fortune to the company.

Solving the quality issues:
First issue mentioned above could be solved by the development teams building the prototype and showcasing to the customer even before touching the code. Second and third issues could be solved by practices like TDD and ATDD.

If developers and customers work closely, ensure to fill the gaps and write quality code, can we still delivery a quality product without testers?

In recent days, some of the lean startup companies have tried customer testing and crowdsourcing the testing without involving the testers. However, it is still not proven if this is the right model. A lot of people also agree that even the most experienced developers make mistake. The developers come with a specialized technical background and they won’t have the same eye as a tester. They are good at unit level testing, and they cannot effectively do the integration/system level testing. This is not something which could be improved by automating the tests.

Role of tester in Agile, Lean environment:
Many teams have tried building products just with developer testing and have got bitten badly as well. As one could see from this discussion, people have reverted back to include testers.

There is no doubt now that dedicated testers are needed in all projects. We still need testers to provide a neutral eye and their specialized skills to improve the quality. In fact, testers have now more empowering role to play than before.

To conclude:

• Testers look at preventing defects and not just identifying them in the end
• They get involved in the beginning of the project working closely with the product owners(PO) and developers
• They ask right questions about requirements and in turn making every get clarity on the requirements
• They write acceptance tests, they do smoke tests and exploratory testing
• They work as a support structure for the development team in ensuring smooth delivery of the project
• They think beyond just testing and embrace tasks that help the team in smooth deployment to production

• Feb - Zephyr’s Integration with eggPlant Automation
• Jan - 2013 is going to be the year of Automation

QTP is one of the most widely used automation tools. We received a bunch of requests to showcase Zephyr’s integration with QTP which also corroborated the fact. While most of the mechanism remains same when it comes to integrating any automation tool with ZBot, two areas that potentially differ are how we kick off actual testcases and how we parse results.

First, how to kick off: QTP comes packaged with a command line utility called cmdrv.exe. This is available inside bin of QTP installation (e.g. C:\Progam Files\HP\QTP\bin\cmdrv.exe). We use this executable and supply actual test and output folder to it. This can be done directly in Zephyr testcase where we provide the entire path with parameters. Alternatively, this can also be done in a batch file which could be specific to a given test machine and contain specific path. In the approach shown in video, we took second approach and created a batch file called qtrunner.bat which take same arguments (test path and output folder path).

Next, how to read results: In its very rudimentary form, ZIP parses the generated log file and looks for any errors. This could easily be changed to looking for specific message or even to use xml output. If there is no error is found, testcase is considered pass else failed. Result is then uploaded back into Zephyr with some comments.

Further in this series, we will show how we can run QTP scripts outside of ZBot and possibly make them part of CI and nightly builds. We will also cover some change in new version of QTP (known as UFT – Unified Functional Test). Stay Tuned.

QTP is one of the most popular testing tool and it’s been a pleasure to work with it and to develop a integrated solution that would enhance the meaning of both the tools for our users. Thanks for reading this post and please feel free to contact us with your issues / comments or chime in on our documentation page (Git Pages)

Check out this video to see the integration in action:

Click here to view the embedded video.

As shocking as this is, it is the norm for most software releases. Even when every requirement was tested and verified thoroughly before being flagged ready to ship. The historical reporting shows that the QA team had finished the regression and high volume testing correctly, and management approved the key quality metrics that were being measured.

So what do you do in this situation?

If you were able to research the historical reporting, you’re probably using a test management system or some type of version control. The first thing to do is check the version control files to see who was supposed to test what, all the communication steps, feedback and issues provided by the end users as part of the product release follow up. Good organization leads to good testing, so hopefully the end users provided input data, screenshots and error messages which could be used to identify, analyse and fix the issues.

The most commonly used method of categorizing issues is based on following types:

• User interface
• Functional
• Performance
• Environmental

Categorizing issues helps management troubleshoot each issue by assigning it to the teams responsible for each piece of work.

User interface issues, play an important role in the user experience. User interface issues are the ones which impact the aesthetic (look and feel) appearance of the application and make it hard for the users to use and find the relevant information in the application (usability). Aesthetic issues can be reproduced and resolved by looking at the screenshots provided by the users. Based on the information present in the screenshots, issues can be further assigned to concerned teams for their verification, analysis and resolution. Aesthetic issues do more harm when the users are using the application on a mobile device. The most common issue with mobile device is – non standard screen size. Every major mobile device manufacturer designs their own screen size which forces applications to be sized accordingly.

A functional issue is when the functionality of the application does not do what it is suppose to do as per the requirement specification. After analyzing the functional description of the issue, map it against the specific feature / functionality / module of the product. This makes it easier to narrow down the problem area and assign it to the QA team member responsible for that feature for further analysis.

Performance issues are primarily linked with slowness of the application. Based on the information recorded in the historical report or in the production server logs, the performance team should be able to replicate the issue in the test environment. This helps the project team identify contention, if any, on any of the infrastructure layers like database, web servers or application servers. One of the most common problems is that the database size in the test environment is very limited and in production with every increase in user base, size of database increases proportionately.

Environmental issues are primarily linked to available free space (very important in database), hardware and software configuration and firewall. Once the team has the ability to replicate the issue in a test environment, the development team can verify and confirm whether or not all of the issues were interrelated and in fact existed in application.

How to prevent this from happening again?

The reality is all software has defects, either; “known defects” or “yet to be discovered defects.” With that said, defects do slip into the production for various reasons and quite often the quality of the release depends on skill levels of the team, tight timelines, ever changing requirements which often force teams to take shortcuts, differences in test and production environments, and difference in size of data sets in test and production environments. You also need to take into account how much human error plays into all of this.

To reduce risk, use as much automation or built in unit test cases as possible. There are plenty of development tools that automate functional testing. With shorter delivery schedules, allocate equal efforts for testing. On a pilot basis, Test Driven Development should be considered. However with proper steps taken during the project life cycle; defects can be prevented or minimized from slipping into the production. Some general guidelines that we have seen work well for project teams include:

• Requirements gathering for the product should be done by skilled business analyst who has domain expertise
• Development team or testing team should prefer to do unit testing for the code base before code is passed on for system testing
• Proper configuration, build management and release management process should be followed during product life cycle
• All change requests should go through change management board and every approved change that goes in the application should have suite of regression test cases
• Every defect which remains unresolved should be accessed for risks and its impact on business
• Usability issues can be prevented by using the focus groups early in project life cycle preferably either during design or system testing phase
• Functional defects can be prevented by executing the subset of high priority functional test cases in production before the application is rolled out to the users
• Performance issues can also be prevented by executing average volume load test on high priority use cases of the application in the production environment before application is rolled out to users
• By performing both performance and functional testing for a brief period of time, It gives us sufficient confidence that environment is good enough to handle the application once it goes live

Its good practice for the project team to provide a short window for testing in production environment before the application goes live. By rigorously standardizing on a process and striving for a well organized test repository, it makes resolving situations like this easier.

Real-Time Test Management from Zephyr Drives Instant Visibility

Keeping pace with agile release cycles is challenging when testing and test management can’t be done in real time. Zephyr’s real-time test-management solutions provide instant visibility into the actual status of tests 24×7, driving teams to improve workflows, processes and collaboration.

“Testing teams used to be in one building or on one floor, but now they’re spread out all over the world as a result of globally distributed teams and outsourcing,” said Samir Shah, CEO of Zephyr. “When you’re working in different time zones, traditional methods of waiting for reports and facilitating status calls are inefficient. Project teams need instant information to stay on the same page at all times.”

All Zephyr products—Zephyr Enterprise Edition, Zephyr for JIRA and Zephyr Community Edition—measurably improve productivity and software quality.

Zephyr Enterprise Edition Improves Complex, Global Test Management
Zephyr Enterprise Edition manages all aspects of software quality, and is suited for departments and enterprises that run multiple projects in parallel, have structured testin cycles, and do automated testing or a combination of automated and manual testing. It offers full two-way integration to JIRA.

“Project teams still behave as if they work in the same location,” said Shah. “They have long conference calls that substitute for meetings in which they ask for status reports; we automate these types of manual processes.”

Using Zephyr Enterprise Edition, data-driven decisions can be made while the information is still relevant. And, when anyone anywhere makes a change to a requirement, test case, assignment, defect or document, the data is automatically pushed out to dashboards so stakeholders can view the current status.

“Updated information is automatically displayed on your dashboard, so you don’t have to press the refresh button or ask someone for it; the data finds you,” said Shah. “Traditionally, testers have had to send their daily reports to managers who package and send the reports to project managers. PMs then extract pieces of data and send it up to management. The whole process is fraught with latency and inefficiency.”

With Zephyr Enterprise Edition, there are no reports to run or metrics to calculate. It all happens automatically. What’s more, the software includes a knowledge base that stores apps, release versions, features, functions and tests so test assets can be reused. It also offers built-in APIs for tight integrations to the user’s preferred automation frameworks, providing manual and automated test metrics in a single view.

“People don’t think about the knowledge that walks out the door,” said Shah. “If you do not have a single repository that everyone is accessing, you lose a lot of efficiency, especially if you’re working in a globally distributed fashion.”

E-commerce platform provider GSI Commerce leveraged both JIRA and Zephyr Enterprise Edition to manage testing across 30 projects, 90+ releases, and 12,000 test cases spread across a team of 280 people in three global locations. The company can now easily access and reuse testing assets globally; create custom reports; and customize the workflow in one system without losing the customization in the other. Zephyr Enterprise Edition is available both as a SaaS solution and as an on-premise solution.

Zephyr for JIRA Provides Seamless Test Management
Zephyr for JIRA is a native add-on built exclusively for the JIRA 5 platform. It completes end-to-end project management in JIRA by adding testing to the overall planning, development, bug-tracking and reporting processes. Zephyr for JIRA is best suited to project teams that want to seamlessly integrate testing into their workflows.

“Zephyr for JIRA gives you sophisticated testing capabilities inside JIRA,” said Shah. “Project teams that have standardized on JIRA want to incorporate testing within the exact same interface. We released it a year ago and, since the day it was launched, it continues to remain as either the No. 1 or No. 2 Top Grossing add-on each week on the Atlassian Marketplace.”

Transportation and language service company ProCare used Zephyr for JIRA to add more structure to its testing process and to create a repeatable process. Before using Zephyr for JIRA, ProCare had been using JIRA, Confluence and spreadsheets. Because a lot of the work was manual, bugs would surface after releases. Now, instead of guessing when test cycles might be completed, ProCare can instantly view the top defects affecting tests, and use burndown charts to predict when test cycles will be completed right within the JIRA interface. The company is benefitting from more detailed analyses of bugs, easy navigation between test cases, and the traceability that comes from linking test cases to issues.

Get Zephyr Community Edition Free
Instead of foregoing test-management tools because the out-of pocket costs are too high, thousands of users have turned to Zephyr Community Edition, which is free for project teams of up to 10 users who want the benefits of real-time updates and the ability to build a knowledgebase. The community edition can be easily converted to Zephyr Enterprise Edition with a new license key. “We decided to offer a free version for small teams that needed the functionality, flexibility and maintenance of an enterprise-class solution but were only working with an open-source budget,” said Shah.

Learn more at www.getzephyr.com.

The test lead of the team shared with me that they were working on a project that was using a waterfall methodology. The test team had three weeks to perform the final regression test at the end of the release. The regression test plan that they presented to management did not include any UI validation testcases. At the time, based on their available resources and their assessment of test type priority and risk, they felt that other types of testing were higher priority. After a discussion on their release development cycle, the management team asked them to include some UI validation testcases for functionality that had been developed earlier in the release cycle–based on their concern that this functionality could have been inadvertently changed by more recent code development.

Since they had fixed testing resources and knew how long it would take to execute each test, adding the UI validation tests meant that some other tests that they had put into the proposed test plan had to be removed. After another discussion, the management team agreed what tests they felt they could remove. By the time they signed off on the test plan, the management team had a good feel for what would be tested in the regression test cycle, what would not be tested, and most importantly, what the risks were of these tradeoffs.

Unsurprisingly, development ran late, but the release date could not be extended. This meant that the regression test cycle was compressed from three weeks to two. This change prompted a revised proposed test plan from the test lead and another management meeting to discuss the risks involved with the changes. As before, after discussing the risks and other factors, they added some types of tests to the plan. But since they knew the execution time of each test, they also had to remove some other tests. However, since they had priority and estimated execution time for each test, they were able to make decisions based on concrete data and manage risk as well as possible in a difficult situation.

“Testing teams used to be in one building or on one floor, but now they’re spread out all over the world as a result of globally distributed teams and outsourcing. When you’re working in different time zones, traditional methods of waiting for reports and facilitating status calls are inefficient,” said, Samir Shah, CEO of Zephyr. “Over the last 2 years, Steve Kehaya has helped us expand market share in Europe by effectively helping customers overcome these evolving challenges. We are confident that his extensive sales leadership will help us transform how development and QA teams in Europe collaborate to develop higher quality software.”

Steve has spent his entire career focusing on introducing US enterprise software technologies into EMEA. Prior to Zephyr, Kehaya served as Director of Sales for Neon Enterprise Software, where he established and managed resellers in France, Spain and Italy and was Neon’s most successful region for new business over two years.  Before Neon, Kehaya was VP of EMEA at Indicative Software, responsible for selling direct and establishing resellers across Europe. He successfully led strategic sales at large corporations in Scandinavia, France and the United Kingdom, signing customers such as; BUPA, Easynet (News Corp), Norwegian Government and Norwegian Power and was Indicative’s top performing sales person worldwide.

“I am excited to be a part of the Zephyr executive team,” said Steve Kehaya. “I appreciate the value that Zephyr puts on its customer relationships and look forward to nurturing those relationships even further to continue Zephyr’s success in Europe.”

About Zephyr
Zephyr is a leading provider of on-demand, real-time enterprise test management solutions, offering innovative applications and unparalleled, metrics based visibility via real time dashboards into the quality and status of software projects. The feature rich solution addresses today’s dynamic and global needs across a variety of industries including finance, healthcare, mobile, IT services, and enterprise software. Zephyr’s global customers experience improved productivity, faster time to market and dramatic cost savings.

What is Continuous Integration?

It‘s no secret that most or all of Agile practices are aimed at reducing risk and improving the feedback cycle, right? Well, Continuous integration is no exception. As the name suggests, continuous integration means that developers integrate their work with the mainline very often, continuously.

What are the advantages?

Continuous Integration reduces risk as integration problems are small (as small as the integration frame) and detected early (Of course, it’s not the same to integrate 4 hours of work to integrating 3 days of work with a mainline that surely contains much more added work in that amount time!). Continuous Integration also improves feedback as the automated tests ran after each integration provides insights into how things went with the integration.

How it should be implemented?

Let’s see an example from the point of view of John, a developer in project X:

1. Before starting building a new feature, John needs to make sure that he has the latest code from the repository. Forgetting to do this may force John to integrate his work with work committed even before he started!
2. John works hard on his assigned task. What should be the length of this frame? XP suggests that between 4 and 8 hrs is fine. Always integrate at least daily! Of course, John adds tests for all the new functionality added. 2a.) Then John finished his work, he needs to pull the changes from the repository again. Why? Because there could be new changes in the repository and he needs to be a %100 sure that his changes won’t break anything. So he pulls the changes, merges and run all tests.
3. Then John is sure that his changes didn’t break anything, he commits his changes to the repository.
4. After each commit, a CI server (which is a server independent of any dev computer) is triggered to automatically check out the latest code.
5. CI Server compiles and run all tests.
6. 6. After the tests ran, the CI Server updates Zephyr Enterprise Edition via its APIs with test results. The system displays the data to the entire team with real-time metrics (devs, qas, managers, etc) of the results. If there’s any failure (compilation or test failure), the build should be fixed as a top priority and no commits should be performed until doing so (it’s not a good idea to add code to a broken codebase). You may be thinking that this should never happen (sure enough the developer ran the tests locally before pushing) but believe me, it does! There could be environment issues or even that the developer forgot (we are all humans after all). In any case, the CI Server enforces the whole process and reminds developers to be more careful.
7. The test result data is automatically pushed to all stakeholders via dashboards and metrics.

How do we keep the rhythm?

The Agile rhythm could be considered hectic (especially at the beginning). Small iterations, short integration frames, new features with production quality delivered at the end of the iteration (while the rest keeps working!). How do we do this sustainably? First, invest in automation. The build should be automated, tests should be automated, creation of reports, etc. Anything that saves us work. Secondly, having the right tools: a good source control system (svn or git), a building tool (like maven or rake), a continuous integration tool (like Bamboo from Atlassian that integrates perfectly with Zephyr) and a test management tool like Zephyr to maintain all testing information and visualize status and progress correctly.

Conclusions

Continuous Integration is one of the easiest and most popular Agile practices today. It helps reducing risk and improves feedback (which leads to a product with better quality). Continuous Integration needs to be paired with automation and the right tools to obtain its full advantages.

Atlassian pre-released JIRA 6.0 EAP 1 on Monday, December 17th and made it available to OnDemand deployments the weekend prior. On the day of their launch, Zephyr Enterprise Edition was the only Test Management solution that was fully compliant with the pre-release of JIRA 6.0. As of Monday March 4, Zephyr is still the only Test Management solution compliant with JIRA 6.0 EAP 5. As a featured add-on, Zephyr continues to be the only Test Management connector to be bundled with JIRA OnDemand.

We updated the Zephyr Enterprise Connector for the pre-released JIRA 6.0 download which can be found on Marketplace. For JIRA OnDemand customers, the connector is already preinstalled on JIRA, so you do not have to do anything, it automatically gets updated.

Why is this important? Simply put, we give project teams the peace of mind that their JIRA upgrade will not impact the collaboration between Dev and QA since the integration between Zephyr and JIRA will continue to work. With JIRA being the pre-dominant issue tracking solution, we’re committed to not only ensuring this but to continuously look at ways of enhancing the integration between the two systems.

Historical releases:

• Zephyr remains lock-step with JIRA 5.2 release
• Zephyr remains lock-step with JIRA 5.1 release

Drop by and have a beer with us or win a t-shirt, we’d love to meet you. The event takes place from 5-7pm, at the Kixeye Headquarters in San Francisco, located at 333 Bush St. You can register on their AUG page.

This user group is open to anyone interested in Atlassian products and would like to share knowledge and mingle with fellow users. The SF Bay Area User Group serves the San Francisco Bay Area with the following objectives:

• Share and learn Atlassian product knowledge, best practices, and case studies with fellow current and prospective users.
• Address the needs of both Atlassian product system administrators as well as novice and experienced product users.
• Network and socialize with fellow Atlassian product users and partners.

Tomorrow’s Agenda:

• Welcome, Mingle, Snacks & Refreshments
• Brief Introduction by SF Bay Area AUG Leaders Raju and Nicholas
• Presentations

Presentations:

• Dave Meyer, Marketplace PMM on Codegeist
• Matt Hodges, Confluence and Team Calendars PMM
• Dan Chuparkoff, JIRA PMM

Here are the next three webinars in the Get More Testing Done series:

How to Organize your Test Repository for Success
Tuesday, February 5, 2013, 8am PST - Sign Up

This webinar is for Test Leads that need help setting up and organizing their test repository. Organizing your test repository is the foundation for efficient test management and will directly impact how testcases are scheduled and executed. We will cover five real world examples of how project teams organize test plans, testcases, test scripts, forms & tools, and test reports. Sign up for this webinar to learn organizational best practices and see the available options for configuring the system.

Improving QA collaboration – getting the entire project team involved
Tuesday, March 19, 2013, 8am PST - Sign Up

The various stages of application development and release are no longer silo’d tasks (they should have never been, in our opinion).  Today, QA is a part of a release cycle that is usually a few weeks long if not a few days long.  In such focused, compressed development environments, it is crucial that all members of the project team – the project managers, business analysts, developers, QA, release team and executive mgmt – have real-time visibility on the progress of testing so that the team as whole can make critical business decisions accurately.

If your current test management system has your QA team stuck in a silo, leaves a significant burden on your QA managers and leads to update various project team members manually and makes it cumbersome for the QA team to easily share test assets, attend this 30 minute webinar to see how Zephyr Enterprise Test Management not only significantly increases productivity of your QA team, but also engages the entire project team.

6 key QA metrics and how to get them in Zephyr
Tuesday, April 9, 2013, 8am PST - Sign Up

Over the years, we in the Zephyr Professional Services team have interacted with many of our 900+ customers worldwide and gathered the key metrics various organizations are using to track their testing progress.

Whether you are new to Zephyr Enterprise or are a seasoned user, attend this webinar to see what metrics QA teams across varied industry verticals are viewing as crucial to the successful rollout of applications and how they are tracking these using real-time metrics and reports.

Catch the recordings of previous webinars in the series:

• How to Organize your Test Repository for Success 

When I started my first real job, it came about because of a happy accident. I was working for a temporary agency that sent me to a small networking company. They were growing fast, with so much work to be done that they were open to anyone with drive and ambition coming in to do it. I had a chance to talk to a number of the testers, and they showed me their lab. As I walked through and looked at their setup, I saw devices that I would later learn were Ethernet and token ring hubs, serial connection boxes, routers and, along the walls on various desks and benches, computers of various makes and models. There were PCs, Macs, UNIX workstations, a DEC VMS minicomputer, and a few other exotic devices. The part that got my attention, though, was the bank of patch panels in the middle of the entire setup. Patch panels are a key and critical part of wiring up a recording studio. Without them, you have to unplug and replug a lot of devices in different ways. With a patch panel, you can easily make connections as you need them, and then break them down when you don’t.

Why do I mention this? It’s because that one simple insight was enough to help me explain how I could help this testing team get a handle on their test lab. I could rewire it, I could make it more efficient. I could make the setup more robust and easier to configure. Why did I think I could do that? Because I knew how to wire mixing boards and outboard effects. I understood the basics of signal processing, and how devices, if they are chained over too great a distance, can degrade sound or introduce noise into the mix. I had knowledge of a particular domain (signal processing from a sound studio and a musician’s perspective) and I also knew a bit about how to configure those devices through their small one inch console screens. Could I leverage those same ideas into designing a networking lab? Understand I had no real knowledge of networking. It was a gamble, but I figured I had little to lose, so why not go with it?

Because of that experience, and a lab administrator willing to indulge me, I set about learning all of the equipment in the lab, and how it fit together. What would it take to make end to end connections for all of the machines? What needed to be in the middle? What devices did I have to learn how to configure? What came down to just wiring? Could the patch panels really be a key to unlock this lab, and its domain, so that I could be effective? The answer proved to be yes.

Over the ensuing couple of years, I learned how networking equipment was similar to, and different from, sound equipment. I saw how ideas such as switching and routing of packets, and the use of various networking protocols, would let me set up networks so that Macs running Appletalk could connect on the same networks as PC’s running TCP/IP, or Novell, or Banyan Vines. I learned how to configure a router to act as a firewall so that some traffic could come in, but other traffic could not, as well as how to contain all but certain connections to go outside of the lab. I learned how to make simulations of multiple network segments connecting together, and using routers and switches to enable traffic to flow from one network to another.

It was heady and fun to work in this environment. Over the course of two years, I got pretty good at being able to configure a broad array of networking setups. As the company grew, and the testing team had a need for more people to do even more testing work, I was asked to join the team as a full time software tester. Thus, over the next seven years, I would get to know small sections of our product at a given time. I would learn about the route processor and what it could do, and what needed to be tested. I would learn about interface cards, and how they were expected to behave under normal and adverse conditions. I would learn about network management protocols like SNMP, and how to manipulate MIBs to get information from systems. I would learn about content delivery devices and caching, and how to measure the load time of various web pages and content with or without these devices.

Networking and sound systems had a small amount in common, but there was enough that I could leverage and use to my advantage. I came to believe that I could test anything after these experiences, that I had a knack for testing, that I understood how things worked, and that I could carry those ideas anywhere. I developed a bit of “hubris” regarding my skills. Hubris is a Greek word that means “a pride in one’s own accomplishments or skills.” Often, it’s not wholly warranted. The ancient Greek philosophers and storytellers often paired hubris with another term, “nemesis.” Nemesis is punishment or retribution meted out to those who commit hubris. I would find out soon enough just how appropriate those terms were.

After I had been with the networking company for a decade, I decided to see what I could do in other environments. I felt confident I could test any software project or product. That is, until I found myself working for a company that made capacitance touch devices. What are those? If you have ever used a touch screen, or a touch pad, or a little eraser sized rubber circle in the middle of your keyboard to navigate, chances are, your using a capacitance touch device. I went to work for a company that was actively involved in developing prototypes and models of these devices for large computer manufacturers. Networking was not very important here. The laws of physics were king. I will hereby confess that my knowledge of physics was woefully inadequate. Still, I believed that I would be able to leverage my testing skills and my previous experiences, and I’d just pick up the rest of the details along the way. While I was partially right (I was able to learn quite a few things, and I was able to test somewhat effectively) there was one big gap that, try as I might, I couldn’t effectively overcome. I saw that there were unusual spikes and variations when I subjected a device to a shaker table test, but I couldn’t effectively communicate the data I was receiving, or present it in a way that was useful. I found myself relying on the hardware engineers to write little tools for me to run the resulting data through and create reports from the data collected from long term vibration runs, deep freezes, extreme heat, pressure tests or utilizing electrostatic discharge. While I could set up those tests, what I couldn’t do was effectively synthesize the information I was receiving, not without a lot of help.

Over time, this became obvious to the director of our group, and I was called into his office one day. When a discussion starts with:

“I’ve been looking over your resume, and you are not a hardware engineer. You’re a software tester who worked on networking equipment. How did you get hired here?”

Well, the writing was on the wall. Needless to say, I found myself looking for other opportunities. My hubris had met its nemesis.

This was a golden lesson to me. It taught me how domain knowledge is often the secret sauce that makes the difference between being just a good tester and being a great tester. Knowing the fundamentals about how to test is one part. Knowing how software is created and understanding the logic of systems is another. The most important aspect, though, is the in-depth knowledge of the field that you are working in. Often, you cannot make a leap from one field to another without preparation or exposure to that second environment over an extended period. I came into the networking world with a relatively simple skill. A kid with some peripheral experience wiring up recording studios was able to leverage those skills as a lab administrator. Over time, I also slowly learned the world of computer networking, as well as software testing and what made for good testing from my peers. The domain knowledge didn’t just come in one fell swoop; it came because I worked with it daily. I wasn’t expected to know networking, but I learned it over time. In the job where I was testing the laws of physics, I was expected to know all that, and I didn’t.

So what can we do if we want to work in an environment where we don’t have the domain knowledge to be effective? How do we gain it?

First, ask yourself why you want to be involved in a particular industry. If you are really interested in a given field, do all you can to learn about it. If you want to work in an area where chemical engineering is an important part of the job, then you will need to go and learn about chemical engineering, which will likely mean you will need to understand peripheral disciplines, such as chemistry, biology, geology, and mathematics. Determine if you are willing to go as deep into those fields as is necessary to be able to speak in a meaningful way about those topics. If you are looking to test a web site that is dealing with very large data collection, then you will need to know how massive databases are created and optimized. A book or two might be all you need to get up to speed enough to be effective. If you want to learn about a technical topic in a fun and interesting way, I’d suggest looking at “The Manga Guide” to series of books offered by NoStarch Press. These take what can be challenging topics, and help break them down in a way that makes them accessible and understandable to a broad audience. They should not be your only source of information, but they can certainly be an excellent start (I wish “The Manga Guide to Physics” was available to me in 2002).

Second, ask others, who already work in the area you are interested in, what they do every day, and what they are expected to know. If you want to work with testing small devices like mobile phones, you may find that you are expected to know a bit about electronics, and how to do basic rework on a circuit board. I don’t mean design or fabrication, but you may find you need to know how to solder a lead wire from one point to another on a very small surface. Yes, software testers may be asked to solder occasionally. That is a domain specific skill. It helps to know when you are going in to a job that requires it if you have a steady hand or not. If you don’t, know that going in, and be honest. If you don’t want to let that stop you, then invest in some simple electronic kits and practice soldering for awhile until you can do it smoothly.

Third, read about the greater trends going on in your industry and determine what publications are available, or what web sites can be accessed that deal with these given fields. Right now, I am working in an environment where a primary customer of our product is the U.S. Government. Because of that, we have special requirements we have to meet with regards to accessibility. This refers to the ability to use peripheral devices and software so that people with disabilities related to vision, auditory or tactile interaction can still use our products. There’s an entire section of a law that is dedicated to this. It’s called the Section 508 Amendment to the Rehabilitation Act, and yes, for my given scope and work, I need to be familiar with it and what it means for our products to be compliant with it. Thus, it falls to me to learn these regulations and put them into practice. You may find that similar needs exist in the area you are interested in working in.

Domain knowledge is, ultimately, the make or break item that determines if you are successful. Testing techniques can be learned and refined with practice and effort. Coding knowledge can likewise be learned and practiced, and expanded over time. If, however, you are deficient in the domain knowledge of your chosen area of testing, no amount of coding or testing skills will overcome it. The good news is that you really can learn about any endeavor if you really want to. Some areas will be more difficult than others. I wouldn’t recommend applying for a job at a biotech company if your background doesn’t already include a fair amount of bio science. Still, if you are determined, and you have a genuine interest, there’s a lot of information that’s just a mouse click (and some steady reading and practice) away.

• How Svante was able to easily customize the integration between Klarna’s Jenkins farm and Zephyr/JIRA
• Access to clear requirements traceability for more efficient decision making
• The benefits the Klarna team achieved with reduced reporting times (down from days to under an hour!)

Click here to view the embedded video.

Highlight Reel:

• Overview of Klarna
• Klarna’s challenges
• Demo of Klarna’s Jenkins farm
• The results

eggPlant is a commercial blackbox GUI automation tool, built by TestPlant, in which can record and replay complex test scenarios. eggPlant works on the principle of image recognition and runtime matching and so interacts with the system/application exactly as a user does making it very easy-to-use but still as powerful as other test automation tools. eggPlant launches tests via VNC and tests can be launched from anywhere as long as test machine has a VNC server running; so eggPlant can work with any kind of GUI and is non-invasive. eggPlant comes with a command line utility to kickoff tests.

We created a integration to eggPlant with our agent ZBot. This integration allows ZBots to kick off eggPlant tests. Once test execution completes, this ZIP reads the output xml to find the test status and in case of failure, it also extracts failure description. This information is sent back to Zephyr and all corresponding desktops and dashboards are updated automatically in real-time.

To configure your automation environment with ZBot and eggPlant ZIP, the detailed procedure is on our wiki.

In short:

• Configure path of your test in Zephyr Testcase
• Download latest ZIP from github.
• Install ZBot.
• Configure ZBot properties and classpath.
• Kickoff tests.

eggPlant is a great GUI testing tool and it’s been a pleasure to work with it and to develop a integrated solution that would enhance the meaning of both the tools for our users. Thanks for reading this post and please feel free to contact us with your issues / comments or chime in on our documentation page (Git Pages).

Check out this video to see the integration in action:

Click here to view the embedded video.

Today I had the pleasure of speaking with Vladimir Prymakov, Kuadriga QA Manager who provided a very detailed analysis of how JIRA, Greenhopper, and Zephyr for JIRA allow his project teams to incorporate SCRUM testing into their JIRA workflows.

Q: Can you tell us a little about Kuadriga?
Sure, the company started in 2008. We use a combination of industry expertise and technology to provide clients around the world with custom software development services.

Q: What type of products does Kuadriga develop?
We develop a wide range of products; web, desktop, mobile and other applications for financial, marketing, insurance, medical, and other industries. Technology choice is mostly justified by clients’ preferences and our suggestions. We take into account the value the technology can bring to the projects.

Q: Which Atlassian tools do you use and how did they make their way into Kuadriga?
Now we use JIRA, Greenhopper, and other Atlassian tools. We have much experience with many other tools, such as Microsoft Project, Trac, Bugzila, Mantis, RedMine, Podio and more. We tried JIRA and Greenhopper on recommendation of our new colleagues on one pilot project. The experience convinced us that these instruments solve most issues in agile development and management. Since then, Atlassian solutions, including Zephyr, became standard and default in Kuadriga.

Q: Before Zephyr for JIRA, what tools were you using for testing?
We used TestRail and Excel for test management. Now for the projects maintained in JIRA, we decided to use Zephyr add-on. It enables us to have everything in one place and tightly integrate testing in the development process. In addition, advantages of Zephyr are an easy way of tracking test coverage, monitoring testing results, storing cycle history, convenient and easy test case management, etc.

Q: Specifically, what challenges was the team wrestling with the previous approach?
Well, when we used Excel, the main challenge was that we had to provide concurrent web-based access to test matrices and the corresponding test cases, link them to requirements, bugs, etc. Also, a lot of time was spent on manual activities. With TestRail, we spent more time on switching between tools. None of the other tools provided us with such an easy way of linking tests, bugs, and requirements, as Zephyr for JIRA.

Q: There are several test management solutions out there, why did you choose Zephyr for JIRA?
Here are the main reasons:

Integration Inside JIRA: This is huge. As testers, we live in JIRA, as well as developers, business analysts, and designers. Using Zephyr for JIRA, we don’t have to switch between tools to manage testing any more. Also, we can easily link tests to requirements, user stories, bugs, and vice versa, which increases traceability a lot.

No Learning Curve: We do a lot of collaboration with the development team in both test creation and test execution.  Because developers already know and use JIRA, transitioning to Zephyr is seamless for them. It goes the same way for the testing team using JIRA.

Just Enough Metrics: We are intentionally light on the process, documentation, and reporting. We do just enough to get our jobs done efficiently. Here are the key metrics we get out of Zephyr for JIRA:

• Test Coverage: By release, by test cycle, by component, by defect, etc. Using labels creatively, we can implement literally any coverage scenario we need. These views of coverage give us just enough information to communicate our level of testing coverage, and analyze risks based on the development changes.
• Test Status: Through the various gadgets and dashboards in JIRA, we get just enough useful information: test execution status by cycle and user, top defects impacting testing, defect distribution by test, and test execution burndown.
• Test Cycles implementation within Zephyr for JIRA is pretty simple, straightforward, and effective.

The Price: The price for Zephyr for JIRA is much more affordable compared to other test management tools with similar functionality and quality.

Q: In terms of measuring the benefits, how do you quantify the impact Atlassian tools have on your project team?
Primarily through utilization. We do not spend a lot of time on analyzing the benefits after the fact, but rather we ask ourselves the following questions:

• Do people use this product/feature on a daily basis?
• Does it help them to do their job?
• What are the processes that do not work or need refinement?

Q: In terms of measuring the benefits, how do you quantify the impact Zephyr for JIRA has on your project team?
Compared to our previous approaches, the scope of our testing and development effort with Zephyr for JIRA dramatically decreased and the testing process became much more transparent and traceable. Also, our QA team is very happy to work with a professional tool that helps them do the job efficiently.

Founded: 2008
Headquarters: Denmark
Industry: Software Outsourcing
Employees: 85
Tools used: JIRA, Greenhopper, Zephyr for JIRA
Website:  www.kuadriga.com

One could see the following root causes behind those inefficiencies:

1. Different ways of thinking by Dev and Ops team
2. Uneven staging and Prod test environments
3. Uneven thrust given to testing
4. Absence of Ops team involvement right from the beginning of the development cycle

Let’s start with the first one; development teams are trained to think in a siloed fashion. They get user stories from the product owners, who in turn guide them till the completion. However, Ops teams only look at the integrated, end to end working and tested application. In order to reduce the gap between the teams, project teams (including testers) should be trained to keep the end goal in mind.

Many companies I have come across have different staging and production environments. Right from the OS to deployment tools and directory structures differ.  This causes too much stress when the undeployed code goes to the final stage of testing.  It is critical that environments match. Stackoverflow has some good discussions around this topic.

Coming to the topic of unevenness in testing, I have noticed that a lot of time and resources are spent during the Dev testing as compared to Ops testing. From the purist point of view, these testings shouldn’t be separated and should be done from the beginning itself, but many companies don’t have the maturity yet to attain the purists state.

I have been working with several large enterprises who frequently release products to market. Some have 3 months and others 6 months cycle (I know that these companies are far from how Facebook releases the code twice a day).  When I say large, project teams of size > 200 with several vendors, dozens of testers, developers and business analysts working on large pieces of functionality on respective code bases but successfully integrating just in time before going live.

The process in these large organizations have reached maturity levels that the release managers across vendors meet on a weekly basis to sync up time lines, functionalities, code bases, etc.  This has resulted in pretty smooth deployment time and again. These companies engage the production support and management team (IT Ops team) right from the initiation phase of the project. Both Dev and IT ops teams sit together, estimate and write the hooks to monitor the application. Continuous testing at unit, integration and system level  are done at the end of each sprint.

To conclude, most of these Dev-Ops related issues could be resolved by proper test planning from the beginning of the dev cycle and involving the Ops team right from the requirement gathering phase.