Test Policy

I am quite interested in this testing artifact and how many people out there actually have one. This is a document that is very short maybe just a page and it highlights at the highest level what the business itself expects from testing.

I have worked in several places that dont have this and all of them seem to have the same issue testing seems to have a free reign and little focus. Within a company every member of the testing should have an awareness of this document. I am a firm believer that this should really be a bit like a cub scout promise.

I will amend this at a later date to add further information on what should be in a test policy and how to write it.

ISEB Intermediate Software Testing Exam

For those of you that think that this is just a piece of paper and proves nothing then I believe that you are wrong. Having just completed this exam I was pleasantly surprised to find how difficult this  actually was. All the info learnt and all the theory are for nothing in this if you cannot apply your knowledge to real situations.

It means that rather than just having a load of techniques, that you are not sure where to use, you can now identify what techniques to use in the planning stages of testing. For me it has given the insight to be able to greater understand a test analyst and test manager role. I also feel i have a best practice to aim for in my work.

One topic that I found especially usefull was the sections that covered risk. This will aid me on a day to day basis with scoping out the areas that I will be able to test.

Agile

You know what really grinds my gears. Not people pointing out problems with Agile or even people deciding that it is not suitable for them. What grinds my gears is people creating problems with Agile without understanding what Agile really is and what is involved. For testing staff it could be a bad thing as many implementations see no place for professional testers but I would see this as an oportunity. By pair programming with one tester and one developer you get the best of both worlds with everything seen from two perspectives.

Last Minute Testing

Every time a release comes round it always ends up leading to last minute testing. For me this time it started with me coming back from holiday and having to test two systems that should already be in test. Had I done any data or test prep the clear answer is 'NO' was all the correct and accurate documentation provided the clear answer again is 'NO'. 

So how to approach this in a way that doesnt compromise my work. There are many ways but two that i will mention here.

1/ You could on the basis that it is in test and the correct documentation and requirements are not provided reject the application hand it back to development and remove it from the release.

2/ You could create a document that scopes what you will be testing and why based on the information that you do have and provide this information to the project and development leads. This will mean that you have provided clear details on what you will cover. Following this you should also produce a risk log of some description that is also provided to the project and development leads. This risk log should contain information of what was not provided to you and any areas that you fear might have been missed, it might even state testing was restricted due to the lack of time provided.

Some people would say that this sounds like covering your own back should anything go wrong. I would say that this is ensuring that you maintain your integrity and demonstrate you have done the best job that you can. Also listing the risks in this fashion brings them to the attention of project leaders so that they might be able to come up with some kind of mitigation.

This kind of information being produced and circulated means that should anyone question your competence or your coverage. You can provide all the documentation as to the process that you have followed and why. 

Planning time

Sorry I have not posted in a while but been looking after number one and enjoying my honeymoon.

Consequently this post came to me on holiday.

There are many things that should be considered when managing testing and one of the most important is the issue of resources. Even when you think you have it all done and everything scheduled into the correct deadlines there is always something that will change. It might be that someone goes off sick... it might be that you missed that someone was going on holiday or even some family berevement. The long and short of it is that your plans must be flexible and should even when tight deadlines are being worked to hold a measure of contingency. If you cannot have any contingency then your testing should be planned in a way that you know what all the high priority test cases are that absolutely must be covered.

This will mean that even for emergency changes you can have a good measure of assurance that the system will function as desired.

Out of the frying pan into the fire.

Have you ever thought that i would be good to say.

• We wont take things that arent unit tested.
• We will only be testing versus assessed risk.
• We will be reviewing all documents early and if not acceptable we will not allow development.
• If development is late then testing will be late and it wont go in the scheduled release.
• If its last minute then it better be a live issue.

Well the lion share of this is what the team I am part of are going to be doing soon. I fear that this is likely to start out being a rough ride as it is not what everyone is used to but it seems that it might well in the long run greatly improve things for everyone. Ultimately our team have little choice as currently we have little resources and alot of project and maintenance work to cover.

Testing danger!

I was sat in the waiting room at my local garage today and had some thoughts about testing and where we would be if people didnt take effort to ensure accurate testing. What would it be like if you had your MOT done then two hours later died in a crash cause by your brakes failing. In response the garage said "Well it all seemed to be passing so we just passed it only missed a few bits out." 

I think the main value in testing comes from doing the best job that you can to ensure quality and covering as many areas as you can in the time given. Then taking the results that you have and formulating a recommendation to proceed or not to proceed.

Deadlines and estimates

It has came to that time in work again when the questions start being asked. 

When is the testing likely to be complete?

How much testing is left to be done?

How is the testing going?

and many more questions. The reality is that you can only give a best guess especially since as testers we are relying on the rest of the development team to come up with the goods for us to test. For me today I took an educated guess that it should be finished just about on time.  I would say be clear with what you are aiming to complete and what you have already completed and then they can make a judgement call aswell on the likelyhood of something being completed. 

Most of all dont just allow something through that doesnt meet the expected!! 

Waiting...

3 days late testing and moving into day 4... I wont be starting the testing till tomorrow. In this case it is not my fault or that of the developers but some important things to consider are:-

* Smoke test the environment in advance to ensure that it is working correctly.
* Where possible smoke test the application/system in advance (even if it is an early build) to check that it is going to work.
* Regularly schedule your testing to fit in with your current testing progress.
* Regularly monitor your testing activities to keep all parties up to date with current progress.
* Keep contact with the developers and business analysts to pick up on any changes to the design and requirements. These can greatly impact the testing activites and you will need to know about this ASAP.

Emergencies

For this my thoughts are why does this make any difference.

Why is it that issues encountered in the live systems get fixes implemented with often little or no testing? Is it maybe a flaw in the testing we do or is that what everyone does?

this is what i would like to see:

Requirements - produced by the business as to what they expect

Coverage - amount of testing needed to provide assurance

Approach - Quick single page on the approach to providing appropriate testing.

Scripting - Test scripting for what is going to be covered likely to be confirmation testing and limited regression testing.

Execution

Sat once more in system test execution and once again having the same old problems. I am a week into the system test period and already around 2-3 days behind schedule. What is more due to the tightness of deadlines we have no contingency... or at least if it slips I wont be there cause I am doing something that I cannot move.. getting married!!

SIGIST conference - what did I take away?

The conference yielded a surprising array of different ideas from the tried and tested to the conceptual ideas. The items that caught my interest big style are highlighted below.

Codependency
So what do i mean? Why does testing accept the bum rap that it gets in brief as the title explains its called codependency.

Every time that we accept:-

Late handover of software to test
Software that is not unit tested
Missing requirements
Taking something not conforming to entry criteria
Signing off on something not meeting exit criteria
...to name a few

We are expected to:-

Test to crunched deadlines
Work overtime
Signoff on something with no confidence in it

Result in the long term:-

We take the rap when things go wrong that we had no time to test.
We end up expected to work overtime every release
We always get late releases
Testing is always crunched
Nothing is unit tested

This is written in my words but based directly on the talk presented today by Lee Copeland from Software Quality Engineering at the SIGIST conference

Requirements

My personal experience is with Requisite Pro so I will likely make any examples relate to this and MQC but it could also apply to any equivalent system.

If you hold your test requirements along side your business requirements within you requirements management system, then they can be synchronized so that areas of contention would be automatically identified should the business requirements change. These test requirements can then be fed through to MQC in a read only format for usage in providing the requirements linking.

For me one of the key things about doing this is that the test requirements (ie the translated business requirements) can be baselined alongside the business requirements. This would also allow the provision for metrics to be gathered on the number of requirement changes and how much those changes are directly effecting testing.

This is written in my words but based directly on the talk presented today by John Cheesman from Strat Software Ltd at the SIGIST conference


The potential issue with all this is that the requirements NEED to be maintained for this to function if the requirements are far beyond what is currently being worked on then this will be useless. But ask yourself a question “If we are testing to provide a measure of software quality back to the stakeholders, then how can we do this without requirements. The requirements are what they want”.

History of Testing

Few of us, me included know much about the history of software testing and those names that designed the core principals of software testing that we now use to base all of our methods on. Names such as Dave Gelperin, Glenford Myers, Boris Beizer, Bill Perry, Michael Fagan and Tom McCabe.

Further to this it is also startling how few testers have even picked up a book on software testing let alone read one. I think the continued learning and development of people in any profession is key to the knowledge of the work that those people are doing.

Many people don't view work and indeed life as a continued learning experience where by study is as important there as it ever was in school back when someone explained that “Two plus two equals four”.

This is written in my words but based directly on the talk presented today by Lee Copeland from Software Quality Engineering at the SIGIST conference

The people factor

Never underestimate the power of saying thankyou for all the work done to people especially staff under you when it is clear that they have done a good job.

Ensure that you reward good work with an appropriate reward this could be financial or leave or even just a thank you as above. The main thing is that this is one of the main things that makes doing these jobs worth while. It is also important to consider that things like talking to senior management to give them credit or getting it mentioned in a newsletter can be useful providing a relatively cheap but effective reward.

This is written in my words but based directly on the talk presented today by Martyn Caswell at the SIGIST conference.

system test or UAT

It seems to be an on going debate where I work as to who should cover exactly what...

System testers end up picking up alot of the unit testing as well as the functional testing of the system. They also end up covering much of what could be held traditionally under the banner of UAT. Testing items under the business processes etc all end up  being part of system test.

Guess I have heard some not so good things relating to UAT today and find myself thinking ... if they are running the same scripts often with the same people is this testing providing any business value. Should UAT not strictly be based on the requirements of what the business wants ... Back to the core training :

SYSTEM TEST := should test a system does what it was designed to do.

UAT := should test a solution does what it was required to do.

Following a recipe

I was sat cooking today and decided that it is a bit like testing when you think about it... sad i know.

When following the recipe you have an expected outcome, a set of ingredients (Inputs) and a collection of steps that you have to follow. The smallest of mistakes in the recipe or the following of the instructions will change the expected outcome (a pasty in this case). If a mistake is made then the pasty will be burnt (a defect) then a new pasty will need to be made correcting the mistake (bug fix)  the recipe will be amended (script corrections) and the new pasty will then be rechecked so that it is ok. At then end the pasty will be checked to see if it is acceptable (exit criteria).

Maybe i will revisit this one day and name it the ... Pasty Method ;)

The truth will out

Why is it that on so many projects that I have worked on, everybody knows the project is going down the toilet? What is more amusing is that everyone knows and still seems to be unwilling to shift the goal posts.

If a project is percieved as likely to be running late come up with a contingency plan and do it early. Not only that dont be unrealistic there is no point trying to force something that is not going to happen. I am sure everyone has found themselves hoping it will all go right now and then... but how often has that happened. 

I realise that sometimes legislation changes etc can force set in stone dates.... but locating what can safely not go in could reduce the time putting the vital changes in would take.

Log those defects

I have seen this again and again, and even been guilty of it myself a time or two... dont debate a defect just log it.

If you are running test scripts and the script doesnt match the system then fail it. This can be revisited to decide if the script or system is in error and corrected. Raise defects for all these errors encountered. This will mean that no one can later come back to you and say "This test will fail it is written wrong, so why have you passed it"

The vital thing is just logging all the defects and failing all the scripts to ensure that nothing is missed.  

Death by email

So I am sat in my chair when in comes an email with a possible issue for my testing. Great they are keeping me in on the loop. Then sadly I get another and another and another as i get emails from everyone responding to the email. 30 emails later!!!!, something that i was happy to hear was leaving me with a throbbing vein. 

I think that the lesson learnt is to read the emails that you recieve carefully and think clearly about what you are responding and to who. While what you say might be usefull to you if you are including a bunch of mailing lists chances are most of the people in the list wont find it important.

While mistakes happen consistently doing it is just going to upset people.

Helping or doing the job?

Couldnt stay quiet about it forever, how comes there are always people that think if they ask you nicely, you will do their job for them? Fair enough when the chips are down I am happy to help others to get work done on time. I would hope that most of my mates in work would do the same but in this case, I have been asked to do the work for a person who probably earns twice what I do. Not only that i also get little credit for the work that is done.

I would recommend the following in this case as it seems to work well for me.

1. Make sure where possible that a team leader/manager is aware of the work you are doing.
2. Keep a copy of all the work that you have produced locally as evidence.
3. Put your name to all the work that is produced.  If this is changed and you feel strongly about it you can take issue with this based on the copy that you have.
4. Get business agreement for the work you are doing. 

Nothings perfect

Nothing is perfect and the best laid plans will need to be modified over and over in the course of a project. The only thing that is inevitable is that change will happen...just to mention a few:

1. A software code drop could be delayed.
2. Members of staff could go off sick.
3. A third party tool could not work properly.

The critical thing is to plan in a measure of contingency so that should issues occur then the tasks can still be completed in time. The other thing to do is to ensure that all tasks are prioritised. If all the tasks are prioritised then it becomes easy to quickly decide which tasks can be run in the time available and which tasks can be left out with a limited risk.

Comes with the job

When you are doing any job then the stress and strains of both work and of life outside work can take their toll at times. I know many people who at some point in their career have needed leave related to this, It is nothing to be ashamed of. 

My thoughts on the matter are two fold:

1/ Looking after yourself is always more important than what is happening at work. If you struggle on working chances are you will just feel worse. Better to get sorted and come back to work fresh.

2/ Dont bottle things up let others know what is going on and that way issues can be addressed sooner.

Trust or Not

Trust in your developers and business analysts....but at the same time plan for when things go wrong. When you get opinions the nature of things is that developers give positive estimates of when things are going to be delivered. Plan for the software to be late and plan in contingency.

If you plan with that in mind, then should things go pear shaped you can pick up the slack and get things completed correctly on time.

Business analysts give the testers the basis for testing but confirm the information and dont be afraid to raise questions over areas that you dont agree with. Nothing is worse than getting a product that meets everything requested only to find that its not what is needed.

A good example of this is a manager requesting a computerised system for a hotel booking system but only having the budget for 1 computer. With 4 people working on the checkin desk this would mean 3 people sat around with nothing to do all day. A business analyst should pick up that the current paper based booking system would be more efficient and that the money could be better spent elsewhere. If they dont pick up this then who else is there?

Requirements

It always amazes me that requirements always seem to be left out and classic statements of project issues are:

"The requirements were defined incorrectly."
"The requirements changed late in the project."
"We delivered what you wanted and this differs from your expectations."

I know this is ideal world stuff but surely with formal review and a little extra time spent at the beginning of a project a great deal of issues would be solved at the end. This would stop the developers guessing what the system should do. This would also stop the testers having little as a basis for testing and mean that anything produced would likely be acceptable.

The other main thing to consider is to produce a workable product early where possible and continually get this reviewed with the potential users and stakeholders to ensure that they know exactly what is being produced and that it is acceptable.

I think in the real world the best way to help with requirements is to pick up the phone, make a meeting, send an email and chase, chase, chase. If you dont have the information that you need make a list of what is missing and then talk to the person that has been charged with the responsability of producing this information. This will both enable you to say that you have done everything that you could and if you obtain what you need it will let you do your job more effectively.

When is enough enough?

Something that I constantly hear is that there are no requirements and no supplementary specifications etc.

Quite to the opposite what I want to say here is "When is enough, enough?"

I have worked on several peices of work lately where I find myself feeling like some of the documentation is being produced for the sake of it rather than to aid in the system development lifecycle. Can it be that sometimes a formal process that describes rigid artifacts for testing and development can make extra work? The simple answer is "Yes" flexibility is needed to enable discretion to be taken on what needs to be produced and how the process should be adapted to suit all different situations.

ISEB Foundation

Guess this is just something that you have to do to get a job in testing. With all the jobs that are around stating ISEB foundation as a requirement you just have to do it.

Dont just do it for the exam though is my tip for this. If you embrace this course and its material and utilise it all for your testing and then develop this with continued learning your skills will build very quickly. It also highlights the areas that you need more practice in and points you in the right direction to improving these areas.

I know many people who would say that this qualification proves little about a testers aptitude but if you know the material then you have a level of understanding of the basics.

If you havnt done the ISEB Foundation in software testing then you can review the sylabus here:

ISEB Foundation

Practical Test Planning

If you have ever thought of taking a practical test planning course then truly I would recommend it. I wouldnt take it if you have no team leadership duties as that and project planning is what this course is aimed at. If you are managing a project though it opens up the whole process making it clear. What you should be producing and what documents will be ideal for resource and budget estimates.

I am going to decline saying who I went with for the training as there are plenty of training companies out there offering the course and all are probably equally as good.

Wasted an hour of my life!

Have you ever been sat in one of those meetings? You sit there listening to the endless drone of pointless conversation. If you want to pass on a report then pass on the report. For a meeting to be worth while then I believe that it should follow some set guidelines.

1. Set out a clear objective or set of objectives for what you aim to achieve from the meeting.
2. Define an agenda to structure the meeting.
3. Any documents should be issued before the meeting with specific areas to be review mentioned if required.
4. A chairman of the meeting identified to make keep the meeting on track.
5. A range of people identified to fill roles if required.

If none of the above are thought about then the likely outcome is a meeting where you sit either bored or rambling about the weather and what you did on holiday. Certainly not getting any constructive work done.

Fee or Free

Why is it that, with so many free tools out there to do the jobs that we do everyday, that we still feel the need to fork out money, on tools that cost alot of cash?

For example: where I work for me to do my job according to the coorporate policy, my collegues and I need to be able to use the following tools

Web King
Mercury Quality Center
WinRunner

Not to mention all the programming tools and standard Microsoft functionality.

Do we really need all of these tools? The simple answer is NO.

Large organisations go with the well known pay for tools and this is why:

1. They have a reputation for quality so little work is required to confirm it will work.
2. Training can be easily obtained.
3. Support is readily available.
4. A tool is likely to continue to be provided and not be dropped.

For a small flexible organisation on a lower budget however the free tools are ideal and all it will take is a little research.

Priorities really?

Well, I know how to use priorities and assign them in my testing. It allows me to be able to structure the coverage that i will give based on the time and resources that are available.

It has struck me looking around that their seems to be alot of people that dont do this either because they dont know or dont see the reasons for doing this. So although I am definatly not the expert on this I thought I would list a few reasons why I think that prioritising is important.

1. Test scenarios with priority can be placed before stakeholders to decide exactly what needs to be covered and what can be left out. This can be done before any real outlay is made.
2. When testing gets crunched as it always does.. rather than struggling to cover everything in to little time, you can say "we will only be able to cover priority 1 and 2 tests in the time available as per the time scales we have already provided".
3. Tests with high priority can be executed where possible first to maximise effectiveness.
4. In large groups it can be made clear to everyone what order script production and test execution needs to be performed.

This is just to name a few things I have thought of off hand ... on a saturday night with a beer in one hand and fork of chicken fried rice in the other.

Testing is easy

Would like to think that this is true.. but it just isnt as you are reading this you will know how much hard work it can be to do testing. We dont just get a new product, mess about with it then say "yea go for it, looks good to me".

Based on requirements we plan all of our testing to be the most effective in the time that is available. We know what to expect and what is acceptable often before we have even seen the system and in a very short space of time provide a method to assess the quality of the system.

I do find that many of the people that I have touched in my job come to understand how much is involved and soon change their minds. So just why do people assume that testing is easy.

Consultants in testing

I like consultant testing staff but a few things that I have found that can be issues are:

1. Make sure that the experience matches what has been stated in CVs.
2. Consultants should be managed closely to ensure they are conforming with any corporate practices.
3. Often people within a company will assume specific business knowledge not possessed by an external consultant. Assistance should be provided to fill in this knowledge gap.
4. Consultants cost more than internal staff so should be sought with a specific purpose in mind.

New Issues of Testing Experience Magazine

Here is a great free magazine.. no I aint getting any cash for saying this it really is a good source of free information. You can subscribe or view the magazine online below

http://www.testingexperience.com/

I get the printed version myself cause i like to read it while travelling.

SIGIST Conference

For those that dont know SIGIST = Special Interest Group In Software Testing.

Just thought I would say that there is a conference for this coming up in london on the 18th September. If you look closely you might even notice me there.

if you want some more info on this
http://www.bcs.org/server.php?show=nav.9264

Unit Testing

Last week I was working on a project where the unit testing had been done by the developers. Trust me that is rare not only had they done this fairly completely in a tracable manner but they had records of it all. Suffice to say that I was very impressed.

Time being short we gave our testing estimates and it clearly wouldnt meet the deployment dates so we brazenly said for this to be tested it will need to be delayed... did they delay it ... no they took the risk to put the software in anyway. Further they also declaired that user acceptance testing was not required.

I being a pain took it upon myself to smoke test it. Within the first 30 minutes of trying to run the software it was clear it wouldnt even run in the environment. Funny thing is when it finally was working we found a unit test defect.

Guess my point with this is back to basics really.. a developer shouldnt test there own software and independant testers will uncover far more defects with a system as they are looking for problems not just checking it works.

Evidence Evidence Evidence

When you are testing a system of any kind something that you must maintain for everything is evidence. This is like the holy grail of testing.

For planning you need a record of:

• What you are testing?
• What steps you are running?
• Why you are testing?
• What order they should be run in?
• What inputs and outputs are expected?
• Who is testing what?
• How many resources?
• When to start testing?
• When to stop testing?

and many more things..

for execution you need a record of:

• Has entry criteria been met?
• What succeeded and failed?
• What tests were run?
• What defects were raised?
• How long it took?
• Current progress
• Has exit criteria been met?

We need this to be able to:

• Provide informed opinions on the quality of a peice of software.
• To display value for money of the testing process (it pains me that we would ever need to prove this... but keeps me on my toes.)
• To feed back traceable defects for correction by the rest of the development team.
• To be able to prioritise and organise a workable solution within time and budget.

etc..

but what really gets me going is that all this is required by testing.. so why is it that requirements gathering and the system development itself is allowed to relatively escape producing the same kind of documentation and accountability is always placed firmly at the testers door.

Weekends

Isnt it funny how I have been working all week planning, designing and executing tests and all that I can think of is more about work. No I dont mean to say that i am not having great fun playing with my daughter and living life up. I however have work floating in the back of my mind all the time.

It seems that when I was a developer I was forced to leave alot of my work in the work place. For testing it seems I find myself challenging everything. Even when I browse the web I find myself looking for the mistakes others have made and looking for better ways to do the smallest of things.