To automate or not?

Its funny how so many think that automation is the key to everything in test... the problem with this is that you cant just slot this in. Here are a few of many things to consider.

- Selection of the appropriate test automation tool.
- Time and cost to create the test scripting.
- Training of people in the use of the tool.
- Additional costs of maintaining the scripting.

With all the risks though do come a number of benefits.

- Faster regression testing.
- repeatable tests (while this is the case with manual often steps are not followed exactly.
- greater reliability
- many of the parts are reusable
- must be fully comprehensive to enable the scripting to function

Never as easy as it seems

In any organization there is what is best? what is right? and what actually happens.

What I have been seeing lately is alot of internal politics which seems to directly stand in the way of what everyone knows is right. Being who I am I stay out of it but this carries its issues. To be seen as suitable for a higher post within my organization I need to get involved in this politics directly.

This has just been a lesson to me to be clear to myself what I need from my job and to strive for that and not just doing my job well. Age old rule that you need to being already doing the job to get the job.

Risk based testing..what is it?

As testers we use risk-based testing every day to ensure that project and product risks are are being identified and addressed. This is accomplished by drawing on the knowledge of business representatives, designers and testers to identify early any risks, establish focus areas to be tested and agree mitigating actions that can be taken to deal them.

These risks are then added into the Test Plan with the established level of risk. The risk level can be calculated by multiplying an estimated probability and impact, the estimated values should be approved by the stakeholders. These risks and their importance are regularly reviewed with stakeholders and mitigating actions to deal with those risks are identified. Testing activities are then targeted at the high risk areas first to prevent these risks being realized and minimize potential cost, quality and time implications.

Take a break..

It doesn't matter how good you are or how easy you take it some times you just need a break. Take the holiday step away from the plate and let the whole world come back into perspective. Sometimes it is good to remind yourself what your priorities are and while you are on holiday is a great time for that.

If you haven't guessed I am on holiday avoiding work and putting my feet up, Looking after my little girl. Nice to have the time to think why do I work and whats it all for.

All I say is .. If you are on holiday ENJOY IT!! save the worry of what is going on in work, for when you are in work. It is unlikely to go anywhere and you are not being paid for the worry.

2 weeks work in 2 days

The software is late to test and needs to be sorted in 2 days and not the original 10. Something needs to give in order for the work to be completed on time and what is it? The example that I heard from a man 40 years in testing is this. If a 2 litre jug is the amount of work needed and a glass is the amount of time we have start pouring. It doesnt matter how much water you pour the water wont all fit in the glass.

With the testing what is needed is a test case, documenting what we are aiming at doing. If the time available is changed then the prioritization within the test case can be quickly utilised and the work done can be de-scoped. If we have tests that range from risk 1 - 9 then it might be that within 2 days we can only run the risk 9 tests, this can then be agreed as an acceptable approach with the stakeholders. It will mean that an informed decision can be made on whether the coverage provided is acceptable or not.

Accessability testing pt .1

pt .1 - Blind or partially blind users

This is a really subjective thing to test and depends what you are looking at checking for. The main thing for web pages is to try using a magnifier and reader with your web pages and that way you will understand the issues that partially blind or fully blind users might encounter. Then you can look at making your site as easy for them to use as possible.

My recommendation to anyone looking at this area would be to view a website like www.bbc.co.uk with a reader first and then try some other sites. This will show how BBC do it right and how wrong some other sites get it.

Work to life balance.

Its a funny world and I am not sure that everyone gets the work to life balance right. To be brutally honest, Im not sure if even I get the work to life balance right. The world today leads people to work as hard as they can training body and mind to work like a machine. To be able to excel within your job you need to be better than those around you.

Whilst thinking about that it is important to consider that people need time out and arnt machines. Take a break, Let go! and allow your mind time to work through problems. Some of the best ideas and most complex problems I have resolved have been sorted when taking a break and not thinking about it. If you have an exam to cram for and something solid to learn burning the candle at both ends can help... this is not always the best thing for working.

Requirement trade off

When testing against releases the process is ongoing stepping from the execution in one software release and into the planning and preparation for testing another release. With everyone working to tight deadlines and us working alongside the software developers often we find that many of the details and requirements are not available at the planning stages. I try to implement the following to ensure that by the time software is ready to test we have the required scripting completed and the requirements/design is available. Firstly i evaluate what should be in the release based on what has been approved and then i review the requirements. I then identify what i believe to be testable requirements and weak areas that will need further review and highlight these areas back to the business analysts. I will then produce test scenarios, scripts and cases to cover the testable areas and repeat this process up until the software is submitted for system testing. If the requirements still are not acceptable then a decision is taken to either reject the release or proceed with a risk. If proceeding with a risk then this should be formally noted against the project.

What definately should not happen is that the developer says to you we dont have the requirements but this is what it is supposed to do. If you take this information and then use it as a test basis then how do you know if that is the product that was requested initially. Ultimately what this will lead to is poorly defined requirements and a request that started out for a 'rope swing' and ended up being a 'crane'.

New testers starting out

My team have in recent times gained a couple new testers. This has highlighted to me some of the simple things that you forget as you become more experienced. Basic principles that are of use to someone completely new to testing.

1. KISS - Keep It Simple Stupid : if you are going to write test scripts keep them simple and write them so that anyone can follow them.


2. Review - When planning test scripts run the scenarios then at least have these peer reviewed i.e. get someone to look through what you are planning and comment on it. Once the scripts are complete i would recommend that for visability these are reviewed by business, technical and test representatives before script execution.


3. Dont debate it log it - When you find a defect then dont try and work out why or debate whether it is or not LOG IT. If you have found a difference between the actual and expected outcomes then the ideal person to resolve this issue is the developer and not the tester.


4. Picture says a thousand words - Where possible attach a screen shot of any issues, this gives more detail than just some text and might provide information to the developers that was not previously noticed. This also provides evidence that the issue was encountered as described.


5. Break it down - Break the scripting down into bite sized chunks in this way it will be much more reusable especially if you have a tool that will allow parameterisation.


6. Assume nothing - When you are testing an application it should not be based on what you think it should do. This information should have been clearly provided in advance in the documentation. If you dont have all of the requirements and design information required then this should be raised as an issue.

Unit test recording

I am in the process of working with some developers in ensuring that they produce an acceptable level of unit test evidence. This can cause issues as we dont wish to add a large overhead to the unit testing process by mandating the use of test scripts and formal test execution but at the same time require some record that the unit testing has been performed.

What is being trialled so far with some development acceptance is a simple document asking for information on the :
Introductory Information

• System - Software application that is being worked on.
• Project - Project Name and Code
• Change - What is being tested? could be the whole system or could be a small new feature.

Tabulated

• Tester - Person performing the unit testing (Ideally not the developer of the code.)
• Test ID - Identification for individual test
• Input - Quick summary of what action is required (e.g. click all close buttons)
• Expected Output - Quick summary of what is expected (e.g. application should close successfully with no exceptions or memory leaks)
• Test passed - Y/N
• Issues - Any issues outstanding.

This it is hoped will help prevent certain areas of unit testing being missed and greatly increase the overall quality of the end solutions.

Office Politics

Isn't it strange how when working within a company, everyone should be working together and separate teams should be effectively communicating, they just DON'T. I think a key issue here is office politics and this is within testing as much as any other area.

What relevance is this? you might be thinking.

My day began with fireworks over what team should be testing a piece of work and ended with finger pointing over who was responsible for the test environment when not enough resources have been scheduled for adequate resource.

To me it stresses that approach is everything when dealing with issues and that the power of talking to someone face to face should never be under estimated. If the communication initially had taken place sat around a table and been recorded then the responsibility in both cases would have been clearly laid out.

To deal with office politics and different team agendas
1/ Get everyone (especially those that have differing approaches) around the table in a meeting.
2/ Make it clear to all of the people attending what will be discussed and what is expected by the end of the meeting.
3/ Ensure that everyone agrees the way forward and that this approach is minuted and issued at the end to all attendes.

Once completed then by definition someone has assumed responsibility for this. Later in the process there should not then be any contention over resourcing etc. For these issues the documents production and agreement is essential as this ensures that everyone is clear of their responsibility.

Test Policy

I am quite interested in this testing artifact and how many people out there actually have one. This is a document that is very short maybe just a page and it highlights at the highest level what the business itself expects from testing.

I have worked in several places that dont have this and all of them seem to have the same issue testing seems to have a free reign and little focus. Within a company every member of the testing should have an awareness of this document. I am a firm believer that this should really be a bit like a cub scout promise.

I will amend this at a later date to add further information on what should be in a test policy and how to write it.

ISEB Intermediate Software Testing Exam

For those of you that think that this is just a piece of paper and proves nothing then I believe that you are wrong. Having just completed this exam I was pleasantly surprised to find how difficult this  actually was. All the info learnt and all the theory are for nothing in this if you cannot apply your knowledge to real situations.

It means that rather than just having a load of techniques, that you are not sure where to use, you can now identify what techniques to use in the planning stages of testing. For me it has given the insight to be able to greater understand a test analyst and test manager role. I also feel i have a best practice to aim for in my work.

One topic that I found especially usefull was the sections that covered risk. This will aid me on a day to day basis with scoping out the areas that I will be able to test.

Agile

You know what really grinds my gears. Not people pointing out problems with Agile or even people deciding that it is not suitable for them. What grinds my gears is people creating problems with Agile without understanding what Agile really is and what is involved. For testing staff it could be a bad thing as many implementations see no place for professional testers but I would see this as an oportunity. By pair programming with one tester and one developer you get the best of both worlds with everything seen from two perspectives.

Last Minute Testing

Every time a release comes round it always ends up leading to last minute testing. For me this time it started with me coming back from holiday and having to test two systems that should already be in test. Had I done any data or test prep the clear answer is 'NO' was all the correct and accurate documentation provided the clear answer again is 'NO'. 

So how to approach this in a way that doesnt compromise my work. There are many ways but two that i will mention here.

1/ You could on the basis that it is in test and the correct documentation and requirements are not provided reject the application hand it back to development and remove it from the release.

2/ You could create a document that scopes what you will be testing and why based on the information that you do have and provide this information to the project and development leads. This will mean that you have provided clear details on what you will cover. Following this you should also produce a risk log of some description that is also provided to the project and development leads. This risk log should contain information of what was not provided to you and any areas that you fear might have been missed, it might even state testing was restricted due to the lack of time provided.

Some people would say that this sounds like covering your own back should anything go wrong. I would say that this is ensuring that you maintain your integrity and demonstrate you have done the best job that you can. Also listing the risks in this fashion brings them to the attention of project leaders so that they might be able to come up with some kind of mitigation.

This kind of information being produced and circulated means that should anyone question your competence or your coverage. You can provide all the documentation as to the process that you have followed and why. 

Planning time

Sorry I have not posted in a while but been looking after number one and enjoying my honeymoon.

Consequently this post came to me on holiday.

There are many things that should be considered when managing testing and one of the most important is the issue of resources. Even when you think you have it all done and everything scheduled into the correct deadlines there is always something that will change. It might be that someone goes off sick... it might be that you missed that someone was going on holiday or even some family berevement. The long and short of it is that your plans must be flexible and should even when tight deadlines are being worked to hold a measure of contingency. If you cannot have any contingency then your testing should be planned in a way that you know what all the high priority test cases are that absolutely must be covered.

This will mean that even for emergency changes you can have a good measure of assurance that the system will function as desired.

Out of the frying pan into the fire.

Have you ever thought that i would be good to say.

• We wont take things that arent unit tested.
• We will only be testing versus assessed risk.
• We will be reviewing all documents early and if not acceptable we will not allow development.
• If development is late then testing will be late and it wont go in the scheduled release.
• If its last minute then it better be a live issue.

Well the lion share of this is what the team I am part of are going to be doing soon. I fear that this is likely to start out being a rough ride as it is not what everyone is used to but it seems that it might well in the long run greatly improve things for everyone. Ultimately our team have little choice as currently we have little resources and alot of project and maintenance work to cover.

Testing danger!

I was sat in the waiting room at my local garage today and had some thoughts about testing and where we would be if people didnt take effort to ensure accurate testing. What would it be like if you had your MOT done then two hours later died in a crash cause by your brakes failing. In response the garage said "Well it all seemed to be passing so we just passed it only missed a few bits out." 

I think the main value in testing comes from doing the best job that you can to ensure quality and covering as many areas as you can in the time given. Then taking the results that you have and formulating a recommendation to proceed or not to proceed.

Deadlines and estimates

It has came to that time in work again when the questions start being asked. 

When is the testing likely to be complete?

How much testing is left to be done?

How is the testing going?

and many more questions. The reality is that you can only give a best guess especially since as testers we are relying on the rest of the development team to come up with the goods for us to test. For me today I took an educated guess that it should be finished just about on time.  I would say be clear with what you are aiming to complete and what you have already completed and then they can make a judgement call aswell on the likelyhood of something being completed. 

Most of all dont just allow something through that doesnt meet the expected!! 

Waiting...

3 days late testing and moving into day 4... I wont be starting the testing till tomorrow. In this case it is not my fault or that of the developers but some important things to consider are:-

* Smoke test the environment in advance to ensure that it is working correctly.
* Where possible smoke test the application/system in advance (even if it is an early build) to check that it is going to work.
* Regularly schedule your testing to fit in with your current testing progress.
* Regularly monitor your testing activities to keep all parties up to date with current progress.
* Keep contact with the developers and business analysts to pick up on any changes to the design and requirements. These can greatly impact the testing activites and you will need to know about this ASAP.