This article describes four standard workflows by which you can quickly and effectively analyze and process incoming problems.
As a rule each error report falls into one of the following 4 categories:
- Known Bug: The user reports a problem that the team already knows about and which is already tracked in the bug tracker.
- Unreproducable: The user’s report doesn’t contain enough information to isolate the cause of the problem.
- Not A Bug: The user has come across behaviour that is not classified as a bug. Perhaps the problem was caused by incorrect user input, or the error report is even desired.
- New Bug: The user has in fact come across a real bug and the error report contains enough information to fix the error.
The diagram below gives you a quick overview of how to proceed in each of the four situations:
Step 1: Analyze The Stack Trace
First you look at the stack trace and the error report. With this information you can usually recognize whether this is a known problem (which, therefore, already exists in your bug tracker), or a new problem.
Linking a problem to an existing bug
If it is an already known error, then you can link this problem directly to the corresponding bug in your bug tracker.
As soon as you have linked the problem, then you are finished using Ctrlflow Automated Error Reporting. From here you can continue with the normal bug tracker workflow: prepare a bug fix, merge the fix, and then close the bug report. That’s it.
You are probably asking yourself “why should I link the problem with a bug?”
The linking of a problem with a bug report has the huge advantage that (i) your problem will be automatically closed when the bug is fixed and (ii) the user can be automatically informed of this change in status without you having to do anything.
If you would like to learn more about the possibilities of keeping the user informed of the status of the bugs that they have found with Ctrlflow Automated Error Reporting, check out the “Talkback Settings in Ctrlflow Automated Error Reporting”.
Step 2: Understanding And Reproducing The Bug
If the problem is not yet tracked in the bug tracker, you have to take a closer look at the error.
It happens sometimes that you are unable to reproduce the bug with just the help of the stack trace. In this case you will need additional information from your reporters, for example, steps to reproduce the error or system configuration information etc.
Here is where the Ctrlflow Automated Error Reporting Talkback feature comes to the fore: If you are ever missing information, you can ask all reporters of a problem (both past and future) to send you the missing information. You can do this in two ways:
- All past reporters: All past reporters are sent an email in which they are asked directly for the missing information, or
- All future reporters: You set the Needinfo flag in the problem.
Contacting the reporter by email
The email-function of Ctrlflow Automated Error Reporting is easily described. As you read in the introductory article, reporters can leave their name and email address for the purpose of follow up questions and status updates.
The Problem page in the Reporter section shows you how many users have supplied their email address. With a single click on the link you can open your desktop email client where you can inquire about the missing information:
The email function makes more sense when the problem is relatively new and the reporters can better remember what exactly was the cause of the problem.
Perhaps a reporter was nice enough to write a long comment describing the problem, but, unfortunately, crucial information is missing… In such cases the simplest possibility to gather the missing information is to send a short email. If the problem has already existed for a long time, then you are probably better off choosing the second way.
Prompt the user by means of the Needinfo flag
The second way you can get additional information from reporters is to use the Needinfo flag. With this you can exactly specify which additional information you would like to get from future reporters of the problem.
If a new reporter comes across the same problem, he will receive a special notification in his UI that will ask him to provide additional information.
You can individually configure the appearance of the notification that the server sends. If, for example, you are deploying Ctrlflow Automated Error Reporting as an in-house error reporting tool, then you could also configure the Reporting-Client (the part that runs on a client desktop) to automatically provide all useful information (such as the Eclipse configuration) – without the user having to do anything.
When the missing information has been specified, you can simply close the problem as “Can’t Reproduce”. The problem then disappears from your work list until the missing information comes in.
Usually you would configure an Automated Action so that all problems that are marked with “Can’t Reproduce” are automatically given a standard Needinfo flag. In this case the configuration step above can be dropped.
Modify your logging code to contain the necessary information
Another option I’d like to propose is to simply tweak the logging output so that it contains the information you are missing. If you are publishing milestones and head builds to your users, this is probably the easiest way to get the missing information in a reasonable time.
When your changes to the code base are small, i.e., your changes do not alter the order and names of the methods in the logged stacktrace, the error reports of the later versions will be assigned to the same problem. You then can savely close the error report and create an automated action that reopens the problem as soon as a new error report arrives with a higher bundle version.
Step 3: If an error report is not actually a bug
Not every error report is necessarily a bug in your software. Often developers use the Error log level very generously, such as in situations when a warning would have been sufficient. Sometimes Error log events are intended as useful information for the user.
You now have to find out whether the error report actually represents a bug in your software or whether it is just a message for your user. How often this occurs generally depends upon your code and the logging API you use. But, when you occasionally receive an error report which definitely does not represent a bug, you can quickly close the problem in the web UI and set the resolution to “Rejected” or “Works as Intended”.
Which resolutions you can choose depends on how you have configured your system. It makes sense to configure the same resolutions as exist in your bug tracker – this makes the switching back and forth between the two systems simple and the meaning of the resolutions transparent for all developers.
What happens to problems when they are set to “Rejected”
For one thing they disappear from your worklist. For another, the reporting clients use this information to decide whether they should activate for a newly logged error.
The best thing to do here would be to give an example. Suppose that whenever a user gives bad input an error is logged. The Error Reporting will now display an error notification popup which asks the user whether the problem should be sent as a report. This behavior in such cases is not desired.
In such situations Ctrlflow Automated Error Reporting has the chance to test in advance (before the error notification popup appears) whether the error should be ignored. Ctrlflow Automated Error Reporting determines whether an error should be ignored from the problem resolution, so, if a problem has been marked as “Rejected”, then, in future, all users for whom this error is logged will be spared the popup.
Optional: Providing a custom Resolution Note
Whenever a reporter sends an error report, the server returns a server response which contains some information about the current status of the problem the user experienced. This information may contain the link to the bug tracker, Needinfo flags, and a standard message. This message is bound to the problem’s current resolution.
However, in some cases the standard message may not be appropriate for a given problem. For instance, if you’d like to provide a hint how to work around this problem until it is fixed etc. The Automated Error Reporting server thus allows you to attach custom Resolution Notes to a problem.
To use them, simply go to the problem’s page and enter the message you’d like to send to your future reporters in the custom message field:
Step 4: If a new bug is reported
If the error report is not already tracked in your bug tracker, if you can reproduce it, and if the problem isn’t just a log message, then it is a real bug in your code.
From here the workflow is similar to Step 1. Click “Create Bug” on the Problem page to enter a new bug in your bug tracker. Now continue with your usual bug tracker workflow: fix the bug, merge the fix and close the bug.
If you like, Ctrlflow can inform all reporters of a problem as soon as it is closed. Details on how to do this can be found in the reference manual.
In this article you have learnt about the four standard workflows which you can use with any new error report. So the Ctrlflow Automated Error Reporting Server does not restrict you in your workflow, but does give you every tool you need for tracking down missing information (for example, by using the Needinfo tags) or for informing users of problem status changes.