Testing a movingtarget_quest_dynatrace

Editor's Notes

• #6: Software in theory is a fairly straightforward activity. For every input, there is a defined and known output. We enter values, make selections, or navigate an application, and compare the actual result with the expected one. If they match, we nod and move on. If they don’t, we possibly have a bug. The point is that we already know what the output is supposed to be. Granted, sometimes an output is not well-defined, and there can be some ambiguity, and you get disagreements on whether or not a particular result represents a bug or something else.
• #7: But there is a type of software where having a defined output is no longer the case. Actually, two types. One is machine learning systems. The second is predictive analytics, or adaptive systems.
• #10: Most machine learning systems are based on neural networks. A neural network is a set of layered algorithms whose variables can be adjusted via a learning process. The learning process involves using known data inputs to create outputs that are then compared with known results. When the algorithms reflect the known results with the desired degree of accuracy, the algebraic coefficients are frozen and production code is generated. Today, this comprises much of what we understand as artificial intelligence.
• #14: These types of software are becoming increasingly common, in areas such as ecommerce, public transportation, automotive, finance, and computer networks. They have the potential to make decisions given sufficiently well-defined inputs and goals. In some instances, they are characterized as artificial intelligence, in that they seemingly make decisions that were once the purview of a human user or operator. Decision augmentation Personal assistants
• #15: Both of these types of systems have things in common. For one thing, neither produces an “exact” result. In fact, sometimes they can produce an obviously incorrect result. But they are extremely useful in a number of situations when data already exists on the relationship between recorded inputs and intended results. Let me give you an example. Years ago, I devised a neural network that worked as a part of an electronic wind sensor. This worked though the wind cooling the electronic sensor based on its precise decrease in temperature at specific speeds and directions. I built a neural network that had three layers of algebraic equations, each with four to five separate equations in individual nodes, computing in parallel. They would use starting coefficients, then adjust those coefficients based on a comparison between the algorithmic output and the actual answer. I then trained it. I had over 500 data points regarding known wind speed and direction, and the extent to which the sensor cooled. The network I created passed each input into its equations, through the multiple layers, and produced an answer. At first, the answer from the network probably wasn’t that close to the known correct answer. But the algorithm was able to adjust itself based on the actual answer. After multiple iterations with the training data, the coefficients should gradually hone in on accurate and consistent results.
• #17: How do you test this? You do know what the answer is supposed to be, because you built the network using the test data, but it will be rare to get an exactly correct answer all of the time. The product is actually tested during the training process. Training either brings convergence to accurate results, or it diverges. The question is how you evaluate the quality of the network.
• #19: Have objective acceptance criteria. Know the amount of error you and your users are willing to accept. Test with new data. Once you’ve trained the network and frozen the architecture and coefficients, use fresh inputs and outputs to verify its accuracy. Don’t count on all results being accurate. That’s just the nature of the beast. And you may have to recommend throwing out the entire network architecture and starting over. Understand the architecture of the network as a part of the testing process. Few if any will be able to actually follow a set of inputs through the network of algorithms, but understanding how the network is constructed will help testers determine if another architecture might produce better results. Communicate the level of confidence you have in the results to management and users. Machine learning systems offer you the unique opportunity to describe confidence in statistical terms, so use them. One important thing to note is that the training data itself could well contain inaccuracies. In this case, because of measurement error, the recorded wind speed and direction could be off or ambiguous. In other cases, the cooling of the filament likely has some error in its measurement.
• #21: Another type of network might be termed predictive analytics, or adaptive systems. These systems continue to adapt after deployment, using a feedback loop to adjust variables and coefficients within the algorithm. They learn while in production use, and by having certain measureable goals, are able to adjust aspects of their algorithms to better reach those goals. One example is a system under development in the UK to implement demand-based pricing for train service. Its goal is to try to encourage riders to use the train during non-peak times, and dynamically adjusts pricing to make it financially attractive for riders to consider riding when the trains aren’t as crowded. This type of application experiments with different pricing strategies and tries to optimize two different things – a balance of the ridership throughout the day, and acceptable revenue from ridership. A true mathematical optimization isn’t possible, but the goal is to reach a state of spread-out ridership and revenue that at least covers costs.