官术网_书友最值得收藏!

Different approaches to validating the improved models

Model quality validation, of course, depends upon the kinds of models that you're building, and the purpose of them. There are a few general types of machine learning problems that I've covered in this book, and each has different ways of validating model quality.

Classification overview

We'll get to the specifics in just a moment, but let's review the high-level terms. One method for quantifying the quality of a supervised classification is using ROC curves. These can be quantified by finding the total area under the curve (AUC), finding the location of the inflection point, or by simply setting a limit of the amount of data that must be classified correctly against percentage of the time.

Another common technique is that of a confusion matrix. Limits can be set on certain cells of the matrix to help drive testing. Also, they can be used as a diagnostic tool that can help identify the issues that come up.

We will typically use the k-fold cross validation. Cross validation is a technique where we take our sample dataset and divide it into several separate datasets. We can then use one of these datasets to develop against one of the others, to validate that our data isn't overfitted, and a third dataset for a final check to see whether the others went well. All of these separate datasets work to make sure that we develop a generally applicable model, and not just one that predicts our training data but falls apart in production.

Regression

Linear regression quality is typically quantified with the combination of an adjusted- R2 value and by checking this, the residuals of the model don't fit a pattern. How do we check for this in an automated test?

The adjusted R2 values are provided by the most statistical tools. It's a quick measure of how much of the variations in the data is explained by your model. Checking model assumptions is more difficult. It is much easier to see patterns visually than via discrete, specific tests.

So, this is hard but there are other tests… perhaps, even more important tests that are easier—cross-validation. By selecting strong test datasets with a litany of misbehavior, we can compare R2 statistics from development, to testing, to ready for production. If a serious drop occurs at any point, then we can circle back.

Clustering

Clustering is the way in which we create our classification model. From there, we can test it by cross validating against our data. This can be especially useful in clustering algorithms, such as k-means, where the feedback can help us tune the number of clusters we want to use to minimize the cluster variation. As we move from one cross-validation dataset to another, it's important to remember not to persist with our training data from the previous tests, lest we bias our results.

主站蜘蛛池模板: 神池县| 仪征市| 体育| 成安县| 潞城市| 晋州市| 吴川市| 神农架林区| 营山县| 北流市| 沧州市| 崇左市| 仲巴县| 奉化市| 资兴市| 旺苍县| 鄯善县| 任丘市| 新河县| 大关县| 额敏县| 扎鲁特旗| 阿拉善右旗| 玛多县| 白沙| 沐川县| 屏山县| 桂平市| 和顺县| 苗栗市| 裕民县| 安吉县| 航空| 清涧县| 霞浦县| 托里县| 依兰县| 巧家县| 高清| 文成县| 沧州市|