官术网_书友最值得收藏!

Preparing data

Data preparation refers to the processes performed on the input dataset before training the algorithm. A rigorous preparation process can simultaneously enhance the quality of the data and reduce the amount of time it will take the algorithm to reach the desired accuracy. The two steps to preparing data are data pre-processing and data transformation. We will go into more detail on preparing data in Chapters 2Setting Up The Development EnvironmentChapter 3Supervised Learning, and Chapter 4, Unsupervised Learning.

Data pre-processing aims to transform the input dataset into a format that is adequate for work with the selected algorithm. A typical example of a pre-processing task is to format a date column in a certain way, or to ingest CSV files into a database, discarding any rows that lead to parsing errors. There may also be missing data values in an input data file that need to either be filled in (say, with a mean), or the entire sample discarded. Sensitive information such as personal information may need to be removed.

Data transformation is the process of sampling, reducing, enhancing, or aggregating the dataset to make it more suitable for the algorithm. If the input dataset is small, it may be necessary to enhance it by artificially creating more examples, such as rotating images in an image recognition dataset. If the input dataset has features that the exploration has deemed irrelevant, it would be wise to remove them. If the dataset is more granular than the problem requires, aggregating it to a coarser granularity may help speed up results, such as aggregating city-level data to counties if the problem only requires a prediction per county.

Finally, if the input dataset is particularly large, as is the case with many image datasets intended for use by deep learning algorithms, it would be a good idea to start with a smaller sample that will produce fast results so that the viability of the algorithm can be verified before investing in more computing resources.

The sampling process will also divide the input dataset into training and validation subsets. We will explain why this is necessary later, and what proportion of the data to use for both.

主站蜘蛛池模板: 留坝县| 麦盖提县| 曲阳县| 珲春市| 海晏县| 漳浦县| 福海县| 绥江县| 静乐县| 通江县| 高清| 肇庆市| 沧源| 海宁市| 铁岭市| 金门县| 临清市| 宜阳县| 罗城| 巴塘县| 彰武县| 汤原县| 大宁县| 涿鹿县| 三河市| 台中市| 曲水县| 辽源市| 托里县| 洪湖市| 曲沃县| 澄江县| 永新县| 蓝田县| 永德县| 额尔古纳市| 嵊泗县| 五家渠市| 宜阳县| 会理县| 沅江市|