官术网_书友最值得收藏!

Ignoring punctuation and specific characters

Usually in natural language processing, some uninformative words or characters, called stop words, can be filtered out for easier handling. When computing word frequencies or extracting sentiment data from a corpus, punctuation or special characters might need to be ignored. This recipe demonstrates how to remove these specific characters from the body of a text.

How to do it...

There are no imports necessary. Create a new file, which we will call Main.hs, and perform the following steps:

  1. Implement main and define a string called quote. The back slashes (\) represent multiline strings:
    main :: IO ()
    main = do
      let quote = "Deep Blue plays very good chess-so what?\ 
        \Does that tell you something about how we play chess?\
        \No. Does it tell you about how Kasparov envisions,\ 
        \understands a chessboard? (Douglas Hofstadter)"
      putStrLn $ (removePunctuation.replaceSpecialSymbols) quote
  2. Replace all punctuation marks with an empty string, and replace all special symbols with a space:
    punctuations = [ '!', '"', '#', '$', '%'
                   , '(', ')', '.', ',', '?']
      
    removePunctuation = filter (`notElem` punctuations)
            
    specialSymbols = ['/', '-']
    
    replaceSpecialSymbols = map $ 
      (\c ->if c `elem` specialSymbols then ' ' else c)
  3. By running the code, we will find that all special characters and punctuation are appropriately removed to facilitate dealing with the text's corpus:
    $ runhaskell Main.hs
    Deep Blue plays very good chess so what Does that tell you something about how we play chess No Does it tell you about how Kasparov envisions understands a chessboard Douglas Hofstadter
    

There's more...

For more powerful control, we can install MissingH, which is a very helpful utility we can use to deal with strings:

$ cabal install MissingH

It provides a replace function that takes three arguments and produces a result as follows:

Prelude> replace "hello" "goodbye" "hello world!"

"goodbye world!"

It replaces all occurrences of the first string with the second string in the third argument. We can also compose multiple replace functions:

Prelude> ((replace "," "").(replace "!" "")) "hello, world!"

"hello world"

By folding the composition (.) function over a list of these replace functions, we can generalize the replace function to an arbitrary list of tokens:

Prelude> (foldr (.) id $ map (flip replace "") [",", "!"]) "hello, world!"

"hello world"

The list of punctuation marks can now be arbitrarily long. We can modify our recipe to use our new and more generalized functions:

removePunctuation = foldr (.) id $ map (flip replace "") 
        ["!", "\"", "#", "$", "%", "(", ")", ".", ",", "?"]
        
replaceSpecialSymbols = foldr (.) id $ map (flip replace " ") 
        ["/", "-"]
主站蜘蛛池模板: 邛崃市| 陵水| 宣化县| 新沂市| 陇南市| 马尔康县| 瑞昌市| 邢台县| 辛集市| 乌拉特中旗| 衡东县| 永定县| 旌德县| 南平市| 卫辉市| 南丹县| 勐海县| 山东省| 杂多县| 玉溪市| 华宁县| 饶阳县| 济源市| 大埔区| 民县| 贵州省| 元谋县| 布尔津县| 苗栗市| 连山| 石景山区| 鹿泉市| 乌鲁木齐市| 宁海县| 闻喜县| 江华| 东宁县| 昌乐县| 德昌县| 晋宁县| 五华县|