Python: These 10 You Need to Remove from Web Scrapped data
Here're ten Python technics to clean the scraped data. The scraped Text has unwanted hidden data. So, as part of cleaning it try to remove these ten in your data.
Python: These 10 You Need to Remove from Web Scrapped data
Data is prime input for text analytics projects. After cleaning, you can feed to Machine/Deep Learning systems.
- Removing HTML tags
- Tokenization
- Removing unnecessary tokens and stop-words
- Handling contractions
- Correcting spelling errors
- Stemming
- Lemmatization
- Tagging
- Chunking
- Parsing
![]() |
10 Technics to Clean Text in Python |
1. Removing HTML tags
The unstructured text contains a lot of noise ( data from web pages, blogs, and online repositories.)when you use web/screen scraping.
The HTML tags, JavaScript, and Iframe tags typically don't add much value to understanding and analyzing text. Our purpose is to remove HTML tags, and other noise.
2. Tokenization
- Tokens are independent and minimal textual components. And have a definite syntax and semantics. A paragraph of text or a text document has several elements. Those you can further break down into clauses, phrases, and words.
- The popular tokenization techniques include sentence and word tokenization. These, you can use to break down a text document (or corpus) into sentences. And each sentence into words.
- Thus, tokenization is the process of breaking down or splitting textual data into smaller and more meaningful components called tokens.
![]() |
Text Analytics in Python |
3. Removing Unnecessary Stop Words
4. Handling contractions
The best examples of contractions are you'll, it's, etc.
5. Correcting spelling errors
Auto correcting spelling errors. While doing a Google search, you will find it corrects your spelling automatically.
6. Stemming
Here you can reduce words to the root level. The best example is Snowball, this you stem it to root level as Snow and Ball.
7. Lemmatization
Based on the context, bring the words to the root level, and make them meaningful.
8. Tagging
This is the concept of group particular words under a Tag.
9. Chunking
It is of constructing from various words of Verbs, Nouns, Adjectives, etc. Check out here on Data Chunking.
10. Parsing
The data will pass through some syntax rules. The output will then feed to Machine Learning systems. The syntax rules vary from project to project.
Comments
Post a Comment
Thanks for your message. We will get back you.