Posts

How to Check Column Nulls and Replace: Pandas

- November 25, 2023

Here is a post that shows how to count Nulls and replace them with the value you want in the Pandas Dataframe. We have explained the process in two steps - Counting and Replacing the Null values. Count null values (column-wise) in Pandas ## count null values column-wise null_counts = df.isnull(). sum() print(null_counts) ``` Output: ``` Column1 1 Column2 1 Column3 5 dtype: int64 ``` In the above code, we first create a sample Pandas DataFrame `df` with some null values. Then, we use the `isnull()` function to create a DataFrame of the same shape as `df`, where each element is a boolean value indicating whether that element is null or not. Finally, we use the `sum()` function to count the number of null values in each column of the resulting DataFrame. The output shows the count of null values column-wise. to count null values column-wise: ``` df.isnull().sum() ``` ##Code snippet to count null values row-wise: ``` df.isnull().sum(axis=1) ``` In the above code, `df` is the Panda

How to Effectively Parse and Read Different Files in Python

- November 11, 2023

Here is Python logic that shows Parse and Read Different Files in Python. The formats are XML, JSON, CSV, Excel, Text, PDF, Zip files, Images, SQLlite, and Yaml. Python Reading Files import pandas as pd import json import xml.etree.ElementTree as ET from PIL import Image import pytesseract import PyPDF2 from zipfile import ZipFile import sqlite3 import yaml Reading Text Files # Read text file (.txt) def read_text_file(file_path): with open(file_path, 'r') as file: text = file.read() return text Reading CSV Files # Read CSV file (.csv) def read_csv_file(file_path): df = pd.read_csv(file_path) return df Reading JSON Files # Read JSON file (.json) def read_json_file(file_path): with open(file_path, 'r') as file: json_data = json.load(file) return json_data Reading Excel Files # Read Excel file (.xlsx, .xls) def read_excel_file(file_path): df = pd.read_excel(file_path) return df Reading PDF files # Read PDF file (.pdf) def rea

A Beginner's Guide to Pandas Project for Immediate Practice

- October 20, 2023

Pandas is a powerful data manipulation and analysis library in Python that provides a wide range of functions and tools to work with structured data. Whether you are a data scientist, analyst, or just a curious learner, Pandas can help you efficiently handle and analyze data. In this blog post, we will walk through a step-by-step guide on how to start a Pandas project from scratch. By following these steps, you will be able to import data, explore and manipulate it, perform calculations and transformations, and save the results for further analysis. So let's dive into the world of Pandas and get started with your own project! Simple Pandas project Import the necessary libraries: import pandas as pd import numpy as np Read data from a file into a Pandas DataFrame: df = pd.read_csv('/path/to/file.csv') Explore and manipulate the data: View the first few rows of the DataFrame: print(df.head()) Access specific columns or rows in the DataFrame: print(df['column_name'])

How to Write Complex Python Script: Explained Each Step

- October 08, 2023

Creating a complex Python script is challenging, but I can provide you with a simplified example of a script that simulates a basic bank account system. In a real-world application, this would be much more elaborate, but here's a concise version. Python Complex Script Here is an example of a Python script that explains each step: class BankAccount: def __init__(self, account_holder, initial_balance=0): self.account_holder = account_holder self.balance = initial_balance def deposit(self, amount): if amount > 0: self.balance += amount print(f"Deposited ${amount}. New balance: ${self.balance}") else: print("Invalid deposit amount.") def withdraw(self, amount): if 0 < amount <= self.balance: self.balance -= amount print(f"Withdrew ${amount}. New balance: ${self.balance}") else: print("Invalid withdrawal amount o

Python Regex: The 5 Exclusive Examples

- September 23, 2023

Regular expressions (regex) are powerful tools for pattern matching and text manipulation in Python. Here are five Python regex examples with explanations: 01 Matching a Simple Pattern import re text = "Hello, World!" pattern = r"Hello" result = re.search(pattern, text) if result: print("Pattern found:", result.group()) Output: Output: Pattern found: Hello This example searches for the pattern "Hello" in the text and prints it when found. 02 Matching Multiple Patterns import re text = "The quick brown fox jumps over the lazy dog." patterns = [r"fox", r"dog"] for pattern in patterns: if re.search(pattern, text): print(f"Pattern '{pattern}' found.") Output: Pattern 'fox' found. Pattern 'dog' found. It searches for both "fox" and "dog" patterns in the text and prints when they are found. 03 Matching Any Digit import re text = "The price of the

Best Practices for Handling Duplicate Elements in Python Lists

- August 27, 2023

Here are three awesome ways that you can use to remove duplicates in a list. These are helpful in resolving your data analytics solutions. 01. Using a Set Convert the list into a set , which automatically removes duplicates due to its unique element nature, and then convert the set back to a list. Solution: original_list = [2, 4, 6, 2, 8, 6, 10] unique_list = list(set(original_list)) 02. Using a Loop Iterate through the original list and append elements to a new list only if they haven't been added before. Solution: original_list = [2, 4, 6, 2, 8, 6, 10] unique_list = [] for item in original_list: if item not in unique_list: unique_list.append(item) 03. Using List Comprehension Create a new list using a list comprehension that includes only the elements not already present in the new list. Solution: original_list = [2, 4, 6, 2, 8, 6, 10] unique_list = [] [unique_list.append(item) for item in original_list if item not in unique_list] All three methods will result in uni

10 Exclusive Python Projects for Interviews

- August 18, 2023

Here are ten Python projects along with code and possible solutions for your practice. 01. Palindrome Checker: Description: Write a function that checks if a given string is a palindrome (reads the same backward as forward). def is_palindrome(s): s = s.lower().replace(" ", "") return s == s[::-1] # Test the function print(is_palindrome("radar")) # Output: True print(is_palindrome("hello")) # Output: False 02. Word Frequency Counter: Description: Create a program that takes a text file as input and counts the frequency of each word in the file. def word_frequency(file_path): with open(file_path, 'r') as file: text = file.read().lower() words = text.split() word_count = {} for word in words: word_count[word] = word_count.get(word, 0) + 1 return word_count # Test the function file_path = 'sample.txt' word_count = word_frequency(file_path) print(word_count) 03. Guess the Nu

How to Fill Nulls in Pandas: bfill and ffill

- August 10, 2023

In Pandas, bfill and ffill are two important methods used for filling missing values in a DataFrame or Series by propagating the previous (forward fill) or next (backward fill) valid values respectively. These methods are particularly useful when dealing with time series data or other ordered data where missing values need to be filled based on the available adjacent values. ffill (forward fill): When you use the ffill method on a DataFrame or Series, it fills missing values with the previous non-null value in the same column. It propagates the last known value forward. This method is often used to carry forward the last observed value for a specific column, making it a good choice for time series data when the assumption is that the value doesn't change abruptly. Example: import pandas as pd data = {'A': [1, 2, None, 4, None, 6], 'B': [None, 'X', 'Y', None, 'Z', 'W']} df = pd.DataFrame(data) print(df) # Output: # A B

How to Handle Spaces in PySpark Dataframe Column

- July 30, 2023

In PySpark, you can employ SQL queries by importing your CSV file data to a DataFrame. However, you might face problems when dealing with spaces in column names of the DataFrame. Fortunately, there is a solution available to resolve this issue. Reading CSV file to Dataframe Here is the PySpark code for reading CSV files and writing to a DataFrame. #initiate session spark = SparkSession.builder \ .appName("PySpark Tutorial") \ .getOrCreate() #Read CSV file to df dataframe data_path = '/content/Test1.csv' df = spark.read.csv(data_path, header=True, inferSchema=True) #Create a Temporary view for the DataFrame df2.createOrReplaceTempView("temp_table") #Read data from the temporary view spark.sql("select * from temp_table").show() Output --------+-----+---------------+---+ |Student| Year|Semester1|Semester2| | ID | | Marks | Marks | +----------+-----+---------------+ | si1 |year1|62.08| 62.4| | si1 |year2|75.94| 76.75| | si

How to Convert Dictionary to Dataframe: Pandas from_dict

- July 20, 2023

Pandas is a data analysis Python library. The example shows you to convert a dictionary to a data frame. The point to note here is DataFrame will take only 2D data. So you need to supply 2D data. Pandas Dictionary to Dataframe import pandas as pd import numpy as np data_dict = {'item1' : np.random.randn(4), 'item2' : np.random.randn(4)} df3=pd.DataFrame. from_dict (data_dict, orient='index') print(df3) Output 0 1 2 3 item1 -0.109300 -0.483624 0.375838 1.248651 item2 -0.274944 -0.857318 -1.203718 -0.061941 Explanation Using the NumPy package, created a dictionary with random values. There are two items - item 1 and item 2. The data_dict is input to the data frame. The from_dict method needs two parameters. These are data_dict and index. Here's the syntax you can refer to quickly. Related Hands-on Data Analysis Using Pandas How to create 3D data frame in Pandas

The Easy Way to Split String Python Partition Method

- July 07, 2023

Here's a way without the Split function you can split (or extract) a substring. In Python the method is Partition. You'll find here how to use this method with an example. How to Split the string using Partition method Returns Left side part Example-1 my_string='ABCDEFGH||10||123456.25|' my_partition=my_string.partition('|')[0] print(my_partition) Output |10||123456.25| ** Process exited - Return Code: 0 ** Press Enter to exit terminal Example-2 Returns from the separator to last of the string. my_string='ABCDEFGH||10||123456.25|' my_partition=my_string.partition('|')[-1] print(my_partition) Output |10||123456.25| ** Process exited - Return Code: 0 ** Press Enter to exit terminal The use of Rpartition to split a string in Python Example-1 Returns except right side last separator. my_string='ABCDEFGH||10||123456.25|' my_partition=my_string.rpartition('|')[0] print(my_partition) Output ABCDEFGH||10||123456.25 ** Process exited -

5 Python Pandas Tricky Examples for Data Analysis

- June 24, 2023

Here are five tricky Python Pandas examples. These provide detailed insights to work with Pandas in Python, #1 Dealing with datetime data ( parse_dates pandas example) import pandas as pd # Convert a column to datetime format data['date_column'] = pd.to_datetime(data['date_column']) # Extract components from datetime (e.g., year, month, day) data['year'] = data['date_column'].dt.year data['month'] = data['date_column'].dt.month # Calculate the time difference between two datetime columns data['time_diff'] = data['end_time'] - data['start_time'] #2 Working with text data # Convert text to lowercase data['text_column'] = data['text_column'].str.lower() # Count the occurrences of specific words in a text column data['word_count'] = data['text_column'].str.count('word') # Extract information using regular expressions data['extracted_info'] = data['text_column'].

2 User Input Python Sample Programs

- June 10, 2023

Here are the Python programs that work on taking user input and giving responses to the user. These are also called interactive programs. Python enables you to read user input from the command line via the input() function or the raw_input() function. Typically, you assign user input to a variable containing all characters that users enter from the keyboard. User input terminates when users press the <return> key (included with the input characters). #1 User input sample program The following program takes input and replies if the given input value is a string or number. my_input = input("Enter something: ") try: x = 0 + eval(my_input) print('You entered the number:', my_input) except: print(userInput,'is a string') Output Enter something: 100 You entered the number: 100 ** Process exited - Return Code: 0 ** Press Enter to exit terminal. #2 User input sample program The following program takes two inputs from the user and calcula

The Quick and Easy Way to Analyze Numpy Arrays

- May 28, 2023

The quickest and easiest way to analyze NumPy arrays is by using the numpy.array() method. This method allows you to quickly and easily analyze the values contained in a numpy array. This method can also be used to find the sum, mean, standard deviation, max, min, and other useful analysis of the value contained within a numpy array. Sum You can find the sum of Numpy arrays using the np.sum() function. For example: import numpy as np a = np.array([1,2,3,4,5]) b = np.array([6,7,8,9,10]) result = np.sum([a,b]) print(result) # Output will be 55 Mean You can find the mean of a Numpy array using the np.mean() function. This function takes in an array as an argument and returns the mean of all the values in the array. For example, the mean of a Numpy array of [1,2,3,4,5] would be result = np.mean([1,2,3,4,5]) print(result) #Output: 3.0 Standard Deviation To find the standard deviation of a Numpy array, you can use the NumPy std() function. This function takes in an array as a par

These 10 Skills You Need to Become Data Analyst

- May 16, 2023

To become a data analyst with Python, there are several technical skills you need to learn. Here are the key ones: #1 Python Programming Python is widely used in data analysis due to its simplicity, versatility, and the availability of powerful libraries. You should have a strong understanding of Python fundamentals, including data types, variables, loops, conditional statements, functions, and file handling. #2 Data Manipulation Libraries Familiarize yourself with libraries like NumPy and Pandas, which are essential for data manipulation and analysis. NumPy provides support for efficient numerical operations, while Pandas offers data structures (e.g., DataFrames) for easy data manipulation, cleaning, and transformation. #3 Data Visualization Gain proficiency in data visualization libraries like Matplotlib and Seaborn. These libraries enable you to create insightful visual representations of data, such as line plots, scatter plots, bar charts, histograms, and heatmaps. #4 SQL (Structu

Search This Blog

ApplyBigAnalytics

Posts

Featured Post