Posts

Showing posts with the label system commnads

Featured Post

How to Read a CSV File from Amazon S3 Using Python (With Headers and Rows Displayed)

-
Image
  Introduction If you’re working with cloud data, especially on AWS, chances are you’ll encounter data stored in CSV files inside an Amazon S3 bucket . Whether you're building a data pipeline or a quick analysis tool, reading data directly from S3 in Python is a fast, reliable, and scalable way to get started. In this blog post, we’ll walk through: Setting up access to S3 Reading a CSV file using Python and Boto3 Displaying headers and rows Tips to handle larger datasets Let’s jump in! What You’ll Need An AWS account An S3 bucket with a CSV file uploaded AWS credentials (access key and secret key) Python 3.x installed boto3 and pandas libraries installed (you can install them via pip) pip install boto3 pandas Step-by-Step: Read CSV from S3 Let’s say your S3 bucket is named my-data-bucket , and your CSV file is sample-data/employees.csv . ✅ Step 1: Import Required Libraries import boto3 import pandas as pd from io import StringIO boto3 is...
Post a Comment
Read more

The best helpful HDFS File System Commands (2 of 4)

-
Image
#Top-Selected-HDFS-file-system-commands CopyFrom Local Works similarly to the put command, except that the source is restricted to a local file reference. hdfs dfs -copyFromLocal URI hdfs dfs -copyFromLocal input/docs/data2.txt hdfs://localhost/user/rosemary/data2.txt HDFS Commands Part-1of 4 copyToLocal Works similarly to the get command, except that the destination is restricted to a local file reference. hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI hdfs dfs -copyToLocal data2.txt data2.copy.txt count Counts the number of directories, files, and bytes under the paths that match the specified file pattern. hdfs dfs -count [-q] hdfs dfs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2 cp Copies one or more files from a specified source to a specified destination. If you specify multiple sources, the specified destination must be a directory. hdfs dfs -cp URI [URI …] hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir du Disp...
Post a Comment
Read more