Posts

Showing posts with the label machine-learning

Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

2 Top Python Libraries to Create ML model

Image
To Create a Model of Machine Learning in Python, you need TWO libraries. One is 'NUMPY' and the other one is 'PANDAS'. 2 Top Libraries You Need To Build a model of Machine learning you need the right kind of data. So, use the data for your project should be refined. Else, it will not produce correct results. The prime steps are Data Analysis and Data Preprocessing. NUMPY - It has the capabilities of Calculations. PANDAS- It has the capabilities of Data processing. How Install Python Machine Learning Libraries  import  NumPy as np # linear algebra import pandas as PD # data processing, CSV file I/O (e.g. PD.read_csv) How to Check NumPy/Pandas installed After '.' you need to give double underscore on both sides of the version.  How Many Types of Data You Need You need two types of data. One is data to build a model and the other one is data you need to test the model. Raw data Evaluate-data How to Build a Model Flowchart I have given a f