Posts

Showing posts with the label Horizontal

Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

-
Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif
Post a Comment
Read more

Cloud Computing: Horizontal Vs. Vertical Scaling

-
Image
The purpose of cloud computing is resource utilization. You can scale up the resources in two ways - vertical and horizontal. Adding resources, you can do either horizontally and vertically . The advantages and drawbacks you can find in simple words. Scaling 1. Horizontal Scaling Advantages You can increase workloads in small steps. The upgrade-cost is far less. Scale the system as much as needed. Drawbacks Dependency on software applications is more for Data distribution and parallel processing. On top of that, fewer software applications exist in the market. You May Also Like:  9 Top Services AWS Provided 2. Vertical Scaling Advantages Since it is a single machine, it is easy to manage. On the fly, you can increase workloads. Drawbacks It is expensive. You need a huge investment. The machine should be powerful to take more workloads - future use. Below is the List of Resources that You can do both Horizontal and Vertical Scaling Platform Scaling Network Scaling Container Scaling Dat
Post a Comment
Read more