Posts

Showing posts with the label IBM

Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

Real thoughts on IBM power8 servers to use on analytics

Image
IBM Servers International Business Machines Corp, in its latest attempt at reviving demand for its hardware products, is launching high-end system servers that it says are 50 times faster than its closest competitor at analysing data.  The POWER8 servers , the product of a $2.4 billion, three-year investment, are part of the company's decade-long shift to higher-value hardware technology.    IBM  said the machines are 50 times faster than the low-end x86-based servers it sold to Chinese PC maker  Lenovo  Group Ltd in January.  The technology services provider said on Wednesday it hopes the servers, designed for large-scale computing, will appeal to clients looking to manage new types of social and mobile computing and mass amounts of data. Last week, the company reported its lowest quarterly revenue in five years, weighed down by falling demand for its storage and server products. IBM dominates the higher-end server market with 57 percent market share, according to res