Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

2 Root Causes for Defects in Software Development

Root causes

Miscommunication


Miscommunication is a common factor, which can be defined as inaccurate statements or information missing that is required for the action to be done successfully. This miscommunication ends up in the documentation or verbal communication that occurs.

Instead of spending time to make sure everything is accurate, statements are made that are untrue or unclear. When this occurs at the beginning of the change process the bad information continues down through the process. Decisions and design are made based on it. 


At some point it gets realized that the information is bad and a defect is created. In the common project process that could be classified as linear, most defects are not found until in the later phase of development and unit testing has started.

Process Defects


This would be similar to a defect a machine makes in manufacturing. Even though the input is accurate, the process itself causes a defect to occur. 


The original process was prone to defects no matter how careful the work was done. Randomly at some point in time a widget would not be created correctly. When the process was changed to reduce the number of handoffs and some steps were moved around, the possibility of creating a defect was reduced.

This same concept occurs with processes. 


As processes add more handoffs and complexity, the process itself is introducing more spots in which a defect can occur, which increases the possibility of defects occurring. 


It becomes a catch 22 in that when companies have issues with the number of defects, they create more complex processes to try to stop them from happening. 


By doing this they only add to the problem. The defects initially do go down, but it's only because of the amount of additional resources and the priority given to the defects. 



Once both resources and priority are moved to other things, then the defect counts go back up and might even increase because with the more complex process there are more spots in which a defect might occur.

It would be wonderful if processes could be made to eliminate all defects, but they don't. There is always some unique situation—whether machine or human—that will always create a defect. Attaining zero defects in most situations is impossible. 


The best that can be done is to greatly reduce the risk of having a defect. This is not to say that each defect does not need to be analyzed, but every defect does not need to be resolved. 


For system-generated ones it might not be monetarily feasible to make changes to eliminate them from happening. Even Six Sigma addresses this by originally stating that the quality goal is to obtain 3.4 defect parts per million (PPM) opportunities. 


It would be impossible and also very costly to attempt to obtain a defect ratio of 0.00. W. Edwards Deming's point #3 states that processes need to be created so that instead of opening up the possibility of defects occurring they will eliminate defects. 


Comments

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

Explained Ideal Structure of Python Class

How to Check Kafka Available Brokers