Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

Top features in the design of data modelling (1 of 2)

Data modelling jobs and career
[Data modelling jobs career]
The analogy with architecture is particularly appropriate because architects are designers and data modeling is also a design activity. In design, we do not expect to find a single correct answer, although we will certainly be able to identify many that are patently incorrect. Two data modelers (or architects) given the same set of requirements may produce quite different solutions.

Data modeling is not just a simple process of "documenting requirements" though it is sometimes portrayed as such. Several factors contribute to the possibility of there being more than one workable model for most practical situations.

First, we have a choice of what symbols or codes we use to represent real-world facts in the database. A person's age could be represented by Birth Date, Age at Date of Policy Issue, or even by a code corresponding to a range ("H" could mean "born between 1961 and 1970").

Second, there is usually more than one way to organize (classify) data into tables and columns. In our insurance model, we might, for example, specify separate tables for personal customers and corporate customers, or for accident insurance policies and life insurance policies.

Third, the requirements from which we work in practice are usually incomplete, or at least loose enough to accommodate a variety of different solutions. Again, we have the analogy with architecture. Rather than the client specifying the exact size of each room, which would give the architect little choice, the client provides some broad objectives, and then evaluates the architect's suggestions in terms of how well those suggestions meet the objectives, and in terms of what else they offer.

Fourth, in designing an information system, we have some choice as to which part of the system will handle each business requirement. For example, we might decide to write the rule that policies of type E20 have a commission rate of 12% into the relevant programs rather than holding it as data in the database. Another option is to leave such a rule out of the computerized component of the system altogether and require the user to determine the appropriate value according to some externally specified (manual) procedure. Either of these decisions would affect the data model by altering what data needed to be included in the database.

Finally, and perhaps most importantly, new information systems seldom deliver value simply by automating the current way of doing things. For most organizations, the days of such "easy wins" have long passed. To exploit information technology fully, we generally need to change our business processes and the data required to support them. (There is no evidence to support the oft-stated view that data structures are intrinsically stable in the face of business change).The data modeler becomes a player in helping to design the new way of doing business, rather than merely reflecting the old.

Unfortunately, data modeling is not always recognized as being a design activity. The widespread use of the term "data analysis" as a synonym for data modeling has perhaps contributed to the confusion. The difference between analysis and design is sometimes characterized as one of description versus prescription.

We tend to think of analysts as being engaged in a search for truth rather than in the generation and evaluation of alternatives. No matter how inventive or creative they may need to be in carrying out the search, the ultimate aim is to arrive at the single correct answer. A classic example is the chemical analyst using a variety of techniques to determine the make-up of a compound.

Related:
Start training in Data modelling

Comments

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

How to Check Kafka Available Brokers

SQL Query: 3 Methods for Calculating Cumulative SUM