Skip to main content

Featured post

8 Top Data Science Platform Developers in the World

Top data science tools and platforms providers across the world. Useful information for data science and data analytics developers.

Data Science is a combination of multiple skills. AI and Machine Learning are part of data science. You can create AI and Machine Learning products with data. 

Related Posts

Top Skills You Need for Data Science CareerData Science Sample Project an Example

The story Hadoop data value less in cost than ETL

Traditional data warehouse

That isn’t to say that Hadoop can’t be used for structured data that is readily available in a raw format; because it can.In addition, when you consider where data should be stored, you need to understand how data is stored today and what features characterize your persistence options. 
  • Consider your experience with storing data in a traditional data warehouse. Typically, this data goes through a lot of rigor to make it into the warehouse.
  •  Builders and consumers of warehouses have it etched in their minds that the data they are looking at in their warehouses must shine with respect to quality; subsequently, it’s cleaned up via cleansing, enrichment, matching, glossary, metadata, master data management, modeling, and other services before it’s ready for analysis. 
  • Obviously, this can be an expensive process. Because of that expense, it’s clear that the data that lands in the warehouse is deemed not just of high value, but it has a broad purpose: it’s going to go places and will be used in reports and dashboards where the accuracy of that data is key. 
Big data in Hadoop

Big Data repositories rarely undergo (at least initially) the full quality control rigors of data being injected into a warehouse, because not only is prepping data for some of the newer analytic methods characterized by Hadoop use cases cost prohibitive (which we talk about in the next chapter), but the data isn’t likely to be distributed like data warehouse data. We could say that data warehouse data is trusted enough to be “public,” while Hadoop data isn’t as trusted (public can mean vastly distributed within the company and not for external consumption), and although this will likely change in the future, today this is something that experience suggests characterizes these repositories.

Specific pieces of data have been stored based on their perceived value, and therefore any information beyond those pre-selected pieces is unavailable. This is in contrast to a Hadoop-based repository scheme where the entire business entity is likely to be stored and the fidelity of the Tweet, transaction, Facebook post, and more is kept intact. 

Data in Hadoop might seem of low value today, or its value nonquantified, but it can in fact be the key to questions yet unasked. IT departments pick and choose high-valued data and put it through rigorous cleansing and transformation processes because they know that data has a high known value per byte (a relative phrase, of course).

ETL and Big data
Stockphotos.io

Why else would a company put that data through so many quality control processes? 

Of course, since the value per byte is high, the business is willing to store it on relatively higher cost infrastructure to enable that interactive, often public, navigation with the end user communities, and the CIO is willing to invest in cleansing the data to increase its value per byte.
  • With Big Data, you should consider looking at this problem from the opposite view: With all the volume and velocity of today’s data, there’s just no way that you can afford to spend the time and resources required to cleanse and document every piece of data properly, because it’s just not going to be economical. 

What’s more, how do you know if this Big Data is even valuable? 

Are you going to go to your CIO and ask her to increase her capital expenditure (CAPEX) and operational expenditure (OPEX) costs by fourfold to quadruple the size of your warehouse on a hunch? 

For this reason, we like to characterize the initial nonanalyzed raw Big Data as having a low value per byte, and, therefore, until it’s proven otherwise, you can’t afford to take the path to the warehouse; however, given the vast amount of data, the potential for great insight (and therefore greater competitive advantage in your own market) is quite high if you can analyze all of that data.
  • The idea of cost per compute, which follows the same pattern as the value per byte ratio. If you consider the focus on the quality data in traditional systems we outlined earlier, you can conclude that the cost per compute in a traditional data warehouse is relatively high (which is fine, because it’s a proven and known higher value per byte), versus the cost of Hadoop, which is low.
Of course, other factors can indicate that certain data might be of high value yet never make its way into the warehouse, or there’s a desire for it to make its way out of the warehouse into a lower cost platform; either way, you might need to cleanse some of that data in Hadoop, and IBM can do that (a key differentiator). 

For example, unstructured data can’t be easily stored in a warehouse.

Indeed, some warehouses are built with a predefined corpus of questions in mind. Although such a warehouse provides some degree of freedom for query and mining, it could be that it’s constrained by what is in the schema (most unstructured data isn’t found here) and often by a performance envelope that can be a functional/operational hard limit. Again, as we’ll reiterate often in this book, we are not saying a Hadoop platform such as IBM InfoSphere BigInsights is a replacement for your warehouse; instead, it’s a complement.
  • A Big Data platform lets you store all of the data in its native business object format and get value out of it through massive parallelism on readily available components. For your interactive navigational needs, you’ll continue to pick and choose sources and cleanse that data and keep it in warehouses. But you can get more value out of analyzing more data (that may even initially seem unrelated) in order to paint a more robust picture of the issue at hand. 
Indeed, data might sit in Hadoop for a while, and when you discover its value, it might migrate its way into the warehouse when its value is proven and sustainable.

Comments

Popular posts from this blog

Hyperledger Fabric Real Interview Questions Read Today

I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…

Blue Prism complete tutorials download now

Blue prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career.
The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews.
RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation.

RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism Blogs

PL/SQL: Popular Reserved Words

Perfect in PL/SQL is an art. To become this you need to understand top reserved words and their meanings. The below list is useful for your projects.


Top List of PL/SQL Reserved Words.. Before you start knowing reserved words, wait one moment. The reserved words all are similar to words that you use in normal SQL. ALL*DESC*ISOLATIONOUTSQLERRMALTER*DISTINCT*JAVAPACKAGESTART*AND*DOLEVEL*PARTITIONSTDDEVANY*DROP*LIKE*PCTFREE*SUBTYPEARRAYELSE*LIMITEDPLS_INTEGERSUCCESSFUL*AS*ELSIFLOCK*POSITIVESUMASC*ENDLONG*POSITIVENSYNONYM*AUTHIDEXCEPTIONLOOPPRAGMASYSDATE*AVGEXCLUSIVE*MAXPRIOR*TABLE*BEGINEXECUTEMINPRIVATETHEN*BETWEEN*EXISTS*MINUS*PROCEDURETIMEBINARY_INTEGEREXITMINUTEPUBLIC*TIMESTAMPINTEGEREXTENDSMLSLABEL*RAISE

Automation developer these are top Skills you need to learn

Robotic process automation is an upcoming IT skill. Three tools are popular. It is difficult to learn all three tool. So, learn anyone tool to start your career in automation.
To get a job in this line, I found in my research that some programming skills and Hand-on training on any one of the tools is required. Also, try to know the differences between popular RPA tools.
Skills Companies Looking in Automation Engineers All big companies looking for candidates having experience in Automation anywhere, Blue Prism and UIPath. It is not possible to learn all tools. Learn anyone tool and do practice well.

Ok.

You may ask a question about how to do it. Join in good training institute and learn one tool.  Take online classes to learn faster.

To learn Uipath try here. Also, you can enroll online course to learn UiPath.

UiPath GO The list of IT skills you needAutomation anywhere/Blue Prism/Uipath .Net/C#/Java/SQL skills MS-Visio Power Builder Python scripts/Unix Scripts/Perl Scripts HTML/CSS/J…

Three popular RPA tools functional differences

Robotic process automation is growing area and many IT developers across the board started up-skill in this popular area. I have written this post for the benefit of Software developers who are interested in RPA also called Robotic Process Automation.


In my previous post, I have described that total 12 tools are available in the market. Out of those 3 tools are most popular. Those are Automation anywhere, BluePrism and Uipath. Many programmers asked what are the differences between these tools. I have given differences of all these three RPA tools.

BluePrism Blue Prism has taken a simple concept, replicating user activity on the desktop, and made it enterprise strength. The technology is scalable, secure, resilient, and flexible and is supported by a comprehensive methodology, operational framework and provided as packaged software.The technology is developed and deployed within a “corridor of IT governance” and has sophisticated error handling and process modelling capabilities to ens…

8 Top Data Science Platform Developers in the World

Top data science tools and platforms providers across the world. Useful information for data science and data analytics developers.

Data Science is a combination of multiple skills. AI and Machine Learning are part of data science. You can create AI and Machine Learning products with data. 

Related Posts

Top Skills You Need for Data Science CareerData Science Sample Project an Example

Top 100 Hadoop Complex Interview Questions (Part 3 of 4)

These are complex Hadoop interview questions. This is my 3rd set of questions useful for your interviews. 1). What are the features of Standalone (local) mode? Ans). In stand-alone mode there are no daemons, everything runs on a single JVM. It has no DFS and utilizes the local file system. Stand-alone mode is suitable only for running MapReduce programs during development. It is one of the least used environments.

2). What are the features of Pseudo mode?
Ans). The pseudo mode is used both for development and in the QA environment. In the Pseudo mode, all the daemons run on the same machine.

3). Can we call VMs as pseudos?
Ans). No, VMs are not pseudos because VM is something different and pseudo is very specific to Hadoop.

4). What are the features of Fully Distributed mode?

Ans). The fully Distributed mode is used in the production environment, where we have ‘n’ number of machines forming a Hadoop cluster. Hadoop daemons run on a cluster of machines. There is one host onto which Namenod…

Tokenization story you need Vault based Vs Vault-less

The term tokenization refers to create a numeric or alphanumeric number in place of the original card number. It is difficult for hackers to get original card numbers.

Vault-Tokenization is a concept a Vault server create a new Token for each transaction when Customer uses Credit or Debit Card at Merchant outlets 
Let us see an example,  data analysis. Here, card numbers masked with other junk characters for security purpose.

Popular Tokenization ServersThere are two kinds of servers currently popular for implementing tokenization.
Vault-based Vault-less Video Presentation on Tokenization
Vault-based server The term vault based means both card number and token will be stored in a Table usually Teradata tables. During increasing volume of transactions, the handling of Table is a big challenge.
Every time during tokenization it stores a record for each card and its token. When you used a card multiple times, each time it generates multiple tokens. It is a fundamental concept.
So the challe…