Skip to main content

Top features of Apache Avro in Hadoop eco-System

Avro defines a data format designed to support data-intensive applications, and provides support for this format in a variety of programming languages.

The Hadoop ecosystem includes a new binary data serialization system — Avro. 

Avro provides:
·     Rich data structures.

·         A compact, fast, binary data format.
·         A container file, to store persistent data.
·         Remote procedure call (RPC).
·       Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.

Its functionality is similar to the other marshaling systems such as Thrift, Protocol Buffers, and so on.

The main differentiators of Avro include the following:

[Hadoop Interview Questions]
[Hadoop Interview Questions]
Dynamic typing — The Avro implementation always keeps data and its corresponding schema together. As a result, marshaling/unmarshaling operations do not require either code generation or static data types. This also allows generic data processing.

Untagged data — Because it keeps data and schema together, Avro
marshaling/unmarshaling does not require type/size information or manually assigned IDs to be encoded in data. As a result, Avro serialization produces a smaller output.

Enhanced versioning support — In the case of schema changes, Avro contains both schemas, which enables you to resolve differences symbolically based on the field names.
Because of high performance, a small codebase, and compact resulting data, there is a wide adoption of Avro not only in the Hadoop community, but also by many other NoSQL implementations (including Cassandra).

At the heart of Avro is a data serialization system. Avro can either use reflection to dynamically generate schemas of the existing Java objects, or use an explicit Avro schema — a JavaScript Object Notation (JSON) document describing the data format. Avro schemas can contain both simple and complex types.

Simple data types supported by Avro include null, boolean, int, long, float, double, bytes, and string. Here, null is a special type, corresponding to no data, and can be used in place of any data type.

Complex types supported by Avro include the following:
Record — This is roughly equivalent to a C structure. A record has a name and optional namespace, document, and alias. It contains a list of named attributes that can be of any Avro type.
Enum — This is an enumeration of values. Enum has a name, an optional namespace, document, and alias, and contains a list of symbols (valid JSON strings).
Array — This is a collection of items of the same type.
Map — This is a map of keys of type string and values of the specified type.
Union — This represents an or option for the value. A common use for unions is to specify nullable values.

Comments

Popular posts from this blog

10 Tricky Interview Questions On Storm

Storm is real time computation system. It is a flagship software from Apache foundation. Has the capability to process in stream data. Storm is capable to integrate traditional databases. The list given below are tricky and highly useful for your next interview.
Bench mark for Storm is a million tuples processed per second per node. Tricky Interview Questions1) Real uses of Storm?

A) You can use in realtime analytics, online machine learning, continuous computation, distributed RPC, ETL

2) What are different availble layers on Storm?
FluxSQLStreams APITrident3)  Real use of SQL API on top of Storm?
A) You can run SQL queries on stream data
4) Most popular integrations to Storm? HDFSCassandraJDBCHIVEHBase 5) What are different possible Containers integration with Storm? YARNDOCKERMESOS6) What is Local Mode?

A) Running topologies in Local server we can say as Local Mode.

7) Where all the Events Stored in Storm?
A) Event Logger mechanism saves all events

8) What are Serializable data types in …

Blue Prism complete tutorials download now

Blueprsim is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career.
The number one and most popular tool in automation is Blue prism. In this post I have given references for popular materials and resources, so that you can use for your interviews.
Why You Need to Learn RPA blue prsim tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is really good option if you are learner of Robotic process automation.
RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism BlogsVideo…

Blockchain Smart contract behind mechanism you need to learn quickly

Smart contract in Blockchain is a kind of software application that works without human intervention based on the transaction logs and provide solution to user request. I want to share the back end mechanism in Smart Contract of Blockchain. Smart Contract Mechanism What is Smart ContractA smart contract is a protocol which can auto execute, facilitate, verify or enforce the negotiation of a contract.Agreement between two parties you can say as a contract.Incorporating the rules of physical contract into computing world, you can say as smart contractBlockchain supports you to create smart contracts.Smart Contracts are self-executing programs which run on the blockchain and are capable of enforcing rulesUsing Blockchain as platform and making an agreement or contract between more than two parties, you can say as Smart Contract.Traditional Markets  4 Top Benefits of Smart ContractCurrently smart contracts are being used only in Crypto CurrenciesNow Smart Contracts being used in all financ…