Skip to main content

How to manage big data and how hadoop will help us to do so?

When the data started moving towards its unstructured form to the structured one and people started moving towards many computers from single computer with heavy configuration (commodity hardware), the need arose for the technology to evolve.
Now firstly we shall understand about which unstructured data are we talking about? So the answer to the question is Big data. Let's understand a little about Big data first.
Big data refers to data sets that are too large or complex for traditional data-processing application software to adequately deal with. Data with many cases (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source. Big data was originally associated with three key concepts: volume, variety, and velocity. Other concepts later attributed with big data are veracity (i.e., how much noise is in the data) and value.
After this concept, let's talk about the problems faced while using this data, there are various levels on which we work or we can say that work like storing, processing of data, use of commodity hardware, in short, data is distributed here n there so that it can be stored on commodity hardware for every application firstly and secondly we can say that failures and crashes were to be faced and handled as computer networks are not very reliable and last but not the least, third one is to write the heart of the program.
All the teams in companies like Google had above three requirements-
Data storage 
Failures 
Heart (core software application)
but later on the requirement got cleared in Google's mind that if all want to write the same things then why don't we share all the common things in a framework so that developers will only have to concentrate on the main thing, that is, the "heart" of the program(whatever they need and whatever problems they are facing to make that) and can leave rest of the work on the framework. Many such frameworks are now a days available but we will talk here about Hadoop.
Hadoop is an open source data processing software commonly used for storing and processing of large amounts of data running on clustered applications. This framework was created by Doug Cutting and Mike Caferella in 2005. 
Some new terms also introduced at that time like "programming paradigm(because of the various machines being used parallel and data stored on them can be used by utilizing their full power), Map Reduce and Google File System. These terms can be understood as-
Programming paradigm-New way of writing the program in such a way that other things will work in parallel way.
Map Reduce- An advanced programming model capable of processing of huge data sets.
Google File System- GFS is an enhanced distributed file system created by Google for the purpose of providing storage and access to large amounts of data using clusters of commodity hardware.

REFERENCES:
What is big data?
Labels:

Comments

Post a Comment

Top 5 Most Read Posts

Who is Peter Lynch and what is his philosophy in equity market investment? 25 Golden Rules of the most successful Fund Manager.

Peter Lynch (born January 19, 1944) is an American investor, mutual fund manager, and philanthropist. As the manager of the Magellan Fund at Fidelity Investments between 1977 and 1990, Lynch averaged a 29.2% annual return, consistently more than doubling the S&P 500 stock market index and making it the best-performing mutual fund in the world. During his 13 year tenure, assets under management increased from $18 million to $14 billion. He also co-authored a number of books and papers on investing and coined a number of well known mantras of modern individual investing strategies, such as Invest in what you know and ten bagger. Lynch is consistently described as a "legend" by the financial media for his performance record. Base on his career I have compiled his investing rules here. 25 GOLDEN RULES by @Peter Lynch 1: Nobody can predict interest rates, the future direction of the economy or the stock market. Dismiss all such forecasts & concentrate on what's actual...
3 comments
Read more

What is version checking in Hibernate ?

Version checking used in hibernate when more then one thread trying to access same data. For example : User A edit the row of the TABLE for update ( In the User Interface changing data This is user thinking time) and in the same time User B edit the same record for update and click the update. Then User A click the Update and update done. Change made by user B is gone. In hibernate you can prevent slate object updation using version checking. Check the version of the row when you are updating the row. Get the version of the row when you are fetching the row of the TABLE for update. On the time of updation just fetch the version number and match with your version number (on the time of fetching).
4 comments
Read more

What is wrong with HDFC securities? Are they doing some fraudulent activities or just causing issues with their platform as usually it don't work during market hours?

I have opened a DEMAT account with HDFC Securities in 2019 as HDFC group is well known for the customer services and I also hold a salary account with HDFC Bank, DEMAT account with the following conditions/offers as expressed by the executive. Trading Account Opening Charges (One Time) :  ₹999 (At that time it offered on lower price, I forget the exact price) Trading Annual Maintenance Charges AMC (Yearly Fee) : ₹0 Demat Account Opening Charges (One Time) : ₹0 Demat Account Annual Maintenance Charges AMC (Yearly Fee) : ₹750, nil if portfolio value below ₹2 lacs. Equity Delivery : 0.50% Equity Intraday : 0.05% Equity Futures : 0.05% Equity Options : ₹100 per lot or 1% of the premium (whichever is higher) Currency Futures : ₹23 per contract Currency Options : ₹20 per contract Commodity Futures : 0.02% for Intraday / 0.025% for positional Commodity Options : 0.02% for Intraday / 0.025% for positional It was going good but after 2-3 months I got to know that there are...
2 comments
Read more

Fundamental Analysis : Asian Paints Ltd.

Asian Paints group is the largest paint manufacturer in India also engaged in the business of manufacturing of varnishes, enamels or lacquers, surfacing preparation, organic composite solvents and thinners. It operates in 15 countries and has 26 paint manufacturing facilities in the world serving consumers in over 60 countries. Besides Asian Paints, the group operates around the world through its various brands viz. Asian Paints Berger, Apco Coatings, SCIB Paints, Taubmans, Causeway Paints and Kadisco Asian Paints. It also manufactures metal sanitary ware such as bath, sinks, washbasins and similar articles. Recently introduced Lightings, Furnishings and Furniture thus adding more products in the Home décor and Interior Design category. Market Cap ₹ 244,763 Cr. Current Price ₹ 2,552 High / Low ₹ 2,873 / 1,483 Stock P/E 89.0 Book Value ₹ 115 Dividend Yield 0.47 % ROCE 33.8 % ROE 27.5 % Face Value ₹ 1.00 Price to book value 22.2 EPS ₹ 28.7 High price all time ₹ 2,873 Low price all time ₹...
10 comments
Read more

What are JEE Containers? What are their different types?

Containers are the interface between a component and the low-level, platform-specific functionality that supports the component. Before it can be executed, a web, enterprise bean, or application client component must be assembled into a Java EE module and deployed into its container. The assembly process involves specifying container settings for each component in the Java EE application and for the Java EE application itself. Container settings customize the underlying support provided by the Java EE server, including such services as security, transaction management, Java Naming and Directory Interface (JNDI) API lookups, and remote connectivity. Here are some of the highlights. The Java EE security model lets you configure a web component or enterprise bean so that system resources are accessed only by authorized users. The Java EE transaction model lets you specify relationships among methods that make up a single transaction so that all methods in one transaction are treated as a ...
4 comments
Read more