Performance utilities in Hive

hive_best_practice

Before taking you in details of utilities provided by Hive, let me explain few components to get execution flow and where the related information stored in system.

Hive is a data warehouse software best suited for OLAP (OnLine Analytical Processing) workloads to handle and query over vast volume of data residing in a distributed storage. The Hadoop Distributed File System (HDFS) is the ecosystem in which Hive maintains the data reliably and survives from hardware failures. Hive is the only SQL-like relational big data warehousing approach developed on top of Hadoop.

HiveQL as described, is an SQL-like query language for expressing queries in Hive. After a query is issued through an interface in Hive, it undergoes several processing phases including parsing, semantic analysis, logical plan generation, physical plan generation, etc; ultimately the plan is transformed to a sequence of mapreduce operations which are then executed over Hadoop.

What is Hive Meta-store:-

Hive system catalog contains schemas, tables, columns, and their types, tables’ locations, statistics and other information essential for data management. Since meta data should be available fast, Hive uses a traditional RDBMS (e.g., Derby SQL Server, MySQL Server, etc.) to manage meta data rather than using the HDFS. Because Java works with objects and an RDBMS uses the relational model, an Object Relational Mapping (ORM), called DataNucleus, is accompanied by the RDBMS to translate between objects and relational schema.

Driver:-

The component that receives the query, after it is received by the UI from the user, and manages the lifespan of a query inside Hive. It also implements the notion of session handles and retrieves the session statistics. Session has the same meaning as it does in traditional databases: “The SQL operations that are performed while a connection is active form a session.”

The Driver consists of three main components, namely, Compiler, Optimizer, and Executor. The compiler translates HiveQL into a DAG (Directed Acyclic Graph) of mapreduce tasks that are executed by the executor or execute engine in the order of their dependencies. The optimizer resides at some point between the compiler and executor to improve the performance.

Hive Server:-

Hive server or Thrift Server allows access to Hive with a single port, that is, it allows programmatically access to Hive remotely. Therefore it provides means to integrate Hive with other applications. Thrift is a scalable cross-language service development framework; or simply, a binary communication protocol. Clients in different programming languages can communicate seamlessly with Hive using the “thrift interface”. Here, by client we mean any source that issues a query.

How the components of Hive architecture interact with each other?
A user submits the query via Hive CLI/Hive web Interface, JDBC/ODBC, or Thrift interface. The Driver receives the query and passes it to the compiler. Compiler does the typical parsing, type checking, semantic analysis, and pings the meta-store if needed. Finally it generates a logical query plan that is sent to the optimizer. The optimized query plan is converted to a DAG of mapreduce jobs. The executor executes these jobs in the order of dependency on Hadoop. Figure 6 shows the steps of this process.

Hive provides the EXPLAIN and ANALYZE statements that can be used as utilities to check and identify the performance of queries.

The EXPLAIN statement:-
Hive provides an EXPLAIN command to return a query execution plan without running the query. We can use an EXPLAIN command for queries if we have a doubt or a concern about performance. The EXPLAIN command will help to see the difference between two or more
queries for the same purpose. The syntax for EXPLAIN is as follows:
EXPLAIN [EXTENDED|DEPENDENCY|AUTHORIZATION] hive_query

The following keywords can be used:-
EXTENDED: This provides additional information for the operators in the plan, such as file pathname and abstract syntax tree.
DEPENDENCY: This provides a JSON format output that contains a list of tables and partitions that the query depends on. It is available since HIVE 0.10.0.
AUTHORIZATION: This lists all entities needed to be authorized including input and output to run the Hive query and authorization failures, if any. It is available since HIVE 0.14.0.

The ANALYZE statement:-
Hive statistics are a collection of data that describe more details, such as the number of rows, number of files, and raw data size, on the objects in the Hive database. Statistics is a metadata of Hive data. Hive supports statistics at the table, partition, and column level.
These statistics serve as an input to the Hive Cost-Based Optimizer (CBO), which is an optimizer to pick the query plan with the lowest cost in terms of system resources required to complete the query.
The statistics are gathered through the ANALYZE statement since Hive 0.10.0 on tables, partitions, and columns as given in the following examples:

ANALYZE TABLE employee COMPUTE STATISTICS;

ANALYZE TABLE employee_partitioned PARTITION(year=2014, month=12) COMPUTE STATISTICS;

ANALYZE TABLE employee_id COMPUTE STATISTICS FOR COLUMNS employee_id;

Once the statistics are built, we can check the statistics by the DESCRIBE EXTENDED/FORMATTED statement. From the table/partition output, we can find the statistics information inside the parameters, such as parameters:{numFiles=1, COLUMN_STATS_ACCURATE=true, transient_lastDdlTime=1417726247, numRows=4,totalSize=227, rawDataSize=223}). The following is an example:

DESCRIBE EXTENDED employee_partitioned PARTITION(year=2014, month=12);
or
DESCRIBE EXTENDED employee;

Hive statistics are persisted in the metastore to avoid computing them every time. For newly created tables and/or partitions, statistics are automatically computed by default if we enable the following setting:
SET hive.stats.autogather=ture;

11 thoughts on “Performance utilities in Hive

  1. Heya i抦 for the first time here. I came across this board and I find It really useful & it helped me out much. I hope to give something back and help others like you helped me.

  2. Somebody essentially help to make seriously articles I would state. This is the first time I frequented your web page and thus far? I surprised with the research you made to create this particular publish amazing. Excellent job!

  3. hello!,I love your writing very so much! share we keep in touch extra about your post on AOL? I need a specialist on this area to solve my problem. May be that’s you! Taking a look ahead to see you.

  4. I抳e been exploring for a little bit for any high-quality articles or blog posts on this sort of area . Exploring in Yahoo I at last stumbled upon this website. Reading this info So i抦 happy to convey that I have a very good uncanny feeling I discovered just what I needed. I most certainly will make sure to do not forget this site and give it a glance on a constant basis.

  5. Thanks for the strategies you have contributed here. In addition, I believe there are numerous factors that will keep your insurance premium straight down. One is, to contemplate buying vehicles that are inside good directory of car insurance providers. Cars which have been expensive are usually more at risk of being robbed. Aside from that insurance policies are also depending on the value of your car or truck, so the more costly it is, then the higher the particular premium you have to pay.

  6. Have you ever thought about creating an ebook or guest authoring on other sites? I have a blog centered on the same topics you discuss and would love to have you share some stories/information. I know my audience would value your work. If you are even remotely interested, feel free to send me an e-mail.

  7. hello there and thank you for your info ?I have certainly picked up something new from right here. I did however expertise some technical points using this site, since I experienced to reload the web site a lot of times previous to I could get it to load correctly. I had been wondering if your web hosting is OK? Not that I’m complaining, but sluggish loading instances times will often affect your placement in google and could damage your quality score if ads and marketing with Adwords. Anyway I am adding this RSS to my e-mail and can look out for a lot more of your respective interesting content. Ensure that you update this again very soon..

  8. I’m really enjoying the theme/design of your blog. Do you ever run into any browser compatibility problems? A couple of my blog readers have complained about my website not working correctly in Explorer but looks great in Opera. Do you have any suggestions to help fix this problem?

  9. Whether you believe in God or not, this is a must-read message!!!

    Throughout time, we can see how we have been slowly conditioned to come to this point where we are on the verge of a cashless society. Did you know that the Bible foretold of this event almost 2,000 years ago?

    In Revelation 13:16-18, we read,

    “He (the false prophet who decieves many by his miracles) causes all, both small and great, rich and poor, free and slave, to receive a mark on their right hand or on their foreheads, and that no one may buy or sell except one who has the mark or the name of the beast, or the number of his name.

    Here is wisdom. Let him who has understanding calculate the number of the beast, for it is the number of a man: His number is 666.”

    Referring to the last generation, this could only be speaking of a cashless society. Why? Revelation 13:17 tells us that we cannot buy or sell unless we receive the mark of the beast. If physical money was still in use, we could buy or sell with one another without receiving the mark. This would contradict scripture that states we need the mark to buy or sell!

    These verses could not be referring to something purely spiritual as scripture references two physical locations (our right hand or forehead) stating the mark will be on one “OR” the other. If this mark was purely spiritual, it would indicate only in one place.

    This is where it really starts to come together. It is shocking how accurate the Bible is concerning the implatnable RFID microchip. These are notes from a man named Carl Sanders who worked with a team of engineers to help develop this RFID chip

    “Carl Sanders sat in seventeen New World Order meetings with heads-of-state officials such as Henry Kissinger and Bob Gates of the C.I.A. to discuss plans on how to bring about this one-world system. The government commissioned Carl Sanders to design a microchip for identifying and controlling the peoples of the world—a microchip that could be inserted under the skin with a hypodermic needle (a quick, convenient method that would be gradually accepted by society).

    Carl Sanders, with a team of engineers behind him, with U.S. grant monies supplied by tax dollars, took on this project and designed a microchip that is powered by a lithium battery, rechargeable through the temperature changes in our skin. Without the knowledge of the Bible (Brother Sanders was not a Christian at the time), these engineers spent one-and-a-half-million dollars doing research on the best and most convenient place to have the microchip inserted.

    Guess what? These researchers found that the forehead and the back of the hand (the two places the Bible says the mark will go) are not just the most convenient places, but are also the only viable places for rapid, consistent temperature changes in the skin to recharge the lithium battery. The microchip is approximately seven millimeters in length, .75 millimeters in diameter, about the size of a grain of rice. It is capable of storing pages upon pages of information about you. All your general history, work history, crime record, health history, and financial data can be stored on this chip.

    Brother Sanders believes that this microchip, which he regretfully helped design, is the “mark” spoken about in Revelation 13:16–18. The original Greek word for “mark” is “charagma,” which means a “scratch or etching.” It is also interesting to note that the number 666 is actually a word in the original Greek. The word is “chi xi stigma,” with the last part, “stigma,” also meaning “to stick or prick.” Carl believes this is referring to a hypodermic needle when they poke into the skin to inject the microchip.”

    Mr. Sanders asked a doctor what would happen if the lithium contained within the RFID microchip leaked into the body. The doctor replied by saying a terrible sore would appear in that location. This is what the book of Revelation says:

    “And the first (angel) went, and poured out his vial on the earth; and there fell a noisome and grievous sore on the men which had the mark of the beast, and on them which worshipped his image” (Revelation 16:2).

    You can read more about it here–and to also understand the mystery behind the number 666: https://2ruth.org/rfid-mark-of-the-beast-666-revealed/

    The third angel’s warning in Revelation 14:9-11 states,

    “Then a third angel followed them, saying with a loud voice, ‘If anyone worships the beast and his image, and receives his mark on his forehead or on his hand, he himself shall also drink of the wine of the wrath of God, which is poured out full strength into the cup of His indignation. He shall be tormented with fire and brimstone in the presence of the holy angels and in the presence of the Lamb. And the smoke of their torment ascends forever and ever; and they have no rest day or night, who worship the beast and his image, and whoever receives the mark of his name.'”

    Who is Barack Obama, and why is he still in the public scene?

    So what’s in the name? The meaning of someone’s name can say a lot about a person. God throughout history has given names to people that have a specific meaning tied to their lives. How about the name Barack Obama? Let us take a look at what may be hiding beneath the surface.

    Jesus says in Luke 10:18, “…I saw Satan fall like lightning from heaven.”

    The Hebrew Strongs word (H1299) for “lightning”: “bârâq” (baw-rawk)

    In Isaiah chapter 14, verse 14, we read about Lucifer (Satan) saying in his heart:

    “I will ascend above the heights of the clouds, I will be like the Most High.”

    In the verses in Isaiah that refer directly to Lucifer, several times it mentions him falling from the heights or the heavens. The Hebrew word for the heights or heavens used here is Hebrew Strongs 1116: “bamah”–Pronounced (bam-maw’)

    In Hebrew, the letter “Waw” or “Vav” is often transliterated as a “U” or “O,” and it is primarily used as a conjunction to join concepts together. So to join in Hebrew poetry the concept of lightning (Baraq) and a high place like heaven or the heights of heaven (Bam-Maw), the letter “U” or “O” would be used. So, Baraq “O” Bam-Maw or Baraq “U” Bam-Maw in Hebrew poetry similar to the style written in Isaiah, would translate literally to “Lightning from the heights.” The word “Satan” in Hebrew is a direct translation, therefore “Satan.”

    So when Jesus told His disciples in Luke 10:18 that He beheld Satan fall like lightning from heaven, if this were to be spoken by a Jewish Rabbi today influenced by the poetry in the book of Isaiah, he would say these words in Hebrew–the words of Jesus in Luke 10:18 as, And I saw Satan as Baraq O Bam-Maw.

    The names of both of Obama’s daughters are Malia and Natasha. If we were to write those names backward (the devil does things in reverse) we would get “ailam ahsatan”. Now if we remove the letters that spell “Alah” (Allah being the false god of Islam), we get “I am Satan”. Coincidence? I don’t think so.

    Obama’s campaign logo when he ran in 2008 was a sun over the horizon in the west, with the landscape as the flag of the United States. In Islam, they have their own messiah that they are waiting for called the 12th Imam, or the Mahdi (the Antichrist of the Bible), and one prophecy concerning this man’s appearance is the sun rising in the west.

    “Then I saw another angel flying in the midst of heaven, having the everlasting gospel to preach to those who dwell on the earth—to every nation, tribe, tongue, and people— saying with a loud voice, ‘Fear God and give glory to Him, for the hour of His judgment has come; and worship Him who made heaven and earth, the sea and springs of water.'” (Revelation 14:6-7)

    Why have the word’s of Jesus in His Gospel accounts regarding His death, burial, and resurrection, been translated into over 3,000 languages, and nothing comes close? The same God who formed the heavens and earth that draws all people to Him through His creation, likewise has sent His Word to the ends of the earth so that we may come to personally know Him to be saved in spirit and in truth through His Son Jesus Christ.

    Jesus stands alone among the other religions that say to rightly weigh the scales of good and evil and to make sure you have done more good than bad in this life. Is this how we conduct ourselves justly in a court of law? Bearing the image of God, is this how we project this image into reality?

    Our good works cannot save us. If we step before a judge, being guilty of a crime, the judge will not judge us by the good that we have done, but rather by the crimes we have committed. If we as fallen humanity, created in God’s image, pose this type of justice, how much more a perfect, righteous, and Holy God?

    God has brought down His moral laws through the 10 commandments given to Moses at Mt. Siani. These laws were not given so we may be justified, but rather that we may see the need for a savior. They are the mirror of God’s character of what He has put in each and every one of us, with our conscious bearing witness that we know that it is wrong to steal, lie, dishonor our parents, murder, and so forth.

    We can try and follow the moral laws of the 10 commandments, but we will never catch up to them to be justified before a Holy God. That same word of the law given to Moses became flesh about 2,000 years ago in the body of Jesus Christ. He came to be our justification by fulfilling the law, living a sinless perfect life that only God could fulfill.

    The gap between us and the law can never be reconciled by our own merit, but the arm of Jesus is stretched out by the grace and mercy of God. And if we are to grab on, through faith in Him, He will pull us up being the one to justify us. As in the court of law, if someone steps in and pays our fine, even though we are guilty, the judge can do what is legal and just and let us go free. That is what Jesus did almost 2,000 years ago on the cross. It was a legal transaction being fulfilled in the spiritual realm by the shedding of His blood.

    For God takes no pleasure in the death of the wicked (Ezekiel 18:23). This is why in Isaiah chapter 53, where it speaks of the coming Messiah and His soul being a sacrifice for our sins, why it says it pleased God to crush His only begotten Son.

    This is because the wrath that we deserve was justified by being poured out upon His Son. If that wrath was poured out on us, we would all perish to hell forever. God created a way of escape by pouring it out on His Son whose soul could not be left in Hades but was raised and seated at the right hand of God in power.

    So now when we put on the Lord Jesus Christ (Romans 13:14), God no longer sees the person who deserves His wrath, but rather the glorious image of His perfect Son dwelling in us, justifying us as if we received the wrath we deserve, making a way of escape from the curse of death–now being conformed into the image of the heavenly man in a new nature, and no longer in the image of the fallen man Adam.

    Now what we must do is repent and put our trust and faith in the savior, confessing and forsaking our sins, and to receive His Holy Spirit that we may be born again (for Jesus says we must be born again to enter the Kingdom of God–John chapter 3). This is not just head knowledge of believing in Jesus, but rather receiving His words, taking them to heart, so that we may truly be transformed into the image of God. Where we no longer live to practice sin, but rather turn from our sins and practice righteousness through faith in Him in obedience to His Word by reading the Bible.

    Our works cannot save us, but they can condemn us; it is not that we earn our way into everlasting life, but that we obey our Lord Jesus Christ:

    “And having been perfected, He became the author of eternal salvation to all who obey Him.” (Hebrews 5:9)

    “Now I saw a new heaven and a new earth, for the first heaven and the first earth had passed away. Also there was no more sea. Then I, John, saw the holy city, New Jerusalem, coming down out of heaven from God, prepared as a bride adorned for her husband. And I heard a loud voice from heaven saying, ‘Behold, the tabernacle of God is with men, and He will dwell with them, and they shall be His people. God Himself will be with them and be their God. And God will wipe away every tear from their eyes; there shall be no more death, nor sorrow, nor crying. There shall be no more pain, for the former things have passed away.’

    Then He who sat on the throne said, ‘Behold, I make all things new.’ And He said to me, ‘Write, for these words are true and faithful.’

    And He said to me, ‘It is done! I am the Alpha and the Omega, the Beginning and the End. I will give of the fountain of the water of life freely to him who thirsts. He who overcomes shall inherit all things, and I will be his God and he shall be My son. But the cowardly, unbelieving, abominable, murderers, sexually immoral, sorcerers, idolaters, and all liars shall have their part in the lake which burns with fire and brimstone, which is the second death.'” (Revelation 21:1-8).

Leave a Reply

Your email address will not be published. Required fields are marked *