Hive Interview Questions and Answers

1) Explain what is Hive?

Hive is an ETL and Data warehousing tool developed on top of Hadoop Distributed File System (HDFS). It is a data warehouse framework for querying and analysis of data that is stored in HDFS. Hive is an open-source-software that lets programmers analyze large data sets on Hadoop.

2) When to use Hive?

• Hive is useful when making data warehouse applications
• When you are dealing with static data instead of dynamic data
• When application is on high latency (high response time)
• When a large data set is maintained
• When we are using queries instead of scripting

3) Mention what are the different modes of Hive?

Depending on the size of data nodes in Hadoop, Hive can operate in two modes.

These modes are,

• Local mode
• Map reduce mode

4) Mention when to use Map reduce mode?

Map reduce mode is used when,

• It will perform on large amount of data sets and query going to execute in a parallel way
• Hadoop has multiple data nodes, and data is distributed across different node we use Hive in this mode
• Processing large data sets with better performance needs to be achieved

5) Mention key components of Hive Architecture?

Key components of Hive Architecture includes,

• User Interface
• Compiler
• Metastore
• Driver
• Execute Engine

6) Mention what are the different types of tables available in Hive?

There are two types of tables available in Hive.

• Managed table: In managed table, both the data and schema are under control of Hive
• External table: In the external table, only the schema is under the control of Hive.

7) Explain what is Metastore in Hive?

Metastore is a central repository in Hive.  It is used for storing schema information or metadata in the external database.

8) Mention what Hive is composed of ?

Hive consists of 3 main parts,

1. Hive Clients
2. Hive Services
3. Hive Storage and Computing

9) Mention what are the type of database does Hive support ?

For single user metadata storage, Hive uses derby database and for multiple user Metadata or shared Metadata case Hive uses MYSQL.

10) Mention Hive default read and write classes?

Hive default read and write classes are

1. TextInputFormat/HiveIgnoreKeyTextOutputFormat
2. SequenceFileInputFormat/SequenceFileOutputFormat

11) Mention what are the different modes of Hive?

Different modes of Hive depends on the size of data nodes in Hadoop.

These modes are,

• Local mode
• Map reduce mode

12) Why is Hive not suitable for OLTP systems?

Hive is not suitable for OLTP systems because it does not provide insert and update function at the row level.

13.Differentiate between Hive and HBase

14) Explain what is a Hive variable? What for we use it?

Hive variable is created in the Hive environment that can be referenced by Hive scripts. It is used to pass some values to the hive queries when the query starts executing.

15) Mention what is ObjectInspector functionality in Hive?

ObjectInspector functionality in Hive is used to analyze the internal structure of the columns, rows, and complex objects.  It allows to access the internal fields inside the objects.

16) Mention what is (HS2) HiveServer2?

It is a server interface that performs following functions.

• It allows remote clients to execute queries against Hive
• Retrieve the results of mentioned queries

Some advanced features Based on Thrift RPC in its latest version include

• Multi-client concurrency
• Authentication

17) Mention what Hive query processor does?

Hive query processor convert graph of MapReduce jobs with the execution time framework.  So that the jobs can be executed in the order of dependencies.

18) Mention what are the components of a Hive query processor?

The components of a Hive query processor include,

• Logical Plan Generation
• Physical Plan Generation
• Execution Engine
• Operators
• UDF’s and UDAF’s
• Optimizer
• Parser
• Semantic Analyzer
• Type Checking

19) Mention what is Partitions in Hive?

Hive organizes tables into partitions.

• It is one of the ways of dividing tables into different parts based on partition keys.
• Partition is helpful when the table has one or more Partition keys.
• Partition keys are basic elements for determining how the data is stored in the table.

20) Mention when to choose “Internal Table” and “External Table” in Hive?

In Hive you can choose internal table,

• If the processing data available in local file system
• If we want Hive to manage the complete lifecycle of data including the deletion

You can choose External table,

• If processing data available in HDFS
• Useful when the files are being used outside of Hive

21) Mention if we can name view same as the name of a Hive table?

No. The name of a view must be unique compared to all other tables and as views present in the same database.

22) Mention what are views in Hive?

In Hive, Views are Similar to tables. They are generated based on the requirements.

• We can save any result set data as a view in Hive
• Usage is similar to as views used in SQL
• All type of DML operations can be performed on a view

23) Explain how Hive Deserialize and serialize the data?

Usually, while read/write the data, the user first communicate with inputformat. Then it connects with Record reader to read/write record.  To serialize the data, the data goes to row. Here deserialized custom serde use object inspector to deserialize the data in fields.

24) What is Buckets in Hive?

• The data present in the partitions can be divided further into Buckets
• The division is performed based on Hash of particular columns that is selected in the table.

25) In Hive, how can you enable buckets?

In Hive, you can enable buckets by using the following command,

set.hive.enforce.bucketing=true;

26) In Hive, can you overwrite Hadoop MapReduce configuration in Hive?

Yes, you can overwrite Hadoop MapReduce configuration in Hive.

27) Explain how can you change a column data type in Hive?

You can change a column data type in Hive by using command,

ALTER TABLE table_name CHANGE column_name column_name new_datatype;

28) Mention what is the difference between order by and sort by in Hive?

• SORT BY will sort the data within each reducer. You can use any number of reducers for SORT BY operation.
• ORDER BY will sort all of the data together, which has to pass through one reducer. Thus, ORDER BY in hive uses a single

29) Explain when to use explode in Hive?

Hadoop developers sometimes take an array as input and convert into a separate table row. To convert complex data types into desired table formats, Hive use explode.

30) Mention how can you stop a partition form being queried?

You can stop a partition form being queried by using the ENABLE OFFLINE clause with ALTER TABLE statement.

31) Compare Pig and Hive

32)What is the definition of Hive? What is the present version of Hive and explain about ACID transactions in Hive?

Hive is an open source data warehouse system. We can use Hive for analyzing and querying in large data sets of Hadoop files. It’s similar to SQL. The present version of hive is 0.13.1. Hive supports ACID transactions: The full form of ACID is Atomicity, Consistency, Isolation, and Durability. ACID transactions are provided at the row levels, there are Insert, Delete, and Update options so that Hive supports ACID transaction.

• Insert
• Delete
• Update

33)Explain what is a Hive variable. What do we use it for?

Hive variable is basically created in the Hive environment that is referenced by Hive scripting languages. It provides to pass some values to the hive queries when the query starts executing. It uses the source command.

34)What kind of data warehouse application is suitable for Hive? What are the types of tables in Hive?

Hive is not considered as a full database. The design rules and regulations of Hadoop and HDFS put restrictions on what Hive can do.Hive is most suitable for data warehouse applications.
Where :

• Analyzing the relatively static data.
• Less Responsive time.
• No rapid changes in data.Hive doesn’t provide fundamental features required for OLTP, Online Transaction Processing.Hive is suitable for data warehouse applications in large data sets.Two types of tables in Hive

1. Managed table.
2. External table.

35)Can We Change settings within Hive Session? If Yes, How?

Yes we can change the settings within Hive session, using the SET command. It helps to change Hive job settings for an exact query.
Example: The following commands shows buckets are occupied according to the table definition.

hive> SET hive.enforce.bucketing=true;

We can see the current value of any property by using SET with the property name. SET will list all the properties with their values set by Hive.

hive> SET hive.enforce.bucketing;

hive.enforce.bucketing=true

And this list will not include defaults of Hadoop. So we should use the below like

SET -v

It will list all the properties including the Hadoop defaults in the system.

Interested in learning Hive? Well, we have the comprehensive Hive Training Course to give you a head start in your career.

36)Is it possible to add 100 nodes when we have 100 nodes already in Hive? How?

Yes, we can add the nodes by following the below steps.

1. Take a new system create a new username and password.
2. Install the SSH and with master node setup ssh connections.
3. Add ssh public_rsa id key to the authorized keys file.
4. Add the new data node host name, IP address and other details in /etc/hosts slaves file
   168.1.102 slave3.in slave3.
5. Start the Data Node on New Node.
6. Login to the new node like suhadoop or ssh -X hadoop@192.168.1.103.
7. Start HDFS of a newly added slave node by using the following command
   ./bin/hadoop-daemon.sh start data node.
8. Check the output of jps command on a new node

37)Explain the concatenation function in Hive with an example .

Concatenate function will join the input strings.We can specify the
‘N’ number of strings separated by a comma.
Example:

CONCAT (‘Intellipaat’,’-‘,’is’,’-‘,’a’,’-‘,’eLearning’,’-’,’provider’);

Output:

Intellipaat-is-a-eLearning-provider

So, every time we set the limits of the strings by ‘-‘. If it is common for every strings, then Hive provides another command

CONCAT_WS. In this case,we have to specify the set limits of operator first.

CONCAT_WS (‘-‘,’Intellipaat’,’is’,’a’,’eLearning’,‘provider’);

Output: Intellipaat-is-a-eLearning-provider.

38)Trim and Reverse function in Hive with examples.

Trim function will delete the spaces associated with a string.
Example:

TRIM(‘ INTELLIPAAT ‘);

Output:

INTELLIPAAT

To remove the Leading space

LTRIM(‘ INTELLIPAAT’);

To remove the trailing space

RTRIM(‘INTELLIPAAT ‘);

In Reverse function, characters are reversed in the string.

Example:

REVERSE(‘INTELLIPAAT’);

Output:

TAAPILLETNI

39)How to change the column data type in Hive? Explain RLIKE in Hive.

We can change the column data type by using ALTER and CHANGE.
The syntax is :

ALTER TABLE table_name CHANGE column_namecolumn_namenew_datatype;

Example: If we want to change the data type of the salary column from integer to bigint in the employee table.
ALTER TABLE employee CHANGE salary salary BIGINT;RLIKE: Its full form is Right-Like and it is a special function in the Hive. It helps to examine the two substrings. i.e, if the substring of A matches with B then it evaluates to true.
Example:

Trueà‘Intellipaat’ RLIKE ‘tell’

True (this is a regular expression)à‘Intellipaat’ RLIKE ‘^I.*’

40)What are the components used in Hive query processor?

The components of a Hive query processor include

• Logical Plan of Generation.
• Physical Plan of Generation.
• Execution Engine.
• UDF’s and UDAF’s.
• Semantic Analyzer.
• Type Checking

41)What is Buckets in Hive?

The present data is partitioned and divided into different Buckets. This data is divided on the basis of Hash of the particular table columns.

42)Explain process to access sub directories recursively in Hive queries.

By using below commands we can access sub directories recursively in Hive

hive> Set mapred.input.dir.recursive=true;

hive> Set hive.mapred.supports.subdirectories=true;

Hive tables can be pointed to the higher level directory and this is suitable for the directory structure which is like /data/country/state/city/

43)What are the components used in Hive query processor?

The components of a Hive query processor include

• Logical Plan of Generation
• Physical Plan of Generation
• Execution Engine
• Operators
• UDF’s and UDAF’s
• Optimizer
• Parser
• Semantic Analyzer
• Type Checking

44)How to skip header rows from a table in Hive?

Header records in log files
System=….
Version=…
Sub-version=….
In the above three lines of headers that we do not want to include in our Hive query. To skip header lines from our tables in the Hive,set a table property that will allow us to skip the header lines.

CREATE EXTERNAL TABLE employee (

name STRING,

job STRING,

dob STRING,

id INT,

salary INT)

ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘ ‘ STORED AS TEXTFILE

LOCATION ‘/user/data’

TBLPROPERTIES(“skip.header.line.count”=”2”);

45)What is the maximum size of string data type supported by hive? Mention the Hive support binary formats.

The maximum size of string data type supported by hive is 2 GB.
Hive supports the text file format by default and it supports the binary format Sequence files, ORC files, Avro Data files, Parquet files.
Sequence files: Splittable, compressible and row oriented are the general binary format.
ORC files: Full form of ORC is optimized row columnar format files. It is a Record columnar file and column oriented storage file. It divides the table in row split. In each split stores that value of the first row in the first column and followed sub subsequently.
AVRO data files: It is same as a sequence file splittable, compressible and row oriented, but except the support of schema evolution and multilingual binding support.

46)What is the precedence order of HIVE configuration?

We are using a precedence hierarchy for setting the properties

1. SET Command in HIVE
2. The command line –hiveconf option
3. Hive-site.XML
4. Hive-default.xml
5. Hadoop-site.xml
6. Hadoop-default.xml

47)If you run a select * query in Hive, Why does it not run MapReduce?

The hive.fetch.task.conversion property of Hive lowers the latency of mapreduce overhead and in effect when executing queries like SELECT, FILTER, LIMIT, etc., it skips mapreduce function

48)How Hive can improve performance with ORC format tables?

We can store the hive data in highly efficient manner in the Optimized Row Columnar file format. It can simplify many Hive file format limitations. We can improve the performance by using ORC files while reading, writing and processing the data.

Set hive.compute.query.using.stats-true;

Set hive.stats.dbclass-fs;

CREATE TABLE orc_table (

idint,

name string)

ROW FORMAT DELIMITED

FIELDS TERMINATED BY ‘\:’

LINES TERMINATED BY ‘\n’

STORES AS ORC;

49)Explain the functionality of Object-Inspector.

It helps to analyze the internal structure of row object and individual structure of columns in HIVE. It also provides a uniform way to access complex objects that can be stored in multiple formats in the memory.
Instance of Java class
A standard Java object
A lazily initialized object
The Object-Inspector tells structure of the object and also ways to access the internal fields inside the object.

50)Whenever we run hive query, new metastore_db is created. Why?

Local metastore is created when we run Hive in embedded mode. And before creating it checks whether the metastore exists or not and this metastore property is defined in the configuration file hive-site.xml. Property is“javax.jdo.option.ConnectionURL” with default value “jdbc:derby:;databaseName=metastore_db;create=true”.So to change the behavior of the location to an absolute path, so that from that location meta-store will be used.

51)How can we access the sub directories recursively?

By using below commands we can access sub directories recursively in Hive

hive> Set mapred.input.dir.recursive=true;

hive> Set hive.mapred.supports.subdirectories=true;

Hive tables can be pointed to the higher level directory and this is suitable for the directory structure which is like /data/country/state/city/

52)What are the uses of explode Hive?

Hadoop developers consider the array as their inputs and convert them into a separate table row. To convert complicate data types into desired table formats Hive is essentially using explode.

53)What is available mechanism for connecting from applications, when we run hive as a server?

1.Thrift Client: Using thrift you can call hive commands from various programming languages. Example: C++, PHP,Java, Python and Ruby.

2.JDBC Driver: JDBC Driver supports the Type 4 (pure Java) JDBC Driver

3.ODBC Driver: ODBC Driver supports the ODBC protocol.

54)How do we write our own custom SerDe?

End users want to read their own data format instead of writing, so the user wants to write a Deserializer than SerDe.
Example: The RegexDeserializer will deserialize the data using the configuration parameter ‘regex’, and a list of column names.
If our SerDe supports DDL, we probably want to implement a protocol based on DynamicSerDe. It’s non-trivial to write a “thrift DDL” parser.

55)Mention the date data type in Hive. Name the Hive data type collection.

The TIMESTAMP data type stores date in java.sql.timestamp format.

Three collection data types in Hive

1. ARRAY
2. MAP
3. STRUCT

56)Can we run UNIX shell commands from Hive? Can Hive queries be executed from script files? How? Give an example.

Yes, we can run UNIX shell commands from Hive using the! Mark before the command .For example: !pwd at hive prompt will list the current directory.
We can execute Hive queries from the script files by using the source command.