Introduction to MS-HMSHTTP - Microsoft · 2019-11-19 · SQL Server 2019 new feature: Big Data Cluster Apache Hadoop cluster Big Data Cluster •Apache Hive •Hive Metadata Database

Introduction to MS-HMSHTTP

Bin Guo

SQL OIPI Test Team

Agenda

Background & Overview

Transport Direction & Stack

Message Structure & Examples

Apache Thrift Generation Demo

Resource

1

2

3

4

5

Protocol Background and Overview

SQL Server 2019 new feature: Big Data Cluster

Apache Hadoop cluster

Big Data Cluster

• Apache Hive

• Hive Metadata Database

Hive Metastore HTTP Server

• Hive MetastoreService

Client

MS-HMSHTTPDerby

DatabaseSQL Server

MS-HMSHTTPTheHiveMetastoreHTTPprotocol

Comparison with Apache Hive IMetaStoreClient

MS-HMSHTTP

LightweightHive

Metastore

Interface

Hive IMetaStoreClient

All kinds of management

functions

Protocol Transport Direction and Layers

Knox, the Gateway in Apache Hadoop

HTTP Post to https://{clusterIp}:{knoxPort}/path

https://ClusterIP:30443/gateway/default/hmshttpthrift/api/hms

Transport Stack (Layers)

Message Structure and Examples

All MS-HMSHTTP Message Operations

• get_all_databases• get_databases• get_database

• get_all_tables• get_tables• get_table• get_tables_by_type

• get_partition_names• get_partitions

Message Data Structures

• Database• Table• Partition

• MetaException• NoSuchObjectException

struct Database {

1: string name,

2: string description,

3: string locationUri,

4: map<string, string> parameters, //

properties associated with the database

5: optional PrincipalPrivilegeSet

privileges,

6: optional string ownerName,

7: optional PrincipalType ownerType

}

Message body in TJSON array format

Position 1

• version

Position 2

• message name

Position 3:

• message type

Position 4

• message seqid

Position 5

• payload

message type

1: CALL 2: REPLY 3: EXCEPTION 4: ONEWAY

Request body Example

Position 1

• version

Position 2

• message name

Position 3:

• message type

Position 4

• message seqid

Position 5

• payload

[1,"get_all_databases",1,1,{}]

Response Body Example

Position 1

• version

Position 2

• message name

Position 3:

• message type

Position 4

• message seqid

Position 5

• payload

[1,"get_all_databases",2,1,{"0":{"lst":["str",2,"default","hmshttptestdatabase"]}}]

Hive Metastore Thrift API signature of get_all_databases()

list<string> get_all_databases() throws(1:MetaException o1)

Thrift in Apache Hive Architecture

Protocol Client and Payload Thrift Generation

MS-HMSHTTP Thrift Appendix

ApacheThrift Tool Chain

A Hive MetastoreThrift HTTP Client

• JavaScript API , PHP API, C# API, …

Thrift Schema in Protocol Appendix

演示 Demo

Protocol client can be created by using the Apache Thrift [ApacheThrift] toolchain to query the Hive Metastore Thrift API.

Demo

On Windows 10

with VS2019

According to

https://thrift.apache.org/

C#/JS HMSHTTP Thrift Client Generation Steps

▅ 1) Download Thrift compiler EXE

▅ 2) Prepare

ThriftHiveMetastore.thrift and

fb303.thrift files

▅ 3) Use VS Developer CMD:

▅ thrift --gen <language> <Thrift

filename>

▅ thrift-0.13.0.exe --gen netstd/JS

hive_metadata.thrift

[MS-HMSHTTP]: Hive Metastore HTTP Protocolhttps://docs.microsoft.com/en-us/openspecs/sql_server_protocols/ms-hmshttp/1f53d4a7-19be-41dc-aecc-f762096f2e92

Apache Hive IMetaStoreClienthttps://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/metastore/IMetaStoreClient.html

[ApacheThrift]https://thrift.apache.org/

What are SQL Server Big Data Clusters? https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-overview?view=sqlallproducts-allversions

SQL-On-Hadoop: Hive - Part Ihttps://www.sqlservercentral.com/articles/sql-on-hadoop-hive-part-i

Resource

Summery

• Protocol Background & Overview

• Transport Direction & Stack

• Message Structure

• Message Example

• Apache Thrift Generation Demo

This is a Sample QR Code, We will replace itwhen the new QR code created.