InfiniFlux append protocol

Append Protocol(High-Speed Input Protocol)

www.infiniflux.com

Table of Contents

Background

CLI API

JDBC API

Multi Table Insert

1

2

3

4

Background

4

Background – Limitations of Conventional Protocol

Client

ServerIdle time

……

• Limitations of Conventional Synchronous Protocol (Request - Reply):Synchronous protocol is to check the results of executions each time a record was entered. Thus;

• Rapidly increase the number of I/O processing in accordance with the number of records• Must check the result of a record each time• Difficult to insert more than 1,000 data per second

Client

Client

Client

Server

Server

Idle time

Idle time

5

Background – Improved Protocol

Flush Append

Flush Append

Append Protocol Diagram

• Introduced Optimistic Asynchronous Protocol (Append Protocol)

• Pile up input records on the buffer consecutively and flush them when the buffer is full.• Not returning results if append succeeds.• Return error code and message to the client when append was failed.• Able to insert more than millions of records per second• Most of cases are successfully inserted, and return errors rarely. Thus, the improved protocol is the best alternative.

Client

Client

Client

No Idle time

No Idle time

Server

Server

Server

6

Background –API Structure

Title Contents

API for establishing communication channels

• Open a channel for appending• Specify a table where data will be inserted• Insert data at a high-speed through this channel

API for data insert• Binding column data that conform to the given table schema• Return errors when the length and format are not matched• Guarantee of fast data transfer through buffering

API for data flush• Force to send the data through network even when the buffer was not

filled.

API for setting an error callback

• Setting up asynchronous user-callback in order to process error code and message received from the server

• Asynchronously operate with data input

API for closing communication channel• Close the protocol that is currently appending• Send all the records left on the buffer through network

CLI API

8

Confirming CLI Installation

• Makefile

• Header file

• Library file

Check the files, shown below, in the “install”, “include”, and “lib” directories where the InfiniFlux has been installed.

$IFLUX_HOME/install/iflux_env.mk

$IFLUX_HOME/include/iflux_sqlcli.h

$IFLUX_HOME/lib/libifluxcli.a$IFLUX_HOME/lib/libifluxcli_dll.so

9

Makefile & Compile

• Move to “sample/cli” directory.

• Check the contents of Makefile.

include $(IFLUX_HOME)/install/iflux_env.mk

INCLUDES += $(LIBDIR_OPT)/$(IFLUX_HOME)/include

all : sample4_append2

sample4_append2 : sample4_append2.o$(LD_CC) $(LD_FLAGS) $(LD_OUT_OPT)$@ $< $(LIB_OPT)ifluxcli$(LIB_AFT) $(LIBDIR_OPT)$(IFLUX_HOME)/lib $(LD_LIBS)

sample4_append2.o : sample4_append2.c$(COMPILE.cc) $(CC_FLAGS) $(INCLUDES) $(CC_OUT_OPT)$@ $<

clean :rm -f *.o sample4_append2

• Compile it.

Check and compile “$IFLUX_HOME/sample/cli/makefile”.

$ cd $IFLUX_HOME/sample/cli

$ make

10

Structure of Append Program

SQLAllocEnv

SQLAllocConnect

SQLDriverConnect

SQLAllocStmt

SQLAppendOpen

SQLAppendDataV2

SQLAppendClose

SQLAppendSetErrorCallback

SQLFreeStmt

SQLDisconnect

SQLFreeConnect

SQLFreeEnv

Extended Functions of InfiniFlux

CLI Standard Connect Functions

CLI Standard Disconnect Functions

Description of “$IFLUX_HOME/sample/cli/sample4_append1.c“

11

SQLAppendOpen

• Open a channel for inserting data into a specified table.• The channel remains open if the channel is not closed.• Able to set up to 1024 statements for each connection.• Use SQLAppendOpen for each statement.

SQLRETURN SQLAppendOpen( SQLHSTMT aStatementHandle, SQLCHAR *aTableName, SQLINTEGER aErrorCheckCount );

• aStatementHandle : statement handle to append• aTableName : name of target table to append• aErrorCheckCount : set the number of evaluations to detect server errors based on the number of inputs.

If the number is 0, evaluation is not required.

12

SQLAppendDataV2

• Function to insert data for the channel.• Supported from the InfiniFlux v2.0.• Able to insert data types of both text and binary.

SQLRETURN SQLAppendDataV2( SQLHSTMT aStatementHandle, SQL_APPEND_PARAM *aData );

• aStatementHandle: statement handle to append.• aData : a pointer to parameter array, “SQL_APPEND_PARAM”

Defined “SQL_APPEND_PARAM” in the “iflux_sqlcli.h”

13

SQLAppendClose

• Close the current channel.• Able to check the number of append success or failure.

SQLRETURN SQLAppendClose( SQLHSTMT aStatementHandle, int *aSuccessCount, int *aFailureCount );

• aStatementHandle: statement handle to append• aSuccessCount : the number of successful append• aFailureCount : the number of failed append

14

SQLAppendSetErrorCallback

• Setting up the callback function to be called when an error was occurred while appending.• If this function was not set, the client will ignore even if an error was occurred in the server.

SQLRETURN SQLAppendSetErrorCallback( SQLHSTMT aStatementHandle, SQLAppendErrorCallback *aFunc );

• aStatementHandle: statement handle to append• aFunc : a pointer to callback function when an error occurs.

typedef void (*SQLAppendErrorCallback)( SQLHSTMT aStatementHandle,SQLINTEGER aErrorCode,SQLPOINTER aErrorMessage,SQLLEN aErrorBufLen,SQLPOINTER aRowBuf,SQLLEN aRowBufLen );

• aStatementHandle: statement handle to append• aErrorCode : 32 bit error code which causes the error• aErrorMessage : the string for the error code• aErrorBufLen : the length of “aErrorMessage”• aRowBuf : the string that has the content of the record that caused the error• aRowBufLen : the length of “aRowBuf”

15

Other Functions

• Additionally, provide useful functions.• Refer to website at http://www.infiniflux.com/document for more information.

Name of Functions Function Declaration Description

SQLAppendDataByTimeV2

SQLRETURN SQL_API SQLAppendDataByTimeV2(SQLHSTMT aStatmentHandle,SQLBIGINT aTime, SQL_APPEND_PARAM *aData)

Like “SQLAppendDataV2()”, it inserts data into a table. However, the value of hidden column, “_arrival_time”, will be used when it was set to a value of a certain time rather than the present time. In other words, send the time of log files as the value of “aTime”.

SQLAppendFlushSQLRETURN SQL_APISQLAppendFlush(SQLHSTMT aStatementHandle)

Transfer all the data left in the buffer of the current channel to the InfiniFlux server.

SQLSetConnectAppendFlush

SQLRETURN SQL_API SQLSetConnectAppendFlush(SQLHDBC aHdbc, SQLINTEGER aTimeOption)

Set whether to transfer data to InfiniFlux at a specified interval.0: auto flush off1: auto flush on

SQLSetStmtAppendInterval

SQLRETURN SQL_API SQLSetStmtAppendInterval(SQLHSTMT aStatementHandle, SQLINTEGER aValue)

When set “auto flush” on by using “SQLSetConnectAppendFlush()”, adjust the interval of flush orset to “auto flush” off.Don’t flush it if “aValue” is 0.The default value is 1000 and unit is “ms”.The value should be multiples of 100.

JDBC API

17

Confirm the Installation of Library

• Library file

• Log4j file

Support Java 1.6 version or more

• Sample

• Makefile

Check the file, shown below, in the “lib” directory where the InfiniFlux has been installed.

$IFLUX_HOME/lib/iflux.jar

$IFLUX_HOME/lib/ifluxLog.jar$IFLUX_HOME/lib/log4j.jar

$IFLUX_HOME/sample/jdbc

$IFLUX_HOME/sample/jdbc/Makefile

18

Append Program Structure

DriverManager.getConnection

Connection.createStatement

executeAppendOpen

executeAppendData

executeAppendClose

executeSetAppendErrorCallback

ResultSet.close

Statement.close

Connection.close

Extended Functions of InfiniFlux

Standard Connect Functions

Standard Disconnect Functions

Description of “$IFLUX_HOME/sample/jdbc/sample4_append.java”

19

executeAppendOpen

• Open a channel for inserting data into a specified table.• The channel remains open if the channel is not closed.• Able to set up to 1024 statements for each connection.

ResultSet IfluxStatement.executeAppendOpen( String aTableName, int aErrorCheckCount );

• aTableName : name of target table to append• aErrorCheckCount: set the number of evaluations to detect server errors based on the number of inputs. If the number is 0,

evaluation is not required.

20

executeAppendData

• Function to insert data for the channel.

int IfluxStatement.executeAppendData( ResultSetMetaData aRsmd, ArrayList aData );

• aRsmd: information of column metadata about the table• aData: array list of data to append

21

executeAppendClose

• Close the current channel.

Run “int IfluxStatement.executeAppendClose();”,and then call “getAppendSuccessCount()” and “getAppendFailureCount()” to check the number of append success or failure.

22

executeSetAppendErrorCallback

• Setting up the callback function to be called when an error was occurred while appending.• If this function was not configured, the client will ignore even if an error was occurred in the server.

int IfluxStatement.executeSetAppendErrorCallback( IfluxAppendCallback aCallback);

• aCallback : to call “callback” function when an error while appending data

IfluxAppendCallback cb = new IfluxAppendCallback() {@Overridepublic void onAppendError(long aErrNo, String aErrMsg, String aRowMsg) {

System.out.format("Append Error : [%05d - %s]\n%s\n", aErrNo, aErrMsg, aRowMsg);}

};

stmt.executeSetAppendErrorCallback(cb);

23

Other Functions

• Additionally provide useful functions.• Refer to the website at http://www.infiniflux.com/document for more information.

Name of Functions Function Declaration Description

executeAppendDataByTime

int executeAppendDataByTime(ResultSetMetaData aRsmd,long aTime,ArrayList aData)

Like “executeAppendData()”, it inserts a data into the channel. However, the value of hidden column, “_arrival_time”, will be used when it was set to a value of a certain time rather than the present time. In other words, send the time of log files as the value of “aTime”.

executeAppendFlush int executeAppendFlush()Transfer all the data left in the buffer of the current channel to the InfiniFlux server.

executeAppendSuccessCount long getAppendSuccessCount() Return the number of records that were appended successfully.

executeAppendFailureCountlong executeAppendFailureCount()

Return the number of records that were failed to append.

Multi Table Insert

25

Multi Table Insert Structure

Description of “$IFLUX_HOME/sample/cli/sample8_multi_session_multi_table.c”

• Create thread in the main function, and each thread will process the tasks written below.

• Each thread will process Alloc and Free functions respectively against SQLHENV, SQLHDBC, and SQLHSTMT.

• Only one statement is allowed to be inserted per table.

• The reason for tasks above is that an error occurs while a thread is sharing statement or connection with

another thread, it might kill the whole process. If possible, it is safe not to share a resource of thread with

others.

• When input cycle is irregular and has long interval, conduct “auto timed flush” by using

“SQLSetConnectionAppendFlush()” and “SQLSetStmtAppendInterval()”.

The World's Fastest Time Series DBMS

for IoT and Big Data

[email protected]

InfiniFlux