Understanding BigQuery Connector for SAP’s Compression Feature
About BigQuery SAP Connector
BigQuery Connector for SAP installs into SAP Landscape Transformation Replication Server (SAP LT Replication Server) and enables near real time replication of SAP data directly into BigQuery.
With your data in BigQuery, you can perform real-time analytics with artificial intelligence or machine learning on live SAP application data. In BigQuery, you can also integrate your SAP data with data from other sources.
BigQuery Connector for SAP uses the standard interfaces and capabilities of SAP Landscape Transformation Replication Server and supports all of the data sources that SAP LT Replication Server supports.
BigQuery Connector for SAP is provided by Google Cloud in the form of transport files that you install into SAP LT Replication Server.
BigQuery Connector for SAP is a Google Cloud implementation of an SAP Business Add In (BAdI) for SAP LT Replication Server. BigQuery Connector for SAP is developed by Google Cloud by following SAP LT Replication Server SDK process.
BigQuery Connector for SAP leverages the change data capture functionality of SAP LT Replication Server.
In this specific blog, let’s take a look at the “Compression Feature” of this connector and understand how it impacts the behaviour of the data that is transferred from SLT to BigQuery.
Compression Feature:
The BigQuery Connector for SAP provides compression through a configuration that can be maintained at the table settings level. This compression feature is available as a setting that can be turned on or off and is called “Send Uncompressed.”
You can access this setting by going to the transaction /GOOG/SLT_SETTINGS select the option “Tables” in the drop down for Settings Table. Provide the Mass Transfer Key and a Table Name.
Here I am providing inputs for the Mass Transfer Key and Table Name that will be used for this demo.
Table setting is maintained as shown below:
Full name is displayed when the column is expanded, which is also shown when the mouse is hovered on the column name as well.
By default this setting is disabled and this is the recommended option.
If you want to enable this setting, then please select the check box at the table level settings as shown in the screenshot above.
The table below explains the behaviour of the connector when this option is enabled v/s disabled
The important thing to note here is this setting has no additional effect when the source data fields have values, it only affects the behaviour when the source data fields are empty (i.e. initial).
Now let’s see this behaviour using the table /GOOG/TEST_REPL which has a field that covers the range of data types supported by the Connector.
This sample table is also included in the BigQuery Connector for SAP v2.5
For this demo, this table contains two records:
- Row #1 contains values for all the fields
- Row #2 contains values for only key fields and all the values are empty
We will replicate this table once without enabling the “Send Uncompressed Flag” and then by enabling the “Send Uncompressed Flag”
Test data in /GOOG/TEST_REPL table
Contents of Row #1
Contents of Row #2
Scenario 1: Compression On
Table Settings
Data replicated in BigQuery Table
JSON Extract from BigQuery for comparison between Row #1 and Row #2
Scenario 2: Compression Off
Table Settings
Data replicated in BigQuery Table
JSON Extract from BigQuery for comparison between Row #1 and Row #2
The following table shows the initial values stored internally in SAP Database Tables based on their data type. This data will also be replicated to BigQuery when the respective fields are empty and compression is disabled.
Comparison of empty rows for the compression on v/s off scenario
When Compression is ON for the row with initial values (i.e. no data) this is how the JSON payload for the corresponding row would look like:
All columns with initial values (i.e no data) are omitted in the JSON, making the payload size smaller.
Conclusion
From the above screenshots covered in scenario 1 and scenario 2 the impact of this setting has been demonstrated, you can now accordingly make use of this setting based on your requirements.
However, please note that when the “Send Uncompressed Flag” is not enabled (i.e. Compression = ON ) the replication performance is improved as additional initial values are not sent to the BQ and the HTTP payload size is comparatively smaller.
Further Reading
You can know more about BigQuery Connector for SAP by visiting the following link