TaCustomTableWriter Plug-ins

I. Quick introduction

TACustomTableWriter provides the ability to create custom tables to the Ta system.

In some cases, the data you need to use may not be represented in the form of user or event, such as some mapping tables, or some external data. If you need to use this part of data, you need to use the TACustomTableWriter plug-in Import custom data into the TA system.

II. Functions and Limitations

Only the custom table function of the TA system is supported.

III. Function Description

3.1 Sample Configuration

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": 123456,
                "type": "long"
              },
              {
                "value": "123123",
                "type": "string"
              },
              {
                "value": "pay",
                "type": "string"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "2222",
                "type": "string"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "test",
                "type": "string"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 10
          }
        },
        "writer": {
          "name": "ta-custom-table-writer",
          "parameter": {
            "appid": "6f9e64da5bc74792b9e9c1db4e3e3822",
            "table": "ta_custom_table",
            "tableDesc": "test table",
            "partitionValue": "2020-01-01",
            "writeMode": "overwrite",
            "column": [
              {
                "type": "double",
                "name": "id"
              },
              {
                "type": "string",
                "name": "distinct_id"
              },
              {
                "type": "string",
                "name": "event_name"
              },
              {
                "type": "string",
                "name": "time"
              },
              {
                "name": "event_time",
                "type": "string"
              },
              {
                "type": "string",
                "name": "account_id"
              },
              {
                "type": "string",
                "name": "testDate"
              },
              {
                "type": "string",
                "name": "timetest2"
              },
              {
                "type": "string",
                "name": "os_1"
              }
            ]
          }
        }
      }
    ]
  }
}

3.2 Parameter Description

• appid
  • Description: The item appid corresponding to the import table can be found in the background of the TA system. It must exist in the system and there is no offline item.
  • Type: string
  • Must choose: Yes
  • Default: None
• table
  • Description: Import the table name into the TA system. Note: The table name cannot be repeated globally. It is recommended to add distinguishable prefixes or suffixes based on different items. The table name can only be a combination of numbers, letters, and underscores
  • Type: string
  • Must choose: Yes
  • Default: None
• tableDesc
  • Description: Import the comment of the table. It is recommended to configure this parameter when importing to facilitate the subsequent query of the table to clarify the meaning of the table
  • Type: string
  • Required: No
  • Default: empty
• partitionValue
  • Description: The imported partition value, the custom table imported by the TA system will bring the partition field $pt by default, so the imported partition value must be specified when importing. Generally, it can be set as the imported data date, for example: 20180701, and it also supports time macro replacement, For example: @[{yyyyMMdd} - {1day}], the specific usage method will be introduced in XXXX section
  • Type: string
  • Required: Yes
  • Default: None
• writeMode
  • Description: TACustomTableWriter data cleaning processing mode before writing:
    • Append, like a **partitionValue **partition, writes data accumulatively.
    • Overwrite, delete the existing file under the **partitionValue **partition, and then write the data imported this time.
    • Required: Yes
    • Default: None
• column
  • Description: Field to write data to
  • Required: Yes
  • Default: None
    • Where: name specifies the field name, type specifies the field type.

The user can specify the Column field information, configured as follows:

"column":
         [
             {
                "name": "userName",
                "type": "string"
             },
             {
                "name": "age",
                "type": "long"
             }
         ]

3.3 Type Conversion

Currently TACustomTableWriter supports the HIVE type, please pay attention to check your type.

The following is a list of TACustomTableWriter conversions for HIVE data types:

DataX internal type	HIVE data type
Long	TINYINT,SMALLINT,INT,BIGINT
Double	FLOAT,DOUBLE
String	STRING,VARCHAR,CHAR
Boolean	BOOLEAN
Date	DATE,TIMESTAMP