TaJsonFileWriter Plug-ins

I. Introduction

TaJsonWriter provides files that write TA-JSON format to local files. TaJsonWriter serves users who need to restore TA cluster data to json text.

II. Functions and Limitations

TaJsonWriter realizes the function of local files from DataX protocol to TA-json format. TaJsonWriter is agreed in the following aspects:

1. Supports and only supports writing to json text files in TA format.
2. Supports multi-threaded writing, with each thread writing different subfiles.

III. Function Description

3.1 Sample Configuration

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": 123123,
                "type": "long"
              },
              {
                "value": "123123",
                "type": "string"
              },
              {
                "value": "login",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2222",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "test",
                "type": "bytes"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 100
          }
        },
        "writer": {
          "name": "ta-json-writer",
          "parameter": {
            "type": "event",
            "path": "/data/export/ta_datafile/",
            "filename": "test",
            "column": [
              {
                "index": "0",
                "colTargetName": "#user_id"
              },
              {
                "index": "1",
                "colTargetName": "#distinct_id"
              },
              {
                "index": "2",
                "colTargetName": "#event_name"
              },
              {
                "index": "3",
                "colTargetName": "#time",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "4",
                "colTargetName": "#event_time",
                "type": "string"
              },
              {
                "index": "5",
                "colTargetName": "#account_id",
                "type": "string"
              },
              {
                "index": "6",
                "colTargetName": "timetest",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "6",
                "colTargetName": "timetest2",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss"
              },
              {
                "index": "7",
                "colTargetName": "os_1",
                "type": "string"
              },
              {
                "index": "7",
                "colTargetName": "os_2",
                "type": "string"
              },
              {
                "index": "8",
                "colTargetName": "booleantest",
                "type": "boolean"
              },
              {
                "index": "0",
                "colTargetName": "testNumber",
                "type": "number"
              },
              {
                "colTargetName": "add_clo",
                "value": "123123",
                "type": "string"
              }
            ]
          }
        }
      }
    ]
  }
}

3.2 Parameter Description

• path
  • Description: The path information of the local file system, TaJsonWriter will write multiple files under the Path directory.
  • Required: Yes
  • Default: None

fileName

• Description: The file name written by TaJsonWriter, which will add a random suffix as the actual file name written by each thread.
• Required: Yes
• Default: None
• writeMode
  • Description: TaJsonWriter data cleaning processing mode before writing:
    • Truncate, clean up all files with fileName prefix under the directory before writing. -append, do not do any processing before writing, DataX TAJsonWriter directly uses filename to write, and guarantees that the filename does not conflict. -NonConflict, if there are files with fileName prefix under the directory, report an error directly.
      • Required: Yes
  • Default: append
• encoding
  • Description: Read the encoding configuration of the file.
  • Required: No
  • Default: utf-8
• column
  • Description: Read the list of fields, type specifies the type of data, index specifies the current column corresponding to the reader (starting with 0), value specifies the current type as a constant, does not read data from the reader, but automatically generates the corresponding value according to the value The column.

The user can specify the Column field information, configured as follows:

[
  {
    "type": "Number",
    "colTargetName": "test_col", //Generate column names corresponding to data
    "index": 0 //Transfer the first column from reader to dataX to get the Number field
  },
  {
    "type": "string",
    "value": "testvalue",
    "colTargetName": "test_col" //Generate the string field of testvalue from within TaDataWriter as the current field
  },
  {
    "index": 0,
    "type": "date",
    "colTargetName": "testDate",
    "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
  }
]

3.3 Type Conversion

The type is defined as TaJsonFileWriter:

DataX internal type	TaJsonWriter data type
Int	Number
Long	Number
Double	Number
String	String
Boolean	Boolean
Date	Date