menu
Is this helpful?

# TaJsonFileWriterプラグイン

# TaJsonFileWriter Plug-in

# Introduction

TaJsonWriter provides files that write TA-JSON format to local files. TaJsonWriter serves users who need to restore TA cluster data to json text.

# Functions and Limitations

TaJsonWriter realizes the function of local files from DataX protocol to TA-json format. TaJsonWriter has the following functions:

  1. Support and only support writing to json text files in TA format.
  2. Support multi-threaded writing, with each thread writing different subfiles.

# Function Description

# 3.1 Sample configuration

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": 123123,
                "type": "long"
              },
              {
                "value": "123123",
                "type": "string"
              },
              {
                "value": "login",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2222",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "test",
                "type": "bytes"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 100
          }
        },
        "writer": {
          "name": "ta-json-writer",
          "parameter": {
            "type": "event",
            "path": "/data/export/ta_datafile/",
            "filename": "test",
            "column": [
              {
                "index": "0",
                "colTargetName": "#user_id"
              },
              {
                "index": "1",
                "colTargetName": "#distinct_id"
              },
              {
                "index": "2",
                "colTargetName": "#event_name"
              },
              {
                "index": "3",
                "colTargetName": "#time",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "4",
                "colTargetName": "#event_time",
                "type": "string"
              },
              {
                "index": "5",
                "colTargetName": "#account_id",
                "type": "string"
              },
              {
                "index": "6",
                "colTargetName": "timetest",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "6",
                "colTargetName": "timetest2",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss"
              },
              {
                "index": "7",
                "colTargetName": "os_1",
                "type": "string"
              },
              {
                "index": "7",
                "colTargetName": "os_2",
                "type": "string"
              },
              {
                "index": "8",
                "colTargetName": "booleantest",
                "type": "boolean"
              },
              {
                "index": "0",
                "colTargetName": "testNumber",
                "type": "number"
              },
              {
                "colTargetName": "add_clo",
                "value": "123123",
                "type": "string"
              }
            ]
          }
        }
      }
    ]
  }
}

# 3.2 Parameter description

  • path
    • Description: path information of the local file system. TaJsonWriter will write multiple files under the Path directory.
    • Required: Yes
    • Default value: none

fileName

  • Description: file name written by TaJsonWriter, which will add a random suffix as the actual file name written by each thread.
  • Required: Yes
  • Default value: none
  • writeMode
    • Description: TaJsonWriter data cleaning processing mode before writing:
      • truncate, clear all files with fileName prefix under the directory before writing. -append, do not do any processing before writing, DataX TAJsonWriter directly uses filename to write, and guarantees that the filename does not conflict.-nonConflict, if there are files with fileName prefix under the directory, report an error directly.
        • Required: Yes
    • Default value: append
  • encoding
    • Description: read the encoding configuration of the file.
    • Required: No
    • Default value: utf-8
  • column
    • Description: read the list of fields. type specifies the type of data, index specifies the current column corresponding to reader (starting with 0). value specifies the current type as a constant, does not read data from reader, but automatically generates the corresponding column according to the value.

The user can specify the Column field information, configured as follows:

[
  {
    "type": "Number",
    "colTargetName": "test_col", //generate the column names corresponding to the data
    "index": 0 //transfer the first column from reader to dataX to get the Number field
  },
  {
    "type": "string",
    "value": "testvalue",
    "colTargetName": "test_col" //generate the string field of testvalue from TaDataWriter as the current field
  },
  {
    "index": 0,
    "type": "date",
    "colTargetName": "testDate",
    "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
  }
]

# 3.3 Type conversion

The type is defined as TaJsonFileWriter:

DataX internal type TaJsonWriter data type
Int
Number
Long
Number
Double
Number
String
String
Boolean
Boolean
Date
Date