Supported Data Formats
DataWeave can read and write many types of data formats, such as JSON, XML, and many others.
DataWeave supports the following formats as input and output:
| MIME Type | ID | Supported Formats |
|---|---|---|
|
|
|
|
|
|
|
|
DataWeave Format (dw) for testing a DataWeave expression |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
DataWeave Readers
DataWeave can read input data as a whole in-memory, in indexed fashion, and for some data formats, part-by-part by streaming the input. When attempting to read a large file, it is possible to run out of memory or to impact performance negatively. Streaming can improve performance but impacts access to file.
-
Indexed and In-Memory:
Allow for random access to data because both strategies parse the entire document. When using these strategies, a DataWeave script can access any part of the resulting value at any time.-
Indexed:
Uses indexes over the disk.DataWeave can only read the following formats in the indexed mode:
CSV, JSON, XML (starting in Mule 4.3.0)
Note that DataWeave does not support reading Excel (XLSX) input in the indexed mode.
-
In-Memory:
Parses the entire document in memory.DataWeave can read all supported formats using the in-memory mode.
-
-
Streaming:
Allows for sequential access to the file. This strategy partitions the input document into smaller items and accesses its data sequentially, storing the current item in memory. A DataWeave selector can access the portion of the file that is getting read.DataWeave supports streaming only for the following formats:
CSV, JSON, Excel (XLSX, starting in Mule 4.2.2), XML (starting in Mule 4.3.0)
For details, see Streaming in DataWeave.
Using Reader and Writer Properties
In some cases, it is necessary to modify or specify aspects of the format
through format-specific properties. For example, you can specify CSV input and
output properties, such as the separator (or delimiter) to use in the CSV file.
For Cobol copybook, you need to specify the path to a schema file using the
schemaPath property.
You can append reader properties to the MIME type (outputMimeType) attribute
for certain components in your Mule app. Listeners and Read operations accept
these settings. For example, this On New File listener example identifies the , separator for a CSV input file:
<file:listener doc:name="On New File" config-ref="File_Config" outputMimeType='application/csv; separator=","'>
<scheduling-strategy >
<fixed-frequency frequency="45" timeUnit="SECONDS"/>
</scheduling-strategy>
<file:matcher filenamePattern="comma_separated.csv" />
</file:listener>Note that the outputMimeType setting above helps the CSV reader interpret
the format and delimiter of the input comma_separated.csv file, not the writer.
To specify the output format, you can provide the MIME type and any writer
properties for the writer, such as the CSV or JSON writer used by a File Write
operation. For example, you might need to write a pipe (|) delimiter in your
CSV output payload, instead of some other delimiter used in the input. To do
this, you append the property and its value to the output directive of a
DataWeave expression. For example, this Write operation specifies the pipe as
a separator:
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
<file:content ><![CDATA[#[output application/csv separator="|" --- payload]]]></file:content>
</file:write>The following example shows a DataWeave script in which the output directive specifies two writer properties, one for the CSV field separator, another to indicate that there is no header.
%dw 2.0
output application/csv separator=";", header=false
---
payloadWriter properties depend on the format. The following example uses the JSON writer property, skipNullOn:
%dw 2.0
output application/json skipNullOn="everywhere"
---
payloadThe following example shows a DataWeave script that uses an invisible ASCII character as a separator for the CSV output. The separator property value is \u001E. \u is the escape sequence representation and 001E is the hexadecimal representation of the record separator (RS) ASCII character.
%dw 2.0
output application/csv separator = "\u001E"
---
[
{
"Name" : "Tom",
"UID" : 2569,
"ShipOrderID" : "ui-288188"
},
{
"Name" : "Alan",
"UID" : 7756,
"ShipOrderID" : "xj-232142"
},
{
"Name" : "Dan",
"UID" : 7821,
"ShipOrderID" : "uk-259459"
}
]The CSV example produces the following output:
Name UID ShipOrderID
Tom 2569 ui-288188
Alan 7756 xj-232142
Dan 7821 uk-259459Scripts that use the read or readUrl methods can also specify reader properties through the readerProperties parameter. See examples in read. However, note that use of reader properties is more common in message sources, such as the HTTP listener, as shown in the CSV reader example.
It is also possible to use a Mule variable as a value of a configuration property. For more complete examples, see Set Reader and Writer Configuration Properties.
Setting MIME Types
You can specify the MIME type for the input and output data that flows through a Mule app.
For DataWeave transformations, you can specify the MIME type for the output data.
For example, you might set the output header directive of an expression in the
Transform Message component or a Write operation to output application/json or
output application/csv.
This example sets the MIME type through a File Write operation to ensure that a format-specific writer, the CSV writer, outputs the payload in CSV format:
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
<file:content ><![CDATA[#[output application/csv --- payload]]]></file:content>
</file:write>Starting in Mule 4.3.0, you can set the output directive using the format ID alone,
instead of using the MIME type. For example, you might set the output header directive
of an expression in the Transform Message component or a Write operation to
output json or output csv. You can also use the format ID to
differentiate the format of the output data from the MIME type in the output header, and
you can use a custom MIME type.
For example, you can write JSON data but customize the MIME type to
application/problem+json by using the DataWeave directive output application/problem+json with json.
See Change a Script’s MIME Type Output
for examples that customize the MIME type output of a script.
The with keyword separates the output MIME type from the DataWeave writer for the script’s output. For example, you can specify 'output applicatio:page-aliases:
For input data, format-specific readers for Mule sources (such as the
On New File listener), Mule operations (such as Read and HTTP Request
operations), and DataWeave expressions attempt to infer the MIME type
from metadata that is associated with input payloads, attributes, and
variables in the Mule event. When the MIME type cannot be inferred from
the metadata (and when that metadata is not static), Mule sources and
operations allow you to specify the MIME type for the reader. For example,
you might set the MIME type for the On New File listener to
outputMimeType='application/csv' for CSV file input. This setting provides
information about the file format to the CSV reader.
<file:listener doc:name="On New File"
config-ref="File_Config"
outputMimeType='application/csv'>
</file:listener>Note that reader settings are not used to perform a transformation from one format to another. They simply help the reader interpret the format of the input.
You can also set special reader and writer properties for use by the format-specific reader or writer of a source, operation, or component. See Using Reader and Writer Properties.