Imports via SFTP

You can upload your user data to Clevertap and then use this data to analyze, engage, and retain users. You can then use the power of our analytics, segmentation, experiences to enhance engagement and retention.
You can upload profile data such as name, email address, and event data such as product purchased and the purchase amount. Once you have established a connection with our servers, you can upload data automatically at regular intervals.

Pre-requisites

You must have an SFTP client to be able to import data into CleverTap's SFTP server.

Step 1 - Start Import

Go to Settings> Partner Data> Import and click the SFTP button.

Import Data

Import Data

Step 2 - Add Credentials

Our SFTP server needs to recognize your SFTP client to start receiving files. To establish this secured file transfer connection, an SSH key pair needs to be generated & the public key file should be shared with Clevertap.

Once our SFTP server receives the SSH public key, a user is created for you. After authentication, you can seamlessly start transferring data-files. This will be a passwordless connection. The user-name & server URL is available in the Saved Keys section.

Add new credentials by clicking the +Credentials button from the Credentials tab. The Add new credentials window appears.

Enter your credentials in the following boxes and click Save:

  • Username - Your account ID is your username. It will be populated automatically.
  • Key name - It is the name of your key. It can be any name to identify your key.
  • SSH key - It is your machine's SSH public key. This is the key of the machine that uploads data to Clevertap.

Your credentials will be verified and saved under the Credentials tab.

Limit

You can add up to 10 credentials.

Refer to the server URL displayed in the notes of the Credentials tab for connecting to the SFTP URL.

Step 3 - Upload CSV files

You can upload Profiles or Events data to CleverTap with an SFTP client. Each upload must be followed with a manifest file to map your data upload with CleverTap properties.

File uploads

All files (Events and Profiles) must be uploaded in the CSV format. The manifest file is a JSON file that provides metadata for the uploaded CSV data file. The processing of the data file will start only after the manifest file is uploaded. Every manifest file must have a unique name.

Events CSV file

The Events CSV file contains all the event details that you want to send to CleverTap.

The Events file must have the following mandatory fields:

Column Header
Description

ts

Timestamp of the event

ObjectId/Identity/guid/fbid

Identity of the user

Following is an example for an Events CSV file:

event_name
time_stamp
identity
cookie_id
item_name
item_price
item_delivery_date

Purchased

1578503667

cookie_123

Awesome Ring - 10

123

$D_1589274432

Purchased

1578503667

cookie_123

Awesome Ring - 11

123

$D_1589274432

Purchased

1578503667

cookie_123

Awesome Ring - 12

123

$D_1589274432

Purchased

1578503667

cookie_123

Awesome Ring - 13

123

$D_1589274432

Data file timestamp

The timestamp column(ts) of the data file must have an epoch format.
If there is any other column that has date-time values, mention these values in the $D_epoch format.

Profiles CSV file

The Profiles CSV file contains all the user profile details that you want to send to CleverTap.

The Profiles file must have the following mandatory fields:

Column Header
Description

ObjectId/Identity/guid

Identity of the user

ts

Timestamp of the upload

Following is an example for a Profiles CSV file:

time_stamp
identity
cookie_id
id_name
id_email
id_phone
id_phone.operation
is_married
age
Playlist
Playlist.operation

1574668407

dummy_cookie-001

JohnDoe-001

+918876543211

Y

21

Rock

$add

1574668408

dummy_cookie-002

JaneDoe-002

+918876543212

N

22

Pop

$remove

1574668409

dummy_cookie-003

JohnRoe-003

+918876543213

$set

Y

25

1574668410

dummy_cookie-004

JaneRoe-004

$delete

N

28

Operations for profile properties

You can use the following operations on profile properties:

  • $add - This operation will add value to the property array. For example, you can add rock songs as Playlist values.
  • $remove- This operation will remove the value from the property array. For example, you can remove Pop from the existing Playlist.
  • $set - Use this operation to replace the current value. In our example, the phone number will be set as "+918876543213."
  • $delete - This operation removes the property. For example, you can completely remove the property id_phone for Jane Roe and all the values in it.

Note

To add or remove multiple values in a property, create a separate row for each value.

Step 4 - Upload Manifest File

The next step after uploading the CSV file is uploading the manifest file. The manifest file helps us to recognize your data.

The manifest file is a JSON file that will map the custom attributes in your CSV files with CleverTap properties. The format for the manifest file is anyfilename.manifest. The following keys are mandatory in a manifest file:

Expected keys
Description

type

The event/profile/(any other type of data that we may support in the future).

fileName

The name of the data file.

clientEmail

This is a mandatory field. We send the update mails to this email ID.

columns

This is an optional field.

  • ctName - This is an optional field. The actual property name in CleverTap.
  • dataType - This is an optional field. The supported dataTypes are STRING, INTEGER, FLOAT, and LONG. If a datType is not defined, then it is considered as STRING.

Manifest file for Events

Create a manifest file to map your custom events to CleverTap Events.
Following is a sample manifest file for Events:

{
	"fileName": "events.csv",
	"type": "event",
	"columns": {
		"event_name": {
			"ctName": "evtName",
        		"dataType": "STRING",
     
		},
		"time_stamp": {
			"ctName": "ts",
      			"dataType": "INTEGER"
		},
		"identity": {
			"ctName": "identity",
      			"dataType": "STRING"
		},
		"cookie_id": {
			"ctName": "objectId",
      			"dataType": "STRING"
		},
		"item_name": {
			"ctName": "Name",
      			"dataType": "STRING"
		},
		"item_price": {
			"ctName": "Price",
      			"dataType": "FLOAT"
		},
		"item_delivery_date": {
			"ctName": "DeliveryDate",
      			"dataType": "STRING"
		}
	},
	"clientEmail": "[email protected]"
  
  
}

Manifest file for Profiles

Create a manifest file to map the details of your user profiles to CleverTap Profiles.

Following is a sample manifest file for Profiles:

{
	"fileName": "profiles.csv",
	"type": "profile",
	"columns": {
		"time_stamp": {
			"ctName": "ts",
      			"dataType": "INTEGER"   
		},
		"identity": {
			"ctName": "identity",
       			"dataType": "STRING"
		},
		"cookie_id": {
			"ctName": "objectId",
       			"dataType": "STRING"
		},
		"id_name": {
			"ctName": "Name",
       			"dataType": "STRING"
		},
		"id_email": {
			"ctName": "Email",
       			"dataType": "STRING"
		},
		"id_phone" : {
			"ctName": "Phone",
       			"dataType": "STRING"
		},
		"is_married": {
			"ctName": "Married",
       			"dataType": "STRING"
		},
		"age": {
			"ctName": "Age",
       			" dataType": "INTEGER"
		}
	},
	"clientEmail": "[email protected]"
}

Manifest note

If a column is not mapped to a CleverTap property name, the data is processed with the column name as the property name.

Step 5 - Upload Data

You are now ready to upload data files to our servers.

  1. Upload the data file first through SFTP.
  2. Upload the manifest file to mark the end of the data upload and begin the processing of data.

Use any SFTP client and enter the following commands to upload your files:

  1. sftp -i ~/.ssh/id_rsa [email protected]
  2. put filename
  3. put file.manifest

We will send you a success email after the upload and processing is complete.

You can now start using this data in our engagement channels.

If the import fails for some reason, we will send you a failure email with the attached CSV file that contains the failed records. You can download this CSV file and check it for errors.

Following is an example of an error file. The last column in this file displays all the errors.

time_stamp
identity
cookie_id
id_name
id_email
id_phone
is_married
age
Failure Description

1574668407

dummy_cookie-001

JohnDoe-001

+918876543211

Y

21

Phone number is not valid

1574668408

dummy_cookie-002

JaneDoe-003

+918876543212

N

22

Phone number is not valid

Troubleshoot errors

There may be instances where you may see some errors during file upload. Following is an explanation of the errors and the steps to resolve them:

Error name
Resolution

No data file found

The data file is not uploaded. Upload the data file again and create a new manifest file for it.

File couldn't be read

The file is not in the expected format. Upload the file again with the correct format, that is, CSV for data files and manifest file for mapping.

Internal infra error

Upload the file again followed by a new manifest file.

Incorrect details in manifest file

This error file is emailed to you. Check the manifest file for incorrect details.

Timestamp is mandatory and must be in epoch format

Check the format for timestamp. So a date such as "Thursday, March 5, 2020 9:55:27 AM" should be represented as "1583402127".

Atleast one of identity / fbId / gpId / objectId is mandatory under 1024 chars to identify the user

Check that a minimum of one ID is present.

[Internal error] Error while importing data

This is an internal error. Try importing the data again.

Event name is mandatory

The event name is blank or the column name is not available.

Event name belongs to CleverTap restricted system events

Do not use the event names of the existing system events.

Cannot have more than 100 event properties

A maximum of 100 event properties is allowed. Check that the event properties do not exceed it.

Profile operation must be one of $add, $set, $remove, $delete

Add a valid profile operation.

The status for each file is sent via email.

View imports

After a successful import, you can view your imported files on the CleverTap dashboard.
Click Settings > Import > Imports tab to check for your uploaded files. The Imports tab displays your uploaded files and their status.

FAQs

Q. What is the maximum file size for upload?
A. You can upload a maximum of 5 GB for each data file.

Q. How much time will it take to complete the file processing?
A. It will take up to 2 hours to complete the processing for a maximum upload of a 5 GB data file.

Q. Is segmentation possible on the uploaded data?
A. Yes.

Q. Is engagement possible from the uploaded data?
A. Yes. You can use our engagement channels to engage with your users.

Q. Do you support live campaigns on the uploaded data?
A. Yes. However, the support is available only for Push, Email, and SMS engagement channels.

Q. Can we do uploads at regular intervals?
A. Yes. You can write a script to connect to CleverTap SFTP and transfer data every day when it is ready.

Q. Can we upload any event or profile file without the manifest file?
A. No. Every data file is recognized by a unique manifest file. Every manifest file should have a unique name.

Q. Can we upload any event or profile file without the column headers in the CSV file?
A. Column headers are mandatory for a file. Without the column headers, the file will go into an error state.

Q. What are the mandatory columns in the event and profile data files?
A. The column headers mandatory are ObjectId & ts.

Updated 4 months ago


Imports via SFTP


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.