Hyperflow HUB
Templates
Documentation
API Docs
AI Reports

↩ API Docs

Data sources ingestion API

Use this API to programmatically ingest data into an existing Data Source

Asynchronous ingestion
Follow this 3-step process to upload a dataset into an existing Data source.

Step 1: Generate an ingestion URL
Step 2: Upload file using the URL from Step 1
Step 3: Initiate ingestion

Step 1: Generate an ingestion URL

Inputs
Request body (json payload)
key
required
input filename
type
required
datasource
Example of Rest API Call
curl https://{{ tenant }}.lightenup.ai/admin/s3_uploadlink \
\
--request POST \
--header "Authorization: Bearer {access_token}" \
--header "Content-Type: application/json" \
--data @- << EOF
{
  "key": "{filename}",
  "type": "datasource"
}
EOF
Responses
Response Code
200
Example of 200 response code
{
headers: {
           'Content-Type': 'application/json',
           'Access-Control-Allow-Headers': '*',
           'Access-Control-Allow-Methods': allowMethods,
          'Access-Control-Allow-Origin': GLOBALS.CORS_ORIGIN,
           'Access-Control-Allow-Credentials': true
       },
body: {"url: string"}
}
Response Code
400
Example of 400 error
{      
       headers: {
           'Content-Type': 'application/json',
           'Access-Control-Allow-Headers': '*',
           'Access-Control-Allow-Methods': allowMethods,
          'Access-Control-Allow-Origin': GLOBALS.CORS_ORIGIN,
           'Access-Control-Allow-Credentials': true
       },
       400,
      {"message":"Missing parameters"},
      Error: "CustomError",
      Cause: {"errorType": "CustomError", "errorMessage": "Missing parameters"}
}
Response Code
500
Example of 500 error
{      
       headers: {
           'Content-Type': 'application/json',
           'Access-Control-Allow-Headers': '*',
           'Access-Control-Allow-Methods': allowMethods,
          'Access-Control-Allow-Origin': GLOBALS.CORS_ORIGIN,
           'Access-Control-Allow-Credentials': true
       },
       500,
      {"message":"Internal server: error processing intake step"},
      Error: "CustomError",
      Cause: {"errorType": "CustomError", "errorMessage": "Internal server: error
processing intake step"}
}

‍

Step 2: Upload file using the URL from Step 1

Inputs
Request body (json payload)
url
required
url from Step 1
file
required
file (maximum 6MB)
Example of Rest API Call
curl {url} \
\
--request PUT \
--header "Content-Type {type}" \
--data @- << EOF
{
   "url": "https://profile-console-data-lighte...."
  file
}
Responses
Response Code
200
Example of 200 response code
// no content in body
Response Code
400
Example of 400 error
// no content in body
Response Code
500
Example of 500 error
// no content in body

‍

Step 3: Initiate ingestion

Inputs
Request body (json payload)
data_source
required
datasource name
type
required
datasource type
uploadurl
required
url from Step 1
Example of Rest API Call
curl https://{{ tenant }}.lightenup.ai/admin/data/ingest/text_async \
\
--request POST \
--header "Authorization: Bearer {access_token}" \
--header "Content-Type: application/json" \
--data @- << EOF
{
  "data_source": "catalog",
  "type": "csv",
   "uploadurl": "https://profile-console-data-lightenup.."
}
Responses
Response Code
200
Example of 200 response code
{
   "message": "Data ingestion in progress",
   "ingestion_id": "f3170179-755e-43f8-b8ea-a544256e404c"
}
Response Code
400
Example of 400 error
{      
       headers: {
           'Content-Type': 'application/json',
           'Access-Control-Allow-Headers': '*',
           'Access-Control-Allow-Methods': allowMethods,
          'Access-Control-Allow-Origin': GLOBALS.CORS_ORIGIN,
           'Access-Control-Allow-Credentials': true
       },
       400,
      {"message":"Missing parameters"},
      Error: "CustomError",
      Cause: {"errorType": "CustomError", "errorMessage": "Missing parameters"}
}
Response Code
500
Example of 500 error
{      
       headers: {
           'Content-Type': 'application/json',
           'Access-Control-Allow-Headers': '*',
           'Access-Control-Allow-Methods': allowMethods,
          'Access-Control-Allow-Origin': GLOBALS.CORS_ORIGIN,
           'Access-Control-Allow-Credentials': true
       },
       500,
      {"message":"Internal server: error processing intake step"},
      Error: "CustomError",
      Cause: {"errorType": "CustomError", "errorMessage": "Internal server: error
processing intake step"}
}

‍