PostgreSQL Connector #

Register PostgreSQL Connector #

curl -XPUT "http://localhost:9000/connector/postgresql?replace=true" -d '
{
  "name" : "PostgreSQL Connector",
  "description" : "Fetch data from PostgreSQL database.",
  "category" : "database",
  "icon" : "/assets/icons/connector/postgresql/icon.png",
  "tags" : [
    "sql",
    "storage",
    "web"
  ],
  "url" : "http://coco.rs/connectors/postgresql",
  "assets" : {
    "icons" : {
      "default" : "/assets/icons/connector/postgresql/icon.png"
    }
  }
}'

Use postgresql as a unique identifier, as it is a builtin connector.

Update coco-server’s config #

Below is an example configuration for enabling the Postgresql Connector in coco-server:

connector:
  postgresql:
    enabled: true
    queue:
      name: indexing_documents
    interval: 30s

Explanation of Config Parameters #

Use the PostgreSQL Connector #

The PostgreSQL Connector allows you to index data from your database by executing a custom SQL query.

Configure PostgreSQL Datasource #

To configure your PostgreSQL connection, you’ll need to provide the connection details and the query to fetch the data.

Connection URI: The full PostgreSQL connection string, including user, password, host, port, and database name.

SQL Query: The SQL query that will be executed to fetch the data for indexing. You can use JOINs and select specific columns.

Last Modified Field: (Optional) For incremental sync, specify a timestamp or datetime column (e.g., updated_at). The connector will only fetch rows where this field’s value is newer than the last sync time.

Enable Pagination: (Optional) A boolean (true or false) to enable paginated fetching. This is highly recommended for large tables to avoid high memory usage.

Page Size: (Optional) The number of records to fetch per page when pagination is enabled. Defaults to 500.

Field Mapping: (Optional) An advanced feature to map columns from your SQL query to specific document fields like id, title, content, and custom metadata.

Example Request #

Here is an example request to configure the Postgresql Connector:

curl -H 'Content-Type: application/json' -XPOST "http://localhost:9000/datasource/" -d '
{
    "name":"My Postgresql Documents",
    "type":"connector",
    "connector":{
        "id": "postgresql",
        "config": { 
            "connection_uri": "postgres://username:password@localhost:5432/coco?sslmode=disable&timezone=Asia/Shanghai",
            "sql": "SELECT * from DOC",
            "pagination": true,
            "page_size": 500,
            "last_modified_field": "updated",
            "field_mapping": { 
                "enabled": true,
                "mapping": { 
                    "hashed": true,
                    "id": "id",
                    "title": "title",
                    "url": "url",
                    "metadata": [ 
                      { 
                        "name": "version",
                        "value": "v"
                      }
                    ]
                }
            }
        }
    }
}'

Supported Config Parameters for PostgreSQL Connector #

Below are the configuration parameters supported by the PostgreSQL Connector: