Overview
Datasources in elvex are files, database tables or website pages that can be connected to the platform and connected with agents. Datasources are helpful so the AI model powering your agent can reference the contents of the document in its responses.
Creating a new Datasource
Click "Datasources" in the left navigation panel and then click the button to create a new Datasource.
Name: The Datasource name will help others find this resource to use for their own agents. The name will also help connected agents find and use the right Datasource for each requested task, so a descriptive name can improve performance.
Tag: Tags make it easier for you and others to find and share a Datasource. Company Admins are able to create and manage the list of selectable tags in their company settings. Datasource creators and editors are able to assign tags to a Datasource from the available list.
Description: It is recommended to include a detailed description of your Datasource to help you and others understand what's contained within. The description will also help connected agents find and use the right Datasource for each requested task, so a descriptive name can improve performance.
Datasource Type: Upload static files from your computer or create dynamic file Datasources with File-based (like Google Drive, Dropbox, or Microsoft OneDrive). Connect directly to schemas and tables with a Database Connection. Use the content of a public website with Website.
Setting Permissions
When creating a new Datasource, you can set the visibility to Private or Public. Setting a Datasource's visibility to Private means only you, company Owners and users you explicitly share the Datasource with will be able to view and use it. Setting a Datasource's visibility to Public means all users in your company will be able to view and use the Datasource, but only Editors will be able to modify it.
You can share a Datasource with specific users and assign them a role of either Viewer or Editor:
Viewers can view the Datasource configuration and files, and connect the Datasource to Agents they have access to.
Editors have all the permissions of a Viewer, plus the ability to modify the Datasource configuration, add/remove files, and share the Datasource with other users.
The user who created the Datasource has all Editor permissions plus the ability to delete the Datasource.
Important: Public Agent can use private Datasources.
By sharing a private Datasource with others, you are allowing those users to attach it to any Agent they create, including public Agents. While this greatly simplifies permissions since the creator of a Datasource does not have to share it with anyone who may have access via an Agent, it does create a loophole. It is possible to create a private Datasource, attach it to a public Agent and any requests to that Agent will work for all users who have access to the Agent.
Datasource Processing Time
Files you add to a Datasource may not be immediately available for use. Once you upload files to a Datasource, they must go through a processing pipeline. The processing time for your files will depend on many variables including number of files, size of files, processing queue, and other factors. There are status indicators on each file and the Datasource overall, to help you understand the processing progress.
In general, small single files can process almost immediately. Larger Datasources may take several minutes or over an hour to fully process. If you are concerned a Datasources is failing to fully process, please contact [email protected] for troubleshooting.
Datasource Types
File-Based
Dynamic Datasources
Files that are uploaded from an available data connector, such as Google Drive, Dropbox or Microsoft OneDrive will be available as a dynamic datasource.
These files will be periodically synced from the data connector, re-processed in elvex and available for use in conversations with the latest version of the file.
We will attempt to sync each file every 5 minutes.
What file types are supported?
For static or dynamic files supported formats include: csv, doc, docx, gdoc, gsheet, gslides, htm, html, js, json, jsonl, md, pdf, ppt, pptx, py, rtf, shortcut, ts, tsv, tsx, txt, xls, xlsx.
If you have a file or document in a currently unsupported format that you'd like to use as a datasource, convert it to one of the supported formats.
Please note that converting files from one type to another may not expose all of the content or data in the original file. For example, when converting a Google Slides presentation to a pdf file, the speaker notes will not be viewable in the pdf file by default. However, if you convert the Google Slides presentation to a txt file, the speaker notes will be available in the file. If there is important data in the speaker notes, it will be important to ensure that data is included in the file you upload so the AI can access it and reference it in its responses.
For files that contain images, elvex uses OCR (Optical Character Recognition) to extract and process text contained in images. Once processed, this extracted text is treated the same as any other text included in the document. Keep in mind that OCR may not be perfect so when possible, provide files that contain primarily text to ensure best results.
Adding multiple files to a single Datasource
In some cases, a Datasource may be a single file you upload. In other cases, it may be a collection of files. For example, a software company may have a "Sales and Support Knowledge Base" as a Datasource and this may include publicly available technical documentation, sales presentations, and customer onboarding documentation. Each of these files should uploaded to the same Datasource, allowing Agents that are connected to the Datasource to reference the full corpus of data when responding to user conversations. There is currently no limit to the number of files that can be uploaded to a Datasource.
Note: There are some cases where, in the above example, you may prefer to create separate Datasources for each of the file categories. This is not a problem as it’s possible to attach multiple Datasources to a single agent.
Database Connections
elvex supports Datasources made up of schemas from Snowflake, databases from Postgres and datasets from BigQuery.
Rather than making an individual Datasource for every table, if they serve the same purpose bundle the same tables into a single Datasource to answer the similar questions (eg. subscriber data, salesforce data).
Check out articles on how to set up external database connections.
Website
Users have the ability to choose publicly accessible websites as agent datasources. This is different in important ways from our “web searching” feature.
A simple explanation is that Web Searching is similar to actively Googling something when you submit a request during a conversation. It’s live, the scraping happens the moment you request it.
Websites as Datasources on the other hand, is set up similarly to other types of datasources: you connect them to your agent before you are inside a conversation, and the webpages also go through our datasource processing pipeline (RAG).
Examples when this Datasource makes sense could include a group of documentation sites that drive a product knowledge base agent, or multiple competitor product pages could supplement a competitor intelligence workflow.
Check out the article on setting up Websites as a Datasource.
Connecting a Datasource to an Agent
Datasources can be connected with Agents only after the Datasource has been created. A single Datasource can be connected to multiple agents with different purposes.