You can send job results directly to your OneDrive account.
Continue to the following topics:
Prerequisites Use the TD Console to Create Your Connection
Create a New Connection Configure Output Results to the Data
Connection Specify the Result Export Target Execute the Query
Example of a Query
Optionally Schedule the Query Export Jobs Execute the Query
Optionally Configure Export Results in Workflow Use the CLI to
Create Your Connection
Install ‘td’ Command For On-Demand Jobs For Scheduled Jobs
Frequently Asked Questions Q: How do I export to a folder that
someone shared with me? Q: How can I export files to a SharePoint
collection?
Appendix Domain Name and Relative Path Get Your Own refresh_token
Requested Permissions
Prerequisites Basic knowledge of Treasure Data, including the TD
Toolbelt A OneDrive or SharePoint account Authorized Treasure Data
account access
Use the TD Console to Create Your Connection
Create a New Connection
When you configure a data connection, you provide authentication to
access the integration. In Treasure Data, you configure the
authentication and then specify the source information.
Go to Integrations Hub -> Catalog and search and select
OneDrive.
The following dialog opens.
To authenticate, complete the following procedure:
Select on to connect to a new account.Click here
Log into your OneDrive account in the popup window and grant access
to the Treasure Data app.
You will be redirected back to TD Console. Repeat the first step
(Create a new connection) and choose your new OAuth
connection.
Name your new OneDrive Connection. Select Done.
Configure Output Results to the Data Connection
In this step, you create or reuse a query. In the query, you
configure the data connection.
Go to the TD Console query editor page.
Access the query that you plan to use to export data.
Compile the query, if you need to.
Specify the Result Export Target
Select .Export Results
Type the connection name in the search box to filter and select
your OneDrive connection.
Edit all the information.
OneDrive Account Plan: there are two account plans:
Business: you must specify a domain, relative path, and folder
path. Personal: you must specify the folder path and whether it is
a shared folder or not.
Domain Name: domain of the folder.
Relative Path: the relative path of the user's folder in OneDrive
or SharePoint site.
Folder Path: directory of the folder in OneDrive or
SharePoint.
File Name: destination file name. Multibyte characters cannot be
used.
This is a shared folder: select if the folder is shared with you by
someone
Format: file extension.
Compression: indicate whether the result is to be compressed or
not. Available Options: None, gz, and bzip2.
Header line: select if the exported data has the column name as the
header line.
Null String: use this value to represent NULL values. Available
options:
Default. select if an empty string ( ) is used to represent the
Null value for file format CSV, and \N is used for the TSV format.
'' Empty string \N NULL null
End-of-line character: the character at the end of lines. Available
options are CRLF, LF, and CR
Quote Policy: available options:
ALL. select if all values are enclosed by double quotes ("").
MINIMAL. select if any value that contains an embedded quote (") is
presented with a consecutive pair of quotes (""). The MINIMAL quote
policy is applied to CSV. NONE. select if no escape for embedded
quote is applied. By default, NONE is applied to TSV file
format.
Max Retries: when an error occurs due to a OneDrive server error or
network error, the number of retries attempted before aborting the
upload.
Initial Retry Time Wait In Millies: when error occurs, initial
waiting time for a retry, double the waiting time after each
retry.
Max Retry Wait In Millies: The upper limit waiting time for a
retry.
Execute the Query
Save the query with a name and run, or just run the query.
After the query has run successfully, the query result are
automatically imported into the specified Container
destination.
Example of a Query
SELECT email, first_name, last_name, region, age, gender, website
FROM ( VALUES ('
[email protected]', 'Kate', 'Tiny', 'Asia', '41 and
above', 'female', 'google.com'), ('
[email protected]', 'R', 'P',
'Americas', '21-30', 'male', 'google.com'), ('
[email protected]',
'M', 'C', 'EMEA', '31-40', 'male', 'facebook.com') ) tbl (email,
first_name, last_name, region, age, gender, website);
Optionally Schedule the Query Export Jobs You can use Scheduled
Jobs with Result Export to periodically write the output result to
a target destination that you specify.
1. Navigate to .Data Workbench > Queries
2. Create a new query or select an existing query.
3. Next to , select None.Schedule
4. In the drop-down, select one of the following schedule
options.
Drop-down Value Description
Custom cron... Review .Custom cron... details
@daily (midnight) Run once a day at midnight (00:00 am) in the
specified time zone.
@hourly (:00) Run every hour at 00 minutes.
None No schedule.
Custom cron... Details
0 0 * * * Run once a day at midnight
0 0 1 * * Run once a month at midnight on the morning of the first
day of the month
"" Create a job that has no scheduled run time.
* * * * * - - - - - | | | | | | | | | +----- day of week (0 - 6)
(Sunday=0) | | | +---------- month (1 - 12) | | +---------------
day of month (1 - 31) | +-------------------- hour (0 - 23)
+------------------------- min (0 - 59)
The following named entries can be used:
Day of Week: sun, mon, tue, wed, thu, fri, sat Month: jan, feb,
mar, apr, may, jun, jul, aug, sep, oct, nov, dec
A single space is required between each field. The values for each
field can be composed of:
Field Value Example Example Description
a single value, within the limits displayed above for each
field.
a wildcard to indicate no restriction based on ‘*’ the
field.
‘0 0 1 * *’
configures the schedule to run at midnight (00:00) on the first day
of each month.
a range , indicating the range of accepted ‘2-5’ values for the
field.
‘0 0 1- 10 * *’
configures the schedule to run at midnight (00:00) on the first 10
days of each month.
a list of comma-separated values , ‘2,3,4,5’ indicating the list of
accepted values for the field.
0 0 1,11,21 * *’
configures the schedule to run at midnight (00:00) every 1st, 11th,
and 21st day of each month.
a periodicity indicator to express how often ‘*/5’ based on the
field’s valid range of values a schedule is allowed to run.
‘30 */2 1 * *’
configures the schedule to run on the 1st of every month, every 2
hours starting at 00:30. configures the schedule to run at midnight
(00:00) every 5 ‘0 0 */5 * *’ days starting on the 5th of each
month.
a comma-separated list of any of the above except the wildcard is
also supported ‘*’ ‘2,*
. /5,8-10’
‘0 0 5,* /10,25 * *’
configures the schedule to run at midnight (00:00) every 5th, 10th,
20th, and 25th day of each month.
5. (Optional) If you enabled the Delay execution, you can
delay the start time of a query.
Execute the Query
Save the query with a name and run, or just run the query. Upon
successful completion of the query, the query result is
automatically imported to the specified container
destination.
Optionally Configure Export Results in Workflow Within Treasure
Workflow, you can specify the use of this data connector to export
data.
Learn more at . Using Workflows to Export Data with the TD
Toolbelt
Example Workflow for OneDrive
Use the CLI to Create Your Connection
Scheduled jobs that continuously fail due to configuration errors
may be disabled on the system side after several
notifications.
For On-Demand Jobs
Add the OneDrive result output destination by using the / option
for the command:-r --result td query
td query -d test_db -w 'SELECT id, via FROM table1' --type presto
-r '{"type":"one_drive", "refresh_token":"
***","account_type":"personal","domain_name":null,"server_relative_path":null,"folder_path":"/abc","
is_shared_folder":false,"file_name":"test_file","replace_existing":null,"format":"csv","compression":"","
header_line":true,"null_string":"default","newline":"CRLF","quote_policy":null,"retry_count":"5","
retry_initial_wait_millis":"1000","max_retry_wait_millis":"300000"}'
For Scheduled Jobs
Add the OneDrive result output destination by using the / option
for the command:-r --result td sched:create
td sched:create every_6_mins "*/6 * * * *" -d test_db -w 'SELECT
id, via FROM table1' --type presto -r '{{"type":"one_drive",
"refresh_token":"***","account_type":"personal","domain_name":null,"
server_relative_path":null,"folder_path":"/abc","is_shared_folder":false,"file_name":"test_file","
replace_existing":null,"format":"csv","compression":"","header_line":true,"null_string":"default","newline":"
CRLF","quote_policy":null,"retry_count":"5","retry_initial_wait_millis":"1000","max_retry_wait_millis":"
300000"}'
Frequently Asked Questions
Q: How do I export to a folder that someone shared with me?
Business Account Plan:
You need the , server , and Go to the Shared With Me tab on the
browser:domain name relative path folder path.
Select to open the shared folder and make note of the domain name,
relative path and folder path as described in Domain Name and
Relative Path.
Personal Account Plan
You must set the field: ( ) to , and drill down to the shared
folder as shown in the image.This is a shared folder
is_shared_folder true
Q: How can I export files to a SharePoint collection?
You need a , server and to import files from SharePoint
collections. See Domain Name and Relative PathDomain Name Relative
Path, Folder Path
Appendix
Domain Name and Relative Path
Domain Name and URL are required to access your OneDrive business
account plan, including your personal folder, SharePoint site
Relative Path collections, and folders shared with you.
Open the folder that you want to import on the browser. Then get
the Domain Name and Relative Path as shown:
Personal folder
SharePoint collection
Folder shared with me
Get Your Own refresh_token
To issue jobs from CLI, you need , and .client_id client_secret,
refresh_token
Register a developer account, and go to Azure App Registration
Portal . On the search box, search for App
Registrationhttps://portal.azure.com/#home
Select New Registration
Enter , select and enter as shown. Select App Name Account Type
Redirect URI Register.
From window, select -> and add permissions as shown:Request API
permissions Microsoft Graph Delegated permissions
From the tab, select on , select and .Certificates & secrets
New client secret Never Add
A new key is created as shown:client secret
From the Overview tab, you see the as shown:client id
Replace the client_id in this URL and open it on a browser.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?
client_id=xxxxxx&response_type=code&redirect_uri=http://localhost&response_mode=query&scope=offline_access
files.readwrite.all openid
sites.readwrite.all&state=1234
Accept the window.Permission Request
Copy the from the address bar:code
Replace the and run a command to get the and code, client_id,
client_secret curl access_token refresh_token
curl -X POST \
https://login.microsoftonline.com/common/oauth2/v2.0/token \
-H 'Content-Type:
application/x-www-form-urlencoded' \
-d 'client_id={xxxxxx}&client_secret={xxxxx}&grant_type=authorization_code&scope=offline_access%20files.
readwrite.all%20openid%20sites.readwrite.all&code={xxxxxx}&redirect_uri=http://localhost'
Requested Permissions
OneDrive Connector will ask you for following permissions if you
are using personal account:
Permission Description Note
User.Read Allows users to sign-in to the app, and allows the app to
read the profile of signed-in users. It also allows the app to read
basic company information of signed-in users.
Files. ReadWrite. AppFolder
Allows the app to read, create, update, and delete files in the
application's folder.
Application's folder is a dedicated, special folder for
"TreasureData OneDrive Connector" app see What is an App
Folder
Files. ReadWrite.All
Allows the app to read, create, update, and delete all files the
signed-in user can access.
The import and export Connector shares the same client application
and only performs read, create and update but not delete
files.
offline_access Allows the app to read and update user data, even
when they are not currently using the app.
Use for schedule import/export
Permission Description Note
Sites.ReadWrite.All Read and write all groups, Edit or delete items
in all site collections only performs read, create and update but
not delete files.
Note: Although the Delete permission is granted by default as part
of the Sites. ReadWrite. All permission, the data connector does
not delete files on Microsoft OneDrive.