File from URL

Read one file from a web link: CSV, JSON, Parquet, Excel or XML.

Category

Input

Connection

REST API

Also known as

download, url, link, http, https, web, website, internet, presigned, open data, csv from url, excel from url

What it does

Downloads one file from a web link and reads it like any file node: CSV, JSON, Parquet, Excel or XML, and a .gz, .bz2, .xz, .zip or tar archive is opened for you. A public or presigned https link needs nothing else. A link that needs signing in uses a REST API connection (its bearer token, basic sign-in, API key or OAuth2) and then stays on that connection's own host. The link may carry tokens like ${date} and the pipeline's parameters.

Inputs and outputs

When to use it

• Load public reference data that is published as a file: exchange rates, a sanctions list, a price index
• Read a file a partner hands you as a presigned link
• Pull a report a vendor publishes at a fixed address every day

Examples

https://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist.zip

https://www.treasury.gov/ofac/downloads/sdn.csv

https://data.example.com/exports/${date-1:%Y%m%d}.csv

Sign in with a REST API connection · link: exports/daily.csv

Tips

• The format settings are the same as on the CSV Files, JSON Files, Excel Files and XML Files inputs: the delimiter, the encoding and the rows to skip, which sheet to read, and the record element.
• Turn on “Add source file name as column” to keep the link each row came from. The link's query is left out, because a presigned link carries its password there.
• Every run downloads the whole file, and so does a Test run (it then reads its 50 sample rows). Each run reads the file afresh: there is no 'only new files' here, because a link has no folder to remember.
• Press Refresh columns (↻) to read the file's columns. Typing a link never downloads it, so a one-time link is not used up while you type.
• Only https links are read without a connection, and a redirect to an http address is refused too. For an http server, create a REST API connection for it and switch on 'Sign in with a REST API connection'.
• An archive holding several files is read as one: later files are matched to the first by column name, and any file that differs is named in the run's messages.
• With a connection, the request only ever goes to that connection's host. A redirect to another host (a presigned download address) is followed without the connection's sign-in.
• A presigned link is a password that expires: anyone who can open this pipeline can use it until then, and the run's messages never print the part of the link after the '?'.
• Only direct file links: a web page with a download button, a script, or a sign-in form cannot be read.
• An Excel link whose address does not end in .xls or .xlsx is read as .xlsx.
• A failed connection, a server error (5xx), a 429 and a download cut short are retried up to three times, each retry named in the run's messages. A 404 or a 401 is not: the run stops and says so.
• Its Files anchor hands the downloaded file to a File Action, which can copy it byte for byte to S3, Azure Blob, GCS, SFTP or a network share. It cannot be moved or deleted.

Related nodes