Nightly multi-source ingestion
Scheduled jobs that connect to SFTP sites, SharePoint document libraries, Azure file shares, websites, REST APIs and email attachments, land the raw files, and keep every version. Nobody downloads a CSV by hand again.
One trusted dataset, every morning, from every source you have.
We build the Databricks pipelines that pull data in nightly from fund administrators, custodians, SharePoint libraries, Azure file shares, SFTP sites, websites and APIs; clean and validate it; calculate what the business needs; and serve it to Power BI, internal workbenches and platforms like Foundation. When a file is late or malformed, the right person knows before breakfast.
What we build on Databricks
The unglamorous plumbing that decides whether your reports, your AI and your investors can trust a number. Built as Databricks Workflows and notebooks, with lineage, tests and alerts, by engineers who have done it for investment funds and reinsurers.
Cleaning, validation and alerts
Schema checks, reconciliations and business rules run on every load. When data doesn't arrive, arrives late, or arrives malformed, an alert goes out with what failed and what to do; the job retries on a schedule you set.
Calculations the business relies on
NAV calculations, positions and exposure roll-ups, reconciliations between administrator and custodian, fee and performance figures: coded, tested and versioned rather than living in a spreadsheet only one person can run.
A consolidated, related dataset
Bronze, silver and gold layers in the lakehouse: positions, prices, NAV, counterparties, deals and documents joined into one model with lineage, ready for reporting, workbenches, Foundation and AI assistants.
A typical night
From nine sources to one truth, before the office opens.
This is the shape of the platform we run for an ILS manager and a reinsurer today: raw files land overnight, Databricks jobs validate and transform them, exceptions raise alerts, and by the time the team arrives the gold tables are refreshed and every downstream tool is reading the same numbers.
The same pattern scales from a handful of feeds to dozens, and the same tables feed Power BI, internal workbenches, Foundation and an AI assistant that can answer questions against the lakehouse.
Where the data lives todayMonday · 06:00
Fund administratorSFTP · nightly CSV● Arrived 04:12
CustodianSFTP · positions● Late
Pricing vendorREST API● 200 OK
Deal filesSharePoint library● 12 new docs
Legacy exportsAzure file share● Malformed header
Market dataWebsite scrape● Arrived 05:40
Broker bordereauxEmail attachments● 3 received
SpreadsheetsAnalyst desktops● Unknown
Policy adminSQL Server● Live
Internal workbenchOps team · same gold tables● Live
FoundationPortfolio roll-up · nightly feed● Synced
AI assistantAsks questions of the lakehouse● Connected
Power BI
Reports delivered quickly, without a fuss.
Good data deserves reporting that keeps up with it. We have extended our Power BI practice with the latest AI agents, which lets us turn a requirements conversation into a working, well-structured report in days, and iterate on it in front of you.
Every report ships with row-level security designed in from the start, so a fund analyst, a client and an executive each see exactly the slice they are entitled to from the same dataset. And because we integrate Power BI with source code repositories, every change is versioned, reviewable and deployable the same way as the pipelines beneath it.
AI-accelerated build
Reports and semantic models drafted with AI agents, then reviewed and tuned by an engineer who knows the data. Days, not weeks.
Row-level security from day one
One report, many audiences. Security rules live in the model and are tested with the data, not bolted on afterwards.
Source control for Power BI
Reports and models stored in Git alongside the Databricks code: pull requests, history, rollback and repeatable deployment.
Served from the lakehouse
Reports read the same gold tables as the workbenches and Foundation, so there is one version of every number.
Built for investment funds and (re)insurers
We run platforms of this shape for an ILS manager and a reinsurer today, and the pattern applies anywhere a business depends on nightly data from administrators, custodians, brokers, cedants or vendors.
ILS and investment managers
Administrator and custodian feeds reconciled nightly; NAV, positions and exposure calculated and served to investor reporting and portfolio tools.
How we work →Reinsurers
Bordereaux, cedant statements, policy admin and payment data consolidated for real-time payment and reporting tracking.
How we work →Anyone drowning in feeds
If the month-end process starts with somebody downloading files, there is a platform like this waiting to be built, usually in weeks.
Talk to us →How a data platform engagement runs
- 01Inventory the feedsA working session with operations and finance: every source, its format, its owner, how it arrives and what breaks. You get the map and a prioritised order.
- 02First pipeline in a weekThe highest-value feed ingested, validated and landed in the lakehouse, with alerts live, so the team sees the pattern working on real data.
- 03Build out and calculateRemaining feeds, the related data model, the calculations (NAV, reconciliations, roll-ups) and the Power BI reports with row-level security, in Git, deployed from a pipeline.
- 04Hand overDocumented, monitored and owned by you, with FlexTech support if you want someone on call for the morning the custodian changes its file format.
Questions we get asked
What sources can you ingest?
SFTP sites, SharePoint document libraries, Azure file shares, websites, REST APIs, email attachments and databases such as SQL Server, on a nightly or more frequent schedule.
What happens when a file is late or malformed?
Validation runs on every load. If data does not arrive, arrives late or fails schema or business-rule checks, an alert goes to the right people with what failed and what to do, and the job retries on your schedule.
Can you do NAV and reconciliation calculations in Databricks?
Yes. NAV, positions and exposure roll-ups, administrator-to-custodian reconciliations and fee calculations are coded, tested and versioned in Databricks rather than living in a spreadsheet one person can run.
How do Power BI reports fit in?
Reports and semantic models read the same gold tables as your workbenches. We build them with AI assistance, design row-level security in from the start, and keep them in source control alongside the pipelines.
How long does a first pipeline take?
The highest-value feed is typically ingested, validated and landed in the lakehouse with alerts live within the first week.
Tired of month-end starting with a download?
Thirty minutes with a data engineer who has built this for funds and reinsurers. You'll leave with a feed inventory template and a scoped first pipeline.