What got us here won’t get us there: The evolution of data tools to meet business needs

If you’ve ever felt like the world of data tools is an ever-increasing list of acronyms and buzzwords that mean less and less by the day - Excel, SQL, ETL, BI, warehouses, lakes, lakehouses - you’re not alone. Over the last decade working with organisations, especially in primary industries, I’ve seen one issue come up time and time again:

We from-excel-to-lakehouses-a-practical-history-of-data-tools-and-how-to-know-which-ones-you-actuallyaren’t struggling because we lack data, we’re struggling because we don’t know how to use it. What tools do we need and why?

A critical point in every organisation’s business journey is their choice of data tools. How can they choose the most appropriate tools if no one ever explained to them (in plain English) what tools are available and what problems they solve?

So today, instead of acronyms and buzzwords, I want to walk through a simple, practical history of data tools. Hopefully by the end you can finally understand the differences between them all and pick what’s right for your current needs.

Excel - where everyone starts

The loyal workhorse. The pocketknife. The chaos-maker.

If data tools had a “first love,” it would be Excel. Everyone starts here. I started here. You probably did too.

When it was released in 1985, Excel turned every employee’s computer into a number-crunching machine. It allowed users to put numbers in a grid, apply formulas, and see results immediately. It’s hands-on and practical with a low barrier to entry for employees but as soon as you need multiple people editing, or data volumes start becoming large, Excel becomes fragile and slow. It’s ideal for personal workloads or small projects but isn't built for heavy workloads.

What it’s good at:

• Quick analysis

• One-off reports

• Prototyping ideas

  
  

What it isn’t so good at:

• Files get too big

• People overwrite each other’s work

• “Final_FINAL_v3(1).xlsx” becomes a way of life

  
  

Excel is a fantastic place to start, it’s a terrible place to scale.

Databases - the grown-up version of Excel

Where real business data lives.

Once business systems evolve beyond a handful of spreadsheets, we need the next step up in data storage capability. We need a consistent and structured database.

A database is like moving your data from a notebook into a custom made storage facility with labelled shelves, barcodes, and a checkout system. A database stores information in a carefully organised structure that’s designed to be optimised for data storage. It can handle handle millions of rows, multiple users, and constant updates without failing. You rarely “see” the database but your business wouldn’t function without it.

It excels at:

• Keeping data consistent

• Powering apps and systems

• Storing transactional information

  
  

But it wasn’t built to answer big questions like:

• “What were our margins over the last three seasons?”

• “How are our processing times trending?”

• “Why does one region perform differently from another?”

  
  

Databases are custom made tools for storing data but they struggle to do any meaningful analysis. 

Data warehouses - the first big unlock

Finally, one place to get the same answers as everyone else.

By the 2000s, companies were drowning in data spread across systems. Finance had one version of revenue, sales had another, operations had a third and no one could agree on anything. Not only was data spread across many disparate locations and creating silos, but analytical workloads layered on top of these systems scaled to a point where they were crashing servers due to highly fluctuating workloads.

Data warehouses were the first tool designed to fix this. A warehouse lifts data out of disparate systems, cleans and transforms it to be in a common table format, and lands it into a storage location optimised for analytics. Think of it as a master library: every dataset is catalogued, quality-checked, and arranged into a single location so anyone can reliably use it for reporting.

This creates:

• Consistent metrics

• Trusted dashboards

• Fast reports

• One shared version of the truth

  
  

For the first time, leaders could rely on numbers instead of arguing about them. But warehouses are highly structured and want neat rows and tidy columns. That’s not always how real-world data behaves.

Data lakes — the messy but brilliant collector

“Dump everything here now; we’ll figure it out later.”

Around the 2010s, businesses realised they were leaving valuable data untouched because it wasn’t neat enough to be stored as tables in a data warehouse. Think about machine logs, drone imagery, PDFs, weather files. None of this could fit into the highly structured tabular format.

Enter the data lake.

A data lake is cheap, flexible, and happy to store absolutely anything you throw at it. Think of it as the C drive on your computer but on a massive scale. When you want to save a new file, you can just open a folder and throw it in there. Storage is extremely cheap by now and cloud-based data stores are easily scalable to meet any demand.

It’s the ideal playground for:

• Data science

• Machine learning

• Historical analysis

• Long-term storage

  
  

But lakes had a pretty obvious downside. Everyone kept on throwing things in without a solid plan and they turned into “data swamps” - full of extremely valuable data but impossible to find it.

Lakehouses - the modern solution that blends the best of everything

Fast like a warehouse, flexible like a lake, one platform instead of two.

A few years ago, companies began asking for something that didn’t exist yet - “Can we have warehouse-quality reporting and lake-level flexibility without maintaining two entire ecosystems?” The answer became the lakehouse.

A lakehouse maintains the cheap and flexible data storage enabled by a lake but adds a layer of structure over the top to emulate a data waerhouse. It’s fast, flexible, and future-proof.

Perfect for organisations that want:

• Real-time insights

• Scalable reporting

• Advanced analytics

• AI/ML experimentation

• Unified architecture

• Lower platform cost

  
  

This is the direction modern platforms - like Microsoft Fabric - are heading and for many organisations, especially those managing complex operations, it’s becoming the go-to approach.

So which tool do you actually need?

Here’s the simple truth:

You don’t need every tool. You need the right tool for your stage of growth.

If you’re early-stage or still small-ish -> start with Excel and a database or two.

If reporting is getting painful -> add a data warehouse

If you need flexibility, AI, or big data -> add a lake or lakehouse

It’s not about buzzwords, it’s about outcomes.

The real point: tools don’t matter - clarity does

No business leader wakes up excited about “building a modern data platform.”

You wake up wanting:

• To make decisions faster

• To reduce manual work

• To trust your numbers

• To understand what’s happening across your operations

  
  

The right tool helps you get there. The wrong tool creates more confusion, cost, and complexity.

My job - and the work we do at Data Wranglers - is helping businesses choose the simplest, smartest path to clarity. Whether that leads you to a warehouse, a lakehouse, or just getting your spreadsheets under control, we build practical solutions that deliver real outcomes.

If you’re unsure where your data maturity sits, or what tools you actually need next, I’d be happy to help.

Just reach out.