WhatsApp
PostgreSQL
MySQL
MongoDB
Redis
Python
AnthropicDatabases
| Database | Type | Best for |
|---|---|---|
| PostgreSQL | Relational | Our default: reliable, feature-rich, great for most apps |
| MySQL / MariaDB | Relational | WordPress, WooCommerce, many PHP applications |
| SQL Server | Relational | Microsoft and .NET environments |
| MongoDB | Document | Flexible, nested data and rapid schema changes |
| Redis | In-memory | Caching, queues, sessions and rate limiting |
| Elasticsearch / OpenSearch | Search | Full-text search, logs and analytics |
AI and machine learning
| Technology | Use |
|---|---|
| Large language models (Anthropic Claude, OpenAI GPT, Google Gemini) | Assistants, summarisation, extraction, agents |
| Cloud AI platforms (Amazon Bedrock, Azure OpenAI, Vertex AI) | LLMs with enterprise data controls |
| Vector search (pgvector, Pinecone, OpenSearch) | Retrieval-augmented generation over your documents |
| PyTorch, TensorFlow, scikit-learn | Custom models for prediction, vision and classification |
| Open-weight models (Llama, Mistral) | Self-hosted AI for privacy or cost at scale |
Data engineering and analytics
Data and AI technologies: getting your data ready
Most AI projects that disappoint do so because of data, not models. Before building, we assess whether the information the AI needs exists, is accessible and is trustworthy. Documents may be scattered across shared drives, email and old intranets; databases may contain duplicates and missing fields; and access permissions may not be documented. An assistant built on messy data gives messy answers.
Preparation work typically includes consolidating sources, removing outdated or duplicate documents, extracting text from PDFs and scans, adding metadata such as department, date and owner, and mapping permissions so users only see what they are entitled to. This work is valuable on its own — it improves search, reporting and compliance even before AI is added.
Analytics that people actually use
Dashboards often go unused because they answer questions nobody asked, or because nobody trusts the numbers. We start with the decisions people make each week and the metrics behind them, agree clear definitions — what counts as an active customer, when revenue is recognised — and build a small set of reliable dashboards on a single, documented data model. Automated data-quality checks flag problems before they reach a board report.
Choosing between building and buying AI
Many AI capabilities are now available inside tools you may already pay for — CRM assistants, help-desk bots, document summarisation in office suites. These are often the fastest way to get value for common tasks. Custom AI development makes sense when the task depends on your proprietary data, needs to be embedded in your own product, must follow specific rules or compliance requirements, or when off-the-shelf tools cannot integrate with your systems. We help you compare both options honestly before investing.
Choosing the right database
Most applications need one primary database plus one or two specialised stores. Starting with a relational database is the safe default; specialised databases are added when a specific workload calls for them.
| Type | Examples | Use it for |
|---|---|---|
| Relational | PostgreSQL, MySQL, SQL Server | Transactions, business records, anything that needs consistency and joins |
| Document | MongoDB, Firestore | Flexible or nested data where the schema changes often |
| Key-value / cache | Redis | Sessions, caching, rate limiting, queues |
| Search | Elasticsearch, OpenSearch, Typesense | Full-text search, filtering and faceted navigation |
| Analytical warehouse | BigQuery, Snowflake, Redshift | Reporting and analytics across large historical datasets |
| Vector | pgvector, Pinecone, Weaviate | Semantic search and retrieval for AI applications |
PostgreSQL covers a surprising range of these needs on its own — including JSON documents, full-text search and vector search through pgvector — which keeps early architectures simple. Learn more about database development.
A reference architecture for AI features
Adding AI to an existing product rarely means training a model from scratch. Most business use cases follow a pattern like this:
This is the approach behind retrieval-augmented generation and most OpenAI API integrations we build. It keeps your data in your systems, respects existing access rights and lets you switch model providers without rebuilding the application.
Data quality, privacy and governance
- Define ownership — every important dataset has a named owner who decides what “correct” means.
- Validate at the edges — check data as it enters pipelines, not after a dashboard looks wrong.
- Minimise sensitive data — send AI models only the fields they need, and mask personal data where possible.
- Respect access rights — retrieval for AI must apply the same permissions users have in the source system.
- Keep lineage — know which source and transformation produced each report or answer.
- Review vendor terms — confirm how AI and analytics providers store, retain and use your data.
Building a data and AI team
Early on, one experienced data engineer who can also build simple dashboards often delivers more than a large team. As needs mature, the roles separate: data engineers own pipelines and the warehouse, analysts own reporting and business questions, and machine-learning or AI engineers own models and AI features. Product and domain experts remain essential at every stage, because they decide whether an answer is actually useful.
You can hire data engineers or AI/ML developers to work with your team, or explore AI & data engineering services.
From spreadsheets to a modern data stack
Most organisations start with exports and spreadsheets. The path to reliable reporting usually follows a few steps: consolidate data from operational systems into a warehouse with automated pipelines, model it into clean business entities (customers, orders, products), build a small number of trusted dashboards on top, and only then add advanced analytics or machine learning. Skipping the middle steps is the most common reason AI and analytics projects disappoint.
Our data engineering and data analytics services cover this foundation; machine learning builds on it.
Build, fine-tune or use a hosted model?
| Approach | When it fits | Trade-offs |
|---|---|---|
| Hosted model via API | Most text, chat and document tasks | Fast to start; ongoing usage costs and dependency on a provider |
| Hosted model with retrieval (RAG) | Answers must come from your own documents | Needs good ingestion and permissions; very flexible |
| Fine-tuned model | Consistent format or domain style that prompting cannot achieve | Requires quality training examples and periodic retraining |
| Open-weight model self-hosted | Strict data residency or high, predictable volume | More infrastructure and MLOps responsibility |
| Classical machine learning | Forecasting, scoring and classification on tabular data | Often cheaper and more explainable than LLMs for these tasks |
For product features that act on a user’s behalf, see AI agent development and generative AI development.
Keeping AI costs predictable
Usage-based AI pricing can surprise teams once a feature reaches real traffic. We design for cost from the start: routing simple requests to smaller, cheaper models; caching answers to repeated questions; trimming the context sent with each request to what is actually relevant; and setting per-user and per-feature budgets with alerts. Dashboards show cost per request alongside quality metrics, so decisions about model choice are made with both in view.
For high, steady volumes, it can become worthwhile to self-host an open-weight model; we model the break-even point before recommending that step.
A 90-day path from scattered data to a first AI feature
-
Days 1–20: inventory and question
List data sources, owners and quality issues, and pick one business question or workflow worth improving.
-
Days 21–50: foundations
Set up ingestion into a warehouse or lakehouse, define the key tables and add basic data-quality tests.
-
Days 51–75: first use case
Ship a dashboard or an AI-assisted workflow — for example document summarisation or demand forecasting — to a small group.
-
Days 76–90: evaluate
Measure accuracy, adoption and time saved; decide whether to scale, adjust or stop.
Ninety days is a rule of thumb for a focused first use case with accessible data. Projects that need significant data clean-up or new data collection take longer, and we say so at the start.
Data and AI glossary
| Term | Plain-English meaning |
|---|---|
| ETL / ELT | Moving data from source systems into an analytics store, transforming it before or after loading |
| Lakehouse | A data platform combining cheap file storage with warehouse-style querying |
| Embedding | A numeric representation of text or images used for similarity search |
| RAG | Retrieval-augmented generation: grounding AI answers in your own documents |
| Model drift | A model’s accuracy falling as real-world data changes |
| Data lineage | A record of where each figure came from and how it was transformed |
Questions to answer before an AI project
- Which decision or task will the AI support, and who is accountable for the outcome?
- Is there enough representative, accessible data — and are we allowed to use it?
- How will we measure accuracy, and what error rate is acceptable?
- When should the system hand off to a person?
- What will it cost to run per user or per request at expected volume?
How we choose
Most applications need one solid relational database (usually PostgreSQL) plus Redis for caching; we add search, vector or warehouse technology only when a feature needs it. For AI, we benchmark models on your real data before committing. See AI & data engineering services, generative AI development and Python development.
Explore more in Software Development Technologies We Use.