Data Engineering in the AI Era: Why Data Matters More Than Models

In the AI industry, there's a well-known saying: 'Garbage in, garbage out.' No matter how sophisticated the model architecture or how abundant the computing power, if the data quality is poor, the AI product will not be effective. Data engineers are the key role in ensuring AI systems 'get good food.'

Why is Data More Important Than the Model?

In 2024, Andrej Karpathy (a founding member of OpenAI) mentioned in a speech: 'In most AI projects, 80% of the time and effort should be spent on data, not on model tuning.' This is not an exaggeration—in actual enterprise AI projects, data preparation indeed occupies the vast majority of the time.

A Real-World Case

An e-commerce company wanted to build an AI product recommendation system. They spent two months tuning model parameters, but the recommendation results were consistently unsatisfactory. Later, a data engineer joined the team and spent three weeks cleaning and organizing user behavior data—removing fake browsing records generated by crawlers, fixing the chaotic product category labels, and filling in missing user profile fields. After the data cleaning was completed, the simplest collaborative filtering algorithm achieved the same results as the previous complex model, and the recommendation accuracy increased by another 25%.

What Do Data Engineers Do?

The core responsibility of a data engineer is: to ensure the right data, in the right format, at the right time, appears where it is needed. Specific tasks include the following aspects.

Data Collection

Collecting data from various sources: business databases, user behavior logs, third-party APIs, web crawlers, sensor data, etc. The key challenge is handling format differences and connection stability across different data sources.

Data Cleaning and Transformation

Raw data is almost always 'dirty'—it has null values, duplicates, inconsistent formats, and erroneous data. Data cleaning involves fixing these issues to make the data usable. This is where data engineers spend most of their time.

**Common Issues and Handling Methods**: Missing values (deletion, filling with default values, or estimation using statistical methods), duplicate records (designing deduplication rules), inconsistent formats (e.g., unifying date formats, standardizing addresses), outliers (determining if they are erroneous data or genuine extreme values).

Data Pipelines

Automating the entire flow of data from source to destination. A typical data pipeline: automatically pulls the previous day's data from the business database at 2 AM daily → cleans and transforms it → loads it into the data warehouse → triggers report updates. This process is called ETL (Extract, Transform, Load).

Data Storage and Management

Choosing the appropriate storage solution: Relational databases (MySQL/PostgreSQL, suitable for structured data and transactions), Data warehouses (BigQuery/ClickHouse, suitable for large-scale analytical queries), Data lakes (suitable for storing raw unstructured data), Vector databases (Milvus/Pinecone, specifically designed for AI retrieval).

New Requirements for Data Engineering in the AI Era

Traditional data engineering primarily served BI (Business Intelligence) and reporting. The AI era brings new demands:

**Feature Engineering**: Transforming raw data into features that models can use. For example, 'user login count in the last 7 days' or 'month-over-month change rate of order amount'—these derived features are crucial for model performance.

**Training Data Management**: AI models require high-quality labeled data for training. Data engineers need to design data labeling processes, manage labeling quality, and maintain versions of training datasets.

**Vectorization and Embedding**: RAG applications require converting documents into vectors for storage in vector databases. This process involves text chunking strategies, embedding model selection, and index optimization.

Core Tools

**Python + pandas**: Foundational tools for data processing, must master. **SQL**: Essential language for interacting with databases. **Apache Airflow/Prefect**: Data pipeline orchestration tools. **dbt**: Data transformation tool, enabling SQL to be managed like software engineering. **Spark**: Large-scale data processing engine, essential for handling TB-level data.

Career Path

The salary level for data engineers is in the upper-middle range among technical positions. In first-tier cities in China, data engineers with 3 years of experience typically have an annual salary of 300,000 to 500,000 RMB, and those with AI project experience can earn even more. Learning path: SQL basics → Python data processing → ETL tools → Cloud platform data services → AI data pipelines.

After understanding the full picture of data engineering, the next chapter will delve into data labeling and quality management—the most critical factors determining the quality of AI models.