Building a CSV Parsing API with Next.js and csv-parser

Introduction

After building the CSV Viewer App with pure HTML, CSS, and JavaScript, I decided to take things a step further by creating a CSV Parsing API using Next.js and the csv-parser npm package. This API allows users to upload a CSV file, which is then parsed into structured JSON data. Along the way, I learned about streaming, error handling, and working with Node.js streams. In this blog, I’ll walk you through the key features, challenges, and lessons learned while building this API.

---

Key Features of the CSV Parsing API

1. File Upload and Base64 Encoding

   • Users upload a CSV file, which is converted to base64 for secure transmission to the backend.

   • Example:

       const buffer = Buffer.from(csvData, 'base64');
     

2. CSV Parsing with csv-parser

   • The csv-parser library is used to parse the CSV data into JSON objects.

   • Streaming ensures efficient processing of large files.

   • Example:

       stream.pipe(csvParser())
         .on('data', (row) => {
           // Process each row
         })
         .on('end', () => {
           resolve(NextResponse.json({ data: records }));
         });
     

3. Handling Invalid Characters

   • Invalid characters like � are replaced with valid ones (-) during parsing.

   • Example:

       cleanedRow[key] = row[key].replace(/�/g, '-');
     

4. API Endpoint

   • A POST API endpoint (/api/parse-csv) handles the CSV parsing logic.

   • Example response:

       { "data": [{ "name": "John", "age": "30" }, { "name": "Jane", "age": "25" }] }
     

---

Challenges and Solutions

1. Challenge: Invalid Characters

   • Problem: Some rows contained invalid characters (�), which caused issues when displaying the data.

   • Solution: Used a regular expression to replace � with -.

2. Challenge: Large Files

   • Problem: Large CSV files could overwhelm the server.

   • Solution: Implemented streaming with Readable to process data in chunks.

3. Challenge: Error Handling

   • Problem: Errors during parsing were not user-friendly.

   • Solution: Added proper error handling and returned meaningful error messages.

---

Lessons Learned

1. Understanding Streams

   • Learned how to use Node.js streams to process large files efficiently.

2. Working with APIs

   • Gained experience creating API endpoints in Next.js and handling requests/responses.

3. Data Cleaning

   • Learned the importance of cleaning and sanitizing data before processing.

4. Error Handling

   • Improved my ability to handle errors gracefully and provide meaningful feedback to users.