If you've been seeking guidance on how to integrate Whisper into your website, and are also seeking clarity on the App Router system introduced in the latest version of Next.js, then you've arrived at the perfect destination.
In this tutorial, we'll walk through the process of building a speech-to-text application using Next.js 13, the latest version of the popular React framework. We'll leverage the new App Router in Next.js 13, which provides a unified, server-centric routing system that simplifies the routing structure and improves the performance of Next.js applications. Let's get started!
If you haven't already, create a new Next.js project by running the following command in your terminal:
npx create-next-app@latest my-appReplace "my-app" with the name of your project. Navigate into your new project directory:
cd my-appNext.js has built-in support for loading environment variables from .env files into process.env.
Create a new file in the root of your project named .env. This file will be used to store your OpenAI API key. The contents of the file should be:
# .env
OPENAI_API_KEY=your-openai-api-keyReplace your-openai-api-key with your actual OpenAI API key.
Important: Never commit your .env file into source control. It should be ignored by your .gitignore file by default. This file contains sensitive information that should not be shared or made public.
Now, when you run your Next.js application, the OPENAI_API_KEY environment variable will be loaded from the .env file, and you can access it in your code using process.env.OPENAI_API_KEY.
The OpenAI package is the only one that isn't part of Next.js or Node.js module in this project. Install it using npm:
npm install openaiThe frontend code of this application is responsible for handling user interactions and managing the state of the application. It provides a user interface where users can start and stop audio recording. When a user starts recording, the application captures audio from the user's microphone and stores it in chunks. Once the user stops recording, the application combines these chunks into a single audio Blob, converts it into a Base64 string, and sends it to the server-side function via a POST request. The server-side function transcribes the audio to text and returns the transcribed text in the response, which the frontend then displays on the page. The frontend code uses React hooks for state management and side effects, and it leverages the MediaRecorder API to capture audio from the user's microphone.
Here's how you can set up the app/page.jsx:
"use client";
import styles from './page.module.css'
import { useState, useEffect } from "react";
export default function Home() {
const [result, setResult] = useState();
const [recording, setRecording] = useState(false);
const [mediaRecorder, setMediaRecorder] = useState(null);
let chunks = [];
useEffect(() => {
if (typeof window !== 'undefined') {
navigator.mediaDevices.getUserMedia({ audio: true })
.then(stream => {
const newMediaRecorder = new MediaRecorder(stream);
newMediaRecorder.onstart = () => {
chunks = [];
};
newMediaRecorder.ondataavailable = e => {
chunks.push(e.data);
};
newMediaRecorder.onstop = async () => {
const audioBlob = new Blob(chunks, { type: 'audio/webm' });
const audioUrl = URL.createObjectURL(audioBlob);
const audio = new Audio(audioUrl);
audio.onerror = function (err) {
console.error('Error playing audio:', err);
};
audio.play();
try {
const reader = new FileReader();
reader.readAsDataURL(audioBlob);
reader.onloadend = async function () {
const base64Audio = reader.result.split(',')[1];
const response = await fetch("/api/speechToText", {
method: "POST",
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ audio: base64Audio }),
});
const data = await response.json();
if (response.status !== 200) {
throw data.error || new Error(`Request failed with status ${response.status}`);
}
setResult(data.result);
}
} catch (error) {
console.error(error);
alert(error.message);
}
};
setMediaRecorder(newMediaRecorder);
})
.catch(err => console.error('Error accessing microphone:', err));
}
}, []);
const startRecording = () => {
if (mediaRecorder) {
mediaRecorder.start();
setRecording(true);
}
};
const stopRecording = () => {
if (mediaRecorder) {
mediaRecorder.stop();
setRecording(false);
}
};
return (
<main className={styles.main}>
<div className={styles.description}>
<h2>Convert audio to text -></h2>
<button onClick={recording ? stopRecording : startRecording}>
{recording ? 'Stop Recording' : 'Start Recording'}
</button>
<p>{result}</p>
</div>
</main>
)
}This server-side code handles POST requests to the /api/speechToText route. It receives audio data in the request body, converts the audio data to text using the OpenAI API, and returns the transcribed text in the response. The code uses the ffmpeg command to convert the audio data to MP3 format, which is required by the OpenAI API.
Next.js allows us to easily create API routes. In your project, create a new file at app/api/route.js. This will be our server-side function for handling the speech-to-text conversion.
import { Configuration, OpenAIApi } from "openai";
import { exec } from 'child_process';
import fs from 'fs';
import { NextResponse } from "next/server";
const util = require('util');
const execAsync = util.promisify(exec);
const configuration = new Configuration({
apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);
export async function POST(request) {
if (!configuration.apiKey) {
return NextResponse.json({ error: "OpenAI API key not configured, please follow instructions in README.md" }, { status: 500 });
}
const req = await request.json()
const base64Audio = req.audio;
const audio = Buffer.from(base64Audio, 'base64');
try {
const text = await convertAudioToText(audio);
return NextResponse.json({ result: text }, { status: 200 });
} catch (error) {
if (error.response) {
console.error(error.response.status, error.response.data);
return NextResponse.json({ error: error.response.data }, { status: 500 });
} else {
console.error(`Error with OpenAI API request: ${error.message}`);
return NextResponse.json({ error: "An error occurred during your request." }, { status: 500 });
}
}
}
async function convertAudioToText(audioData) {
const mp3AudioData = await convertAudioToMp3(audioData);
const outputPath = '/tmp/output.mp3';
fs.writeFileSync(outputPath, mp3AudioData);
const response = await openai.createTranscription(
fs.createReadStream(outputPath),
'whisper-1'
);
fs.unlinkSync(outputPath);
return response.data.text;
}
async function convertAudioToMp3(audioData) {
const inputPath = '/tmp/input.webm';
fs.writeFileSync(inputPath, audioData);
const outputPath = '/tmp/output.mp3';
await execAsync(`ffmpeg -i ${inputPath} ${outputPath}`);
const mp3AudioData = fs.readFileSync(outputPath);
fs.unlinkSync(inputPath);
fs.unlinkSync(outputPath);
return mp3AudioData;
}Now, you should be able to run your application and test the speech-to-text functionality. Start your Next.js development server by running:
npm run devNavigate to http://localhost:3000 in your web browser. You should see your application and be able to start and stop recording. The recorded audio will be sent to the Whisper API for conversion to text, and the result will be displayed on your page.
Remember to handle errors and edge cases appropriately in your application. This guide provides a basic example, and there may be additional considerations for your specific use case.
That's it! You've successfully integrated the Whisper API into a Next.js application.
If you encounter issues while following this tutorial, here are some tips that might help you debug your application:
If you're still having trouble, you can refer to the complete project code on GitHub.
Remember, debugging is a normal part of the development process. Don't get discouraged if things don't work right away. With patience and persistence, you'll be able to solve any issues you encounter. Happy coding!