Ever wish you could send your data as a stream so that the client can start manipulating it and rendering it as it arrives? Tired of waiting for your entire JSON object to resolve before your users see anything interesting? As you may have seen in my previous article on David Walsh Blog, this is now possible with the Fetch API! Stream responses are supported in Chrome 52 and in development in Firefox and Edge. This quick tutorial will show you how to set up a simple Express API to emit a ReadableStream of NDJSON.
Just want to see the code? Check out and remix it here.
What is NDJSON?
NDJSON is a data format that separates individual JSON objects with a newline character (\n). The 'nd' stands for newline-delimited JSON. You may have a file that contains JSON objects, each on their own line:
{"item":"first"}\n
{"item":"second"}\n
{"item":"third"}\n
{"item":"fourth"}\n
Each line can be sent individually over a stream, which allows our client to receive the data as a stream. Instead of reading lines from a file, you can also setup your service later to append newlines to each row received from a database before sending it to the client. In this example we will walk through how to read NDJSON from a file and stream it to the client.
Getting Started
First, make sure you have a recent version of Node.js installed.
Next, create a new project folder (and switch to it), initialize a package.json file, and install Express:
$ mkdir ndjson-stream-demo && cd ndjson-stream-demo
$ npm init -y
$ npm i express --save
Setup Basic Express Server
Now create a server file named server.js in your project directory:
$ touch server.js
Paste the following code for a basic setup.
/server.js
const express = require('express');
const app = express();
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});
Test your basic server setup by running this command in the same directory that your server.js file is in:
$ node server.js
Then navigate to http://localhost:3000/ to make sure you see "Hello World" displayed.
Add the demo page
Create a file using this HTML skeleton, name it ndjson-stream-demo.html and place it in the public/ directory, which should be in your project directory at the same level as your server.js file.
$ mkdir public && cd public
$ touch ndjson-stream-demo.html
Open that file and copy in this HTML. We will throw a script tag in this file later to interact with our data.
/public/ndjson-stream-demo.html
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>NDJSON Stream Demo</title>
</head>
<body>
<h1> NDJSON demo </h1>
</body>
</html>
Serve your demo page
Install the path dependency:
$ npm i path --save
Add these lines to serve your static public directory:
/server.js
const express = require('express');
const app = express();
const path = require('path');
app.use(express.static(path.join(__dirname, 'public')));
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});
Run your server and navigate to localhost:3000/ndjson-stream-demo.html
$ node server.js
Add sample data locally in the NDJSON format
todos.ndjson into your root directory.
Add filesystem to your project
Add the fs module to your server file, and have it read from your local NDJSON file. Make sure the path is correct to your NDJSON file, which should be named todos.ndjson:
/server.js
const express = require('express');
const app = express();
const path = require('path');
const fs = require('fs');
app.use(express.static(path.join(__dirname, 'public')));
app.get('/api', (req, res) => {
let readStream = fs.createReadStream(__dirname + '/todos.ndjson');
//setup headers
res.writeHead(200, {'Content-Type': 'application/ndjson'});
readStream.on('open', () => {
readStream.pipe(res); //pipe stream to response object
});
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});
When you make a fetch request to localhost:3000/api, it will download a file with an output that looks like this:
{"date":"2017-02-24 03:07:45","user":"21109850","fuel":"37","ammo":"2","steel":"13","baux":"5","seaweed":"0","type":"LOOT","product":"134"}
{"date":"2017-02-22 04:40:13","user":"21109850","fuel":"37","ammo":"2","steel":"13","baux":"5","seaweed":"0","type":"LOOT","product":"75"}
{"date":"2017-02-21 20:47:51","user":"26464462","fuel":"37","ammo":"3","steel":"19","baux":"5","seaweed":"1","type":"LOOT","product":"81"}
...
Source Code
Take a look at the example code for this step.
Convert regular JSON stream to an NDJSON stream
Now we are piping the data as regular JSON. Let's get started on that by using the ndjson package to parse our NDJSON in the service layer before streaming it to the client. For this example, we will be using a setInterval to throttle the stream so that we can see it in action.
Install the ndjson module
$ npm i ndjson --save
/server.js
const express = require('express');
const app = express();
const path = require('path');
const fs = require('fs');
const ndjson = require('ndjson');
app.use(express.static(path.join(__dirname, 'public')));
app.get('/', (req, res) => {
let readStream = fs.createReadStream(__dirname + '/todos.ndjson').pipe(ndjson.parse());
const chunks = [];
readStream.on('data', (data) => {
chunks.push(JSON.stringify(data));
});
readStream.on('end', () => {
var id = setInterval(() => {
if (chunks.length) {
res.write(chunks.shift() + '\n');
} else {
clearInterval(id);
res.end();
}
}, 500);
});
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});
Source Code
Check out the code for this step here.
Now your server is serving one line of NDJSON every 500 milliseconds! Move on to the next section to see your demo page consume the stream!*
*Note: fetch with ReadableStream is only supported in Chrome 52 or higher.
Receive and render your NDJSON:
Copy these scripts into your ndjson-stream-demo.html file to see your server in action. For more information about this code and the can-ndjson-stream npm module which parses your NDJSON stream into a ReadableStream of JS objects, see this blog and the documentation on canjs.com.
public/ndjson-stream-demo.html
//load the global can-ndjson-stream module to parse your NDJSON into JavaScript objects.
<script src='https://unpkg.com/can-ndjson-stream@0.1/dist/global/can-ndjson-stream.js'></script>
<script>
const streamerr = e => {
console.warn("Stream error");
console.warn(e);
}
fetch("/api").then((response) => {
return can.ndjsonStream(response.body);
}).then(todosStream => {
var reader = todosStream.getReader();
reader.read().then(read = result => {
if (result.done) {
console.log("Done.");
return;
}
console.log(result.value);
render(result.value);
reader.read().then(read, streamerr);
}, streamerr);
});
let counter = 0;
render = val => {
const div = document.createElement('div');
div.append('Fetched NDJSON row ', ++counter, ' : ', JSON.stringify(val));
document.getElementsByTagName('body')[0].append(div);
}
</script>
Final Source Code
See the final product or check out the example code to see it all together.
What's next?
Now that you know how to serve NDJSON, learn how to use can-ndjson-stream to stream data with fetch() and ndjson on the client-side.
Tell us about the creative ways you are using NDJSON in your application by tweeting to us at @canjs! If you need any help, please reach out to us in the CanJS Gitter or on the forums!
open source
Previous Post
