Nodejs

Distributed adventures

created by @tralamazza

Disclaimer

Shameless

Spark Plugs by Aidan Wojtas

Minion @

We do

Automatic software testing

Server

• THE middleware
• SES
• SNS
• Cloudfront
• S3
• Mixpanel
• SQL
• Redis ...
• 20k LoC
• npm-shrinkwrap.json ~ 23KB

Distributed Applications

• error prone
• hard to get it right
• difficult difficult lemon difficult
• ... and so on and so on (Slavoj Žižek ™)

A Rover in Mars

Still trying to name things

• mDNS
• DNS-SD
• SSDP (MS)
• Bonjour (Apple)
• SLP (hahaha)
• DHCP (!!!)

dnode

Well known stream based RPC solution.

It just works [tm].

dservice

or how I can't name things

• Stream (dnode) based
• Remote dependencies
• Self healing, load balanced, and so on and so on
• Semver'd

A Dnode server

// var dnode = require('dnode');
var dservice = require('dservice');

// var server = dnode({
var server = dservice('d1', {
  transform: function (s, cb) {
    cb(s.replace(/[aeiou]{2,}/, 'oo').toUpperCase());
  }
});
// server.listen(5004);
server.start();
					

and its client

var dservice = require('dservice');

var service = dservice();
service.dnodeFor('d1', function (err, remote, d) {
  remote.transform('beep', function (s) {
    console.log('beep => ' + s);
    d.end();
  });
});
					

Remote dependencies

var dservice = require('dservice');

var server = dservice('d2', {
  jsconf: function (cb) {
    server.dnodeFor('d1', function (err, remote, d) {
      remote.transform('rulez', cb);
    });
  }
});
server.rely('d1');
server.start();
					

[WiP]

• Cycle detection
• Multiple engines
• package.json hack: RemoteDependency

Streams vs Transactions

• Streams are composable, awesome, etc
• Don't solve all the problems
• You will buffer
• Sometimes you have to repeat operations

Transactional behaviour

• I want repeatable operations
• I want durability
• and I specially want Sugar[TM]

How it works?

• user implements idempotent steps
• serialize your command
• 1 input + 2 working queue per step
• handles timeouts and retries

Implementing a step

var dns = require('dns');
function MyResolveStep(logger, pipeline) {
  var self = this;
  self.input_queue = 'my_input_queue';
  self.execute = function (payload, done) {
    dns.resolve(payload.data.address, function (err, addresses) {
      done(err, addresses, err ? 'my_resolved_addr_q' : 'my_error_q');
    });
  };
  return self;
}
module.exports = MyResolveStep;
					

Everything to be open sourced

So Long

and Thanks for all the Fish