This project is an implementation of a microblogging system (similar to the basic functionality of Twitter) using the microservice architecture and Node.js. It is the example system discussed in Chapter 1 of The Tao of Microservices book.
This purpose of this code base to help you learn how to design and build microservice systems. You can follow the construction through the following steps:
The system uses the Seneca microservice framework to provide inter-service communication, and the fuge microservice development tool to manage services on a local development machine.
The system is also a demonstration of the SWIM protocol for peer-to-peer service discovery. A service registry is not needed as the network automatically reconfigures as microservices are added and removed.
The system shows implementations of some of the essential features of a microblogging system, but not all. Of particular focus is the use of use of separate microservices for separate content pages, the use of messages for data manipulation, and the use of a reactive message flow for scaling.
The system does not provide for full accounts, or user authentication. This could be added relatively easily using the seneca-auth and seneca-user plugins. Avoiding the need to login makes it easier to experiment as you can check multiple user experiences in the browser.
The system exposes a (RESTish) JSON API over HTTP. However, the user interface does not use any client-side JavaScript, and entirely delivered by server-side templates. This is an old school POST and redirect architecture to keep things simple and focused on the server-side.
The system does not use persistent storage. You can easily make the data persistent by using a Seneca data storage plugin. Keeping everything in memory makes for faster development, easier experimentation, and lets you reboot the system if you end up with corrupted data during development.
This example codebase does not provide a production deployment configuration. To see such a configuration for a similar system, using Docker, take a look at the NodeZoo code base.
The system is implemented in Node.js. You will need to have Node.js version 4.0 or greater installed.
To run the system, follow these steps:
Follow the instructions at fuge repository.
fuge is a development tool that lets you manage and control a microservice system for local development. The ramanjun repository is preconfigured for fuge (see the fuge folder), so you don't have to set anything up. The ramanujan system has 14 microservices (at last count), so you really do need a local tool to help run the system.
This is trade-off that you make when you choose the microservice architecture. You can move faster because you have very low coupling, and thus lower technical debt, but you will need more automation to manage the higher number of moving parts.
Use git to clone the repository to a local development folder of your choice
$ git clone https://github.com/senecajs/ramanujan.gitThe system needs a number of Node.js modules from npmjs.org to function correctly. These are the only external dependencies.
$ npm installWait until the downloads complete. Some modules will require local compilation. If you run into problems due to your operating system, using a Linux virtual machine is probably your fastest solution.
From within the repository folder, run the fuge shell.
$ fuge shell fuge/system.ymlThis will start fuge, output some logging messages about the ramanujan services, and then place you in an interactive repl:
...
starting shell..
? fuge> Enter the command help to see a list of commands. Useful commands
are ps to list the status of the services (try it!), and exit to
shutdown all services and exit. If your system state becomes corrupted
in some way (this often happens during development due to bugs in
microservices), exit fuge completely and restart the fuge shell.
To start the system, use the fuge command:
...
? fuge> start allYou see a list of startup logs from each service. fuge prefixes the logs for each service with the service names, and gives them different colors so they are easy to tell apart. This also makes is easy to review message flows. The system takes about a few seconds to start all microservices.
Now use the ps command to list the state of the services. They
should all be running.
Open your web browser to interact with the system. The steps below define a "happy path" to validate the basic functionality of the system.
Open http://localhost:8000/foo.
This is the homepage for the user foo, and shows their timeline. The timeline is a list of recent microblog entries from all users that the user foo follows, and also entries from foo themselves.
At first there are no entries, so go ahead and post an entry, say:
three colors: blue
Click the post button or hit return. You should see the new entry.
Post another entry, say:
three colors: white
You should see both entries listed, with the most recent one at the top. This is the timeline for user foo.
Open http://localhost:8000/mine/foo (Or click on the Mine navigation tab).
This shows only the entries for user foo, omitting entries for followers.
You can use this page to verify the entry list for a given user.
Open http://localhost:8000/search/bar.
You are now acting as user bar. Use the text blue as a search query:
Click on the follow button. Now user bar is following user foo.
Open http://localhost:8000/bar (Or click on the Home navigation tab).
You should see the entries from user foo, as user bar is now a follower.
Enter and post the text:
the sound of music
The timeline for user bar now includes entries from both users foo and bar.
Return to user foo. Open http://localhost:8000/foo.
Post a new entry:
three colors: red
You should see entries only for user foo, as foo does not follow bar.
Go back to user bar. Open http://localhost:8000/bar.
You should see an updated list of entries, included all the entries for user foo, as bar does follow foo.
One of the main benefits of a microservice system is that you can deploy services independently. In a local development setting this means you should be able to start and stop services independently, without stopping and starting the entire system. This has a huge productivity benefit as you don't have to wait for the entire system to ready itself.
To work on a particular service, update the code for that service, and then stop and restart the service to see the new functionality. The rest of the system keeps working. To really get the maximum benefit from this technique, you need to avoid the use of schema validation for your messages, and you must avoid creating hard couplings (services should not know about each other). That is why the Seneca framework provides pattern matching and transport independence as key features - they enable rapid development.
The payoff for more deployment complexity is that you can change parts of the system dynamically - don't lose that ability!
fuge allows you to start and stop services using the 'start' and 'stop' commands.
To stop a service (say, search), use the command:
? fuge> stop searchIf you now try to use the search feature, it will fail, but other pages will still work. Another important benefit of microservices is that they can isolate errors in this way.
To restart the search service, use:
? fuge> start searchAnd the search functionality works again. Notice that you did not have to do any manual configuration to let the other services know about the new instance of the search service. Notice also that the other services knew almost instantaneously about the the new instance of the search service. That's becuase the SWIM algorithm propogated that information quickly and efficiently throughout the network. No need for 30 second timeouts to detect errors - SWIM works much more quickly as it has many observers (the other services) so can detect failure, and new services, very quickly with a high degree of confidence.
You can also run multiple instances of the same service. This lets you scale to handle load. The underlying seneca-mesh network will automatically round-robin messages between all available services for a given message. Just start the service again:
? fuge> start searchAnd is you now run the ps command in fuge, you'll see the count is 2
instances.
The system comes with a REPL service that lets you submit messages to the network manually. This is very useful for debugging. Access the REPL by telnetting into it:
$ telnet localhost 10001Use the following message to see the user foo's timeline:
seneca 2.0.1 7k/repl> timeline:list,user:foo
IN 000000: { timeline: 'list', user: 'foo' } # t7/39 timeline:* (6ln6zlc2qaer) transport_client
OUT 000000: { '0':
{ user: 'foo',
text: 'three colors: red',
when: 1461759716373,
can_follow: false },
'1':
{ user: 'foo',
text: 'three colors: white',
when: 1461759467135,
can_follow: false },
'2':
{ user: 'foo',
text: 'three colors: blue',
when: 1461759353996,
can_follow: false } }You can enter messages directly into the terminal, in JSON format (the
format is lenient, see
jsonic). The output will show
the message data IN and OUT of the network.
The REPL is a JavaScript console environment. There is a seneca
object that you can use directly, calling any methods of the seneca
API.
seneca 2.0.1 7k/repl> seneca.id
'7k/repl'To get a list of all services on the network, and which messages they listen for, try:
seneca 2.0.1 7k/repl> role:mesh,get:members
IN 000001: { role: 'mesh', get: 'members' } # aa/ie get:members,role:mesh (9mxp6qx6zyox) get_members
OUT 000001: {
...
'4':
{ pin: 'timeline:*',
port: 54932,
host: '0.0.0.0',
type: 'web',
model: 'consume',
instance: 'gt/timeline-shard' },
...
}This message is so useful, that the repl service defines an alias for it: m.
The default configuration of the system uses shortened identifers to make debugging easier.
TODO
TODO
TODO
Copyright (c) Richard Rodger and other contributors 2015-2016, Licensed under [MIT][].