Benoit Simard

Your computer is a graph !

Sat, 29 Jun 2019 00:00:00 +0200

Table of Contents

The model
Import items
Conclusion

In this article I will show you how to import all the components of a Linux computer (package network connection, user, group, …), into a graph with the help of Neo4j.

In an next article, we will play with this dataset and see what we value we can get.

The model

This is the model I want to produce :

Basicaly, I have a server that is composed of a list of:

installed packages with their dependencies
network interfaces with their connections
hardware (as tree)
groups and users
process (as a tree)

From that model, we can create the indexes/constraints for the database :

CREATE CONSTRAINT ON (n:Server) ASSERT (n.fqdn) IS NODE KEY;
CREATE CONSTRAINT ON (n:Group) ASSERT (n.gid, n.fqdn) IS NODE KEY;
CREATE CONSTRAINT ON (n:User) ASSERT (n.uid, n.fqdn) IS NODE KEY;
CREATE CONSTRAINT ON (n:Process) ASSERT (n.pid, n.fqdn) IS NODE KEY;
CREATE CONSTRAINT ON (n:PackageSection) ASSERT (n.name) IS NODE KEY;
CREATE CONSTRAINT ON (n:Package) ASSERT (n.name) IS NODE KEY;
CREATE CONSTRAINT ON (n:PackageVersion) ASSERT (n.version, n.architecture,n.name ) IS NODE KEY;
CREATE CONSTRAINT ON (n:Hardware) ASSERT (n.id, n.fqdn ) IS NODE KEY;
CREATE CONSTRAINT ON (n:Vendor) ASSERT (n.name ) IS NODE KEY;
CREATE CONSTRAINT ON (n:Product) ASSERT (n.name, n.vendor ) IS NODE KEY;

Import items

Computer name

The first thing to retrieve is the FQDN of the computer. For this I will use the command : hostname --fqdn.

$> hostname --fqdn
pythagore

This will help us to create the node Server :

MERGE (:Server {fqdn:$fqdn});

Hardware

To retrieve the entire composition of the server, I will use the command lshw that list all the hardware in a tree format.

Moreover, the command can generate a JSON file if we pass the argument -json:

FILE="$NEO4J_HOME/import/$1/hardware.json"
  sudo lshw -json > $FILE

{
  "id" : "pythagore",
  "class" : "system",
  "claimed" : true,
  "description" : "Computer",
  "width" : 64,
  "capabilities" : {
    "vsyscall32" : "processus 32 bits"
  },
  "children" : [
    {
      "id" : "core",
      "class" : "bus",
      "claimed" : true,
      "description" : "Motherboard",
      "physid" : "0",
      "children" : [
        {
          "id" : "memory",
          "class" : "memory",
          "claimed" : true,
          "description" : "Mémoire système",
          "physid" : "0",
          "units" : "bytes",
          "size" : 16695070720
        }
        ...
      ]
    }
  ]
}

The import process of this file will be a little complexe, because it’s a JSON tree with an undeterminated depth. So I need to create a recursive query to create its structure in neo4j.

For this I will use massively APOC for that !

For each item of the JSON, I want to create this : (Vendor)-→(Product)←-(ProductInstance)←-(ParentHardwareInstance)

If I translate that in Cypher in a custom procedure, it gives that :

CALL apoc.custom.asProcedure(
	'hardware',
  "
  MATCH (p) WHERE id(p)=$parent

  MERGE (vendor:Vendor { name:coalesce($item.vendor, 'unknown')})
  MERGE (product:Product { name:coalesce($item.product, 'unknown'), vendor:coalesce($item.vendor, 'unknown')})
    ON CREATE SET product.description = $item.description
  MERGE (vendor)-[:HAS_PRODUCT]->(product)

  MERGE (hardware:Hardware {id:$item.id + '-' + coalesce($item.physid, '') + '-' + coalesce($item.handle, '') , fqdn:$fqdn})
  SET hardware +=apoc.map.removeKeys($item,['id', 'children', 'capabilities', 'configuration', 'vendor', 'product'])
  MERGE (p)-[:HAS_HARDWARE]->(hardware)
  MERGE (hardware)-[:TYPE_OF]->(product)

  WITH  hardware
  UNWIND coalesce($item.children, []) AS newItem
    CALL custom.hardware($fqdn, id(hardware), newItem) YIELD row
    RETURN row
  ",
  "write",
  [["row","MAP"]],
  [
    ['fqdn','STRING'],
    ['parent','LONG'],
    ['item','MAP']
  ]
);

As you see at the line 18, I recall the procedure itself if there are some children. Pretty cool no ?

Note	The documentation for the cypher custom procedure feature is available here : https://neo4j-contrib.github.io/neo4j-apoc-procedures/#cypher-based-procedures-functions

Then I just have to call the procedure with the data like that :

CALL apoc.load.json("file:///" + $fqdn + "/hardware.json") YIELD value WITH value
 MATCH (s:Server {fqdn:$fqdn})
 CALL custom.hardware($fqdn, id(s), value) YIELD row
 RETURN count(*);

And to finish, I put the class attribut as a Label :

MATCH (n:Hardware)
CALL apoc.create.addLabels( id(n), [ n.class ] ) YIELD node
REMOVE node.class
RETURN count(*)

If you want to see the result in the browser, this is the query for just displaying the tree structure of hardwares :

MATCH p=(n:Server {fqdn:'pythagore.logisima'})-[:HAS_HARDWARE]->(:Hardware)-[:HAS_HARDWARE*]->(h:Hardware)
WHERE NOT (h)-[:HAS_HARDWARE]->()
RETURN *

User & Group

Users

Users are stored in the file /etc/users, where each line represents a user. For example : bsimard:x:1000:1000:Benoit Simard,,,:/home/bsimard:/bin/bash

Username: The login of the user, bsimard
Password: Generally password is set to x. It indicates that password are stored in the file /etc/shadow.
User ID (UID): The ID of the user.
Group ID (GID): The ID of the main group of the user
User ID Info: Some information about the user seperated by a ,, like the fullname, phone, email
Home directory: The home directory of the user /home/bsimard
Command/shell: The command for the shell, in my case /bin/bash

This is the Cypher script to import this file in Neo4j :

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/users.csv" AS row FIELDTERMINATOR ':'
  MERGE (g:Group { gid:toInteger(row.gid), fqdn:$fqdn})
  MERGE(u:User {uid: toInteger(row.uid), fqdn:$fqdn})
  SET u.username = row.username,
      u.fullname = split(row.info, ',')[0],
      u.home_directory = row.home,
      u.shell = row.shell
  MERGE (g)-[:HAS_USER]->(u);

Groups

Groups are stored in the file /etc/groups, where each line represents a group. For example : adm:x:4:syslog,bsimard

Group name: Name of the group, in our example it’s adm
Password: Generally password is not used, hence it is empty/blan, ie. x
Group ID (GID): Each group must be assigned a group ID, , in our example it’s 4
Group List: It is a list of user names of users who are members of the group, separated by a ,. In our example syslog and bsimard

This is the Cypher script to import this file in Neo4j :

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/groups.csv" AS row FIELDTERMINATOR ':'
  MATCH (s:Server {fqdn:$fqdn})
  MERGE (g:Group { gid:toInteger(row.gid), fqdn:$fqdn })
  MERGE (s)-[:HAS_GROUP]->(g)
  SET g.name = row.name
  WITH row, g
  UNWIND split(row.users, ",") AS username
    MATCH (u:User { username:username, fqdn:$fqdn})
    MERGE (g)-[:HAS_USER]->(u);

The result

Process list

To retrieve the list of all the running process, with their dependencies, I will use the ps command with those arguments :

e: Select all processes
o: To specify the format with
- pid: The ID of the process
- ppid: The ID of the parent process
- comm: The command that has launched the process
- ruser: The username of the user that owns the process

$> ps -eo pid,ppid,comm,ruser

PID,PPID,%CPU,SIZE,COMMAND,RUSER
1,0,0.0,18964,systemd,root
2,0,0.0,0,kthreadd,root
4,2,0.0,0,kworker/0:0H,root
6,2,0.0,0,mm_percpu_wq,root

With the help of awk, we can generate a CSV file :

FILE="$NEO4J_HOME/import/$1/processes.csv"
  ps -eo pid,ppid,comm,ruser | awk '{print $1","$2","$3","$4}' > $FILE

And this is the cypher script to import it :

MATCH (s:Server {fqdn:$fqdn})
MERGE (p:Process {pid:0, command:'init', fqdn:$fqdn})
MERGE (s)-[:HAS_PROCESS]->(p);

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/processes.csv" AS row
WITH row ORDER BY row.PPID, row.PID  ASC
  MATCH (u:User {username:row.RUSER, fqdn:$fqdn})
  MERGE (pp:Process {pid:toInteger(row.PPID), fqdn:$fqdn})
  MERGE (p:Process {pid:toInteger(row.PID), fqdn:$fqdn})
  SET p.command = split(row.COMMAND, '/')[0]
  MERGE (pp)-[:HAS_PROCESS]->(p)
  MERGE (p)-[:OWNED_BY]->(u);

When the import is finished, you should have something similar to this by excuting the following query :

MATCH path=(n:Server {fqdn:'pythagore.logisima'})-[:HAS_PROCESS]->(:Process)-[:HAS_PROCESS*]->(p:Process)
WHERE NOT (p)-[:HAS_PROCESS]->()
RETURN *

We can also search all the processes of a user :

MATCH (n:Server {fqdn:'pythagore.logisima'})-[:HAS_GROUP|:HAS_USER*2..2]->(u:User {name:'bsimard'})
WITH u
MATCH (u)<-[:OWNED_BY]-(p)
RETURN *

List of installed packages

Because I have a debian system, I will use the dpkg command, and produce a CSV from its output :

FILE="$NEO4J_HOME/import/$1/packages.csv"
  dpkg-query -Wf '${Section}\t${Package}\t${Version}\t${Architecture}\t${Homepage}\t${Depends}\n' >> $FILE

I can import it with the following cypher script :

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/packages.csv" AS row FIELDTERMINATOR '\t'
  MATCH (s:Server {fqdn:$fqdn})

  MERGE (section:PackageSection {name:coalesce(row.section, 'None') })
  MERGE (package:Package {name:row.package})
  SET package.url = row.homepage
  MERGE (section)-[:HAS_PACKAGE]->(package)

  MERGE (pv:PackageVersion { version:row.version, architecture:row.architecture, name:row.package})
  MERGE (package)-[:HAS_VERSION]->(pv)

  MERGE (s)-[:HAS_PACKAGE]->(pv)

  FOREACH( dep IN split(row.dependencies, ',') |
    MERGE (depNode:Package {name:split(dep, '(')[0] })
    MERGE (pv)-[r:HAS_DEPENDENCY]->(depNode)
    SET r.constraint = replace(split(dep, '(' )[1], ')','')
  );

If you want to see the result, you can run the following query :

MATCH (n:Server {fqdn:'pythagore.logisima'})-[:HAS_PACKAGE]->(:Package)-[:HAS_PACKAGE*]->(p:Package)
WHERE NOT (p)-[:HAS_PACKAGE]->()
RETURN *

Network

Network interfaces

To get the list of the network interface, I will use the ip command with the following arguments :

r: See the routing table
n: Display the IP adresses
w: Don’t truncate IP addresses

$>ip -o addr show

1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: wlp58s0    inet 10.0.1.11/24 brd 10.0.1.255 scope global dynamic wlp58s0\       valid_lft 2720sec preferred_lft 2720sec
2: wlp58s0    inet6 fe80::cc06:f22c:2e3b:6aaf/64 scope link \       valid_lft forever preferred_lft forever
3: br-8a43a73a9b0d    inet 172.24.0.1/16 brd 172.24.255.255 scope global br-8a43a73a9b0d\       valid_lft forever preferred_lft forever
4: br-f8fefbd33c18    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-f8fefbd33c18\       valid_lft forever preferred_lft forever
5: br-0ba6b709cb23    inet 172.21.0.1/16 brd 172.21.255.255 scope global br-0ba6b709cb23\       valid_lft forever preferred_lft forever
6: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
7: br-667de2307c35    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-667de2307c35\       valid_lft forever preferred_lft forever
8: br-72683ea3a3d0    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-72683ea3a3d0\       valid_lft forever preferred_lft forever

For my need, I need the IP addresses (v4 & v6) of the Interface and its name. So by using awk, we can produce a CSV file :

FILE="$NEO4J_HOME/import/$1/network_interfaces.csv"
  ip -o addr show  | awk '{print $2","$3","$4}' >> $FILE

Now I can load it in Neo4j :

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_interfaces.csv" AS row
WITH row,
  CASE  WHEN  row.type = 'inet' THEN split(row.ip, '/')[0] ELSE NULL END AS ipv4,
  CASE  WHEN  row.type = 'inet6' THEN split(row.ip, '/')[0] ELSE NULL END AS ipv6

  MATCH (s:Server {fqdn:$fqdn})

  MERGE (i:Interface { name: row.name, fqdn:$fqdn })
  SET i.ip = coalesce(ipv4, i.ip),
      i.ipv6 = coalesce(ipv6, i.ipv6)

  MERGE (s)-[:HAS_INTERFACE]->(i);

It gives me the following graph :

Network connections

To see all the connections on our computer, I will use the netstat command, with the following arguments :

a: Display every socket
u: Filter on UDP sockets
t: Filter on TCP sockets
p: Display the process ID that use the socket
n: Display the IP adresses
w: Don’t truncate IP addresses

Note	you need to run this command as a root to see all the connections

$> netstat -alpuetn

Connexions Internet actives (serveurs et établies)
Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      124        28742       1317/mysqld
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      0          35483       3192/dnsmasq
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      0          4024761     5070/cupsd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      125        28000       1434/postgres
tcp        0      0 127.0.0.1:38630         127.0.0.1:7687          ESTABLISHED 1000       4856935     14182/firefox
tcp        0      0 10.0.1.11:39694         198.252.206.25:443      ESTABLISHED 1000       5159133     14182/firefox
tcp        0      0 127.0.0.1:39044         127.0.0.1:7687          ESTABLISHED 1000       4908177     14182/firefox
tcp        0      0 10.0.1.11:38388         52.222.163.46:443       ESTABLISHED 1000       5190592     14182/firefox
tcp        0      0 127.0.0.1:39034         127.0.0.1:7687          ESTABLISHED 1000       4903190     14182/firefox
tcp        0      0 10.0.1.11:49408         216.58.209.238:443      ESTABLISHED 1000       4994841     14182/firefox
tcp        0      0 10.0.1.11:42724         172.217.18.197:443      ESTABLISHED 1000       5163155     14182/firefox
tcp        0      0 10.0.1.11:38316         52.222.163.46:443       ESTABLISHED 1000       5157419     7587/libpepflashpla
tcp        0      0 127.0.0.1:38632         127.0.0.1:7687          ESTABLISHED 1000       4853218     14182/firefox
tcp        0      0 10.0.1.11:39698         198.252.206.25:443      ESTABLISHED 1000       5159174     14182/firefox
tcp        0      0 10.0.1.11:58296         10.0.0.1:445            ESTABLISHED 0          4628873     -
tcp        0      0 127.0.0.1:39046         127.0.0.1:7687          ESTABLISHED 1000       4914563     14182/firefox

Like above, I will create a CSV :

FILE="$NEO4J_HOME/import/$1/network_connections.csv"

I will split in two parts (with sub-parts), the import of this file :

processes that listen on a port
- On IPv4
- On IPv6
- On all interfaces
processes that have an established connection
- With an IPv4
- With an IPv6

Processes that listen

On Ipv4

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'LISTEN' AND
    size(split(row.local, '.')) > 1 AND // only IPv4
    (NOT row.local = '::' AND NOT row.local = '0.0.0.0') // Not for all interfaces
  WITH
    row,
    replace(row.local, ':'+ local_port, '') AS local_ip,
    local_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (i:Interface { ip: local_ip, fqdn:$fqdn })
    MERGE (p:Port { number: local_port, ip:local_ip })
    MERGE (i)-[:USES_PORT]->(p)
    SET p.ip = i.ip,
        p.ipv6 = i.ipv6

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:LISTEN_ON]->(p)
    );

On Ipv6

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'LISTEN' AND
    size(split(row.local, '.')) = 0 AND // only IPv6
    (NOT row.local STARTS WITH '::' AND NOT row.local STARTS WITH '0.0.0.0') // Not for all interfaces
  WITH
    row,
    replace(row.local, ':'+ local_port, '') AS local_ip,
    local_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (i:Interface { ipv6: local_ip, fqdn:$fqdn })
    MERGE (p:Port { number: local_port, ipv6:local_ip})
    MERGE (i)-[:USES_PORT]->(p)
    SET p.ip = i.ip,
        p.ipv6 = i.ipv6

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:LISTEN_ON]->(p)
    );

On all interfaces

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'LISTEN' AND
    ( row.local STARTS WITH '0.0.0.0') //  all interfaces
  WITH
    row,
    local_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (i:Interface {fqdn:$fqdn })
    MERGE (p:Port { number: local_port, ip:i.ip})
    SET p.ipv6 = i.ipv6
    MERGE (i)-[:USES_PORT]->(p)

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:LISTEN_ON]->(p)
    );

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'LISTEN' AND
    ( row.local STARTS WITH ':::*') //  all interfaces
  WITH
    row,
    local_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (i:Interface {fqdn:$fqdn })
    MERGE (p:Port { number: local_port, ipv6:i.ipv6})
    SET p.ip = i.ip
    MERGE (i)-[:USES_PORT]->(p)

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:LISTEN_ON]->(p)
    );

Graph example

This is so cool, at the end we obtain the graph below just by running the following query :

MATCH path=(n:Server {fqdn:'pythagore.logisima'})-[:HAS_INTERFACE]->(i:Interface)-[:USES_PORT]->(:Port)<-[:LISTEN_ON]-(p:Process)
RETURN path

Processes with a connection

This time, I import all the connections (incoming/outgoing) that computers have.

On Ipv4

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(last(split(row.remote, ':'))) AS remote_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'ESTABLISHED' AND
    size(split(row.local, '.')) > 1 // only ipv4
  WITH
    row,
    replace(row.local, ':'+ local_port, '') AS local_ip,
    local_port,
    replace(row.remote, ':'+ remote_port, '') AS remote_ip,
    remote_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (s:Server {fqdn:$fqdn})

    MATCH (iLocal:Interface {ip: local_ip, fqdn:$fqdn })
    MERGE (iLocal)-[:USES_PORT]->(pLocal:Port { number: local_port})
    SET pLocal.ip = iLocal.ip,
        pLocal.ipv6 = iLocal.ipv6

    MERGE (iLocal)-[:USES_PORT]->(pLocal)

    MERGE (pRemote:Port { number: remote_port, ip:remote_ip })

    MERGE (con:Connection { fqdn:$fqdn, pid:coalesce(pid,'-'), local:row.local, remote:row.remote})
    MERGE (con)-[:SOURCE]->(pLocal)
    MERGE (con)-[:TARGET]->(pRemote)

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:HAS_CONNECTION]->(con)
    );

On Ipv6

LOAD CSV WITH HEADERS FROM "file:///" + $fqdn + "/network_connections.csv" AS row
  WITH
    row,
    toInteger(last(split(row.local, ':'))) AS local_port,
    toInteger(last(split(row.remote, ':'))) AS remote_port,
    toInteger(split(row.pid,'/')[0]) AS pid
  WHERE
    row.state = 'ESTABLISHED' AND
    size(split(row.local, '.')) = 0
  WITH
    row,
    replace(row.local, ':'+ local_port, '') AS local_ip,
    local_port,
    replace(row.remote, ':'+ remote_port, '') AS remote_ip,
    remote_port,
    pid,
    CASE WHEN pid IS NULL THEN [] ELSE [1] END AS ifPid

    MATCH (s:Server {fqdn:$fqdn})

    MATCH (iLocal:Interface {ipv6: local_ip, fqdn:$fqdn })
    MERGE (iLocal)-[:USES_PORT]->(pLocal:Port { number: local_port})
    SET pLocal.ip = iLocal.ip,
        pLocal.ipv6 = iLocal.ipv6

    MERGE (pRemote:Port { number: remote_port, ip:remote_ip })

    MERGE (con:Connection { fqdn:$fqdn, pid:coalesce(pid,'-'), local:row.local, remote:row.remote})
    MERGE (con)-[:SOURCE]->(pLocal)
    MERGE (con)-[:TARGET]->(pRemote)

    FOREACH( x IN ifPid |
      MERGE (process:Process {pid:pid, fqdn:$fqdn})
      MERGE (process)-[:HAS_CONNECTION]->(con)
    );

Graph example

And the result is :

MATCH path=(n:Server {fqdn:'pythagore.logisima'})-[:HAS_INTERFACE]->(i:Interface)-[:USES_PORT]->(:Port)<-[:SOURCE]-(:Connection)-[:TARGET]->(p:Port)
OPTIONAL MATCH path2=(p)<-[:USES_PORT]-(:Interface)<-[:HAS_INTERFACE]-(:Server)
RETURN path, path2

Conclusion

It’s really cool to see all the components of a computer as a graph, and if we do that on every computer of a network (hey nmap), we can have a complete vision of it, just with a discovery process.

In a next article, I will show you what we can do with those data, stay tune !

How to use Kettle to import data in Neo4j

Mon, 03 Sep 2018 00:00:00 +0200

Table of Contents

Kettle
Neo4j connector
Conclusion

Kettle

Kettle (or Pentaho Data Integration) is a well known ETL tool. It is mature tool, simple to learn due to its GUI and open-source.

Moreover, Kettle has a lot of plugins for all various data editors, so can really do every thing you want with your data. In this post, I will show you how to use the Neo4j components, but firstly, you need to have ketlle on your computer.

To install kettle, it’s pretty easy, you only need a JDK installed on your laptop (>= version 8), and then follow those steps :

Download the last binaries at this location : https://sourceforge.net/projects/pentaho/files/Data%20Integration/7.1/
Extract it
Run spoon.sh

Note	Spoon is the application name of the graphic interface on which you will design your process

Neo4j connector

There are some Neo4j connectors on the Kettle marketplace, but I strongly recommend you to use this one : https://github.com/knowbi/knowbi-pentaho-pdi-neo4j-output It is efficient, updated and maintained.

It is based on the official Neo4j Java Driver and have all the options you want to import your data in Neo4j, even on a cluster environment.

For now, this plugin is not on the marketplace but it will be.

How to install it

Go to this page https://github.com/knowbi/knowbi-pentaho-pdi-neo4j-output/releases/, and get the latest release.
Unzip it into the plugins folder of your Kettle installation
Restart spoon

Components

This plugin comes with 2+1 components :

Neo4j Cypher : a very useful component where you can write a cypher query, and it can be used as an input or output.
Neo4j Output : allow you to dynamically create (or update) nodes or relationships. It will generate the cypher query for you.
Neo4j Graph Output: By doing a graph mapping on the input fields, it will create/update the graph for you. But this component is in WIP.

Neo4j Cypher

This is (from my POV) the main component of this plugin (I’m a Cypher expert, so I like to write my own query ^^).

The first thing you need to do is to declare a Neo4j connection. If you have previously declare a Neo4j connection you can reuse it, otherwise you need to create a new one by fulfill this form :

As you can see, the connection is using the Bolt protocol, and you can also configure a cluster connection with a routing policy. So it’s really complete.

The batch size parameter allow you define the size of your transaction. In my example, Every 500 rows the component will do a commit. This is really important to control the transaction size during an import process, to have the best performances.

Your Cypher query can be totally dynamic, ie, can be defined from an input field. To do it, you just have to enable the Get Cypher from input field? and then specify the Cypher input field. Otherwise, you have to write your query in the Cypher field.

Another good point is that the component allows you to use query parameters. In my example, $name and $year come from the input fields. As you can see, I have defined them into the Parameters section, just below.

And last point, If you want to use this component as an Input one, you have to define the Returns section with the name and type.

For me, you have the hand on everything to create an efficient import process. The only lacks are in the UI (because I’m a lazy person ^^), where I would like to have :

a Get Fields button on the Parameters section, to fulfill it with all the input fields
a compute return values button to automatically fulfill the Return section.

The github issue is already created, and Matt Caster (the father of Kettle and one of the dev. of this plugin) is on it.

Neo4j Output

This component allow you to create nodes and relationships without writing a cypher query, it will do it for you.

You can use it if you have Input fields define :

A node : you have to use only the From Node tab
Two nodes plus the relationship between them : you have to define the From node, To node and Relationship tabulation.

Same as before, you can define the Batch size, and moreover you can also tell the component to create the Neo4j UNIQUE CONSTRAINTS for you if you enable the Create indexes? and specify the Primary fields.

This component has been designed to have the best performances for Neo4j. For example if you create nodes with a batch size, the component will collect your input fields in an array and generate a query like that :

WITH $data
UNWIND $data as $object
  CREATE ....

So you will not have many queries, but only one that match your batch size. This is really cool for performances.

Example for nodes and relationship

Imagine that you have the following input fields : label, id, name, parent_id, type What you want to do is to create :

a node with the label label and with the properties id and name.
a relationship between the current node and its parent_id with the type type

Then you need to fulfill the component by following those screenshots

Note	don’t use the `CREATE` mode for such a process, otherwise the `To` node will be created each time.

Tips

This component needs to have an input field for the label of nodes and for the relationship type. If you don’t have one because those value are static, you can use the Add constants component that allows you to a constant field to the fields :

If you are using the MERGE mode, sometimes it is useful to define a default value for the property on which do the merge. To do it, you can use the Value Mapper component like this :

in this example, if the parent_id is not set, I replace it with the value 0.

Conclusion

This post is just an overview of Kettle, but as you can see its integration with Neo4j is really easy. I recommend you to test it, and if you have questions, requests or issues, don’t hesitate to create an issue on the github repository.

Have fun, and boil your graph imports !

Monitoring Neo4j with influxdb TICK stack

Tue, 22 May 2018 00:00:00 +0200

Table of Contents

Introduction
Influxdb
- Concept
- The TICK stack
The architecture
The monitored server
- Neo4j
- Telegraf
The monitoring server
Conclusion

Introduction

When you push a service in production, it’s really important to monitor its health status. This will allow you to see if everything is OK, to be alerted if something is going wrong but also in a case of a problem, to investigate.

This rule should be applied for Neo4j, and you will see how to do it with the TICK stack (from influxDB).

But before to explain how to do it for Neo4j, lets talk a little about Influxdb.

Influxdb

Concept

Influx DB is a time series database. It’s made to store and query data in times, so it’s the perfect tool to store metrics of a system.

Influx has 7 key concepts : time, field, tag, measurement, series, retention policy and continous queries.

It’s better to take an example to understand those concepts.

Imagine that you have a captor of temperature and humidity in your living room and an other one in your bedroom. Each time you will read one, you will receive the data at a point of time.

Because it’s the same kind of data, you will store them at the same location. But you also want to be able to distinguish the data from the living room and the bedroom.

At the end, you will have something like that :

name: captor_temperature_humidity
----------------------------------
time                    temperature   humidity   room
2015-08-18T00:00:00Z    18.3          51.2       Living Room
2015-08-18T00:00:00Z    16.7          48.9       Bedroom
2015-08-18T00:01:00Z    18.5          51.1       Living Room
2015-08-18T00:01:02Z    16.9          49.0       Bedroom
...

Here we have a measurement name called captor_temperature_humidity (you can think about it like a table in SQL) and each line is measurement.

Measurement is composed of : time , temperature, humidity and room

Temperature and humidity are measurement’s fields, and their values are the data you want to follow in time.

On each measurement, I have added the room to know where comes the measurement. It’s a measurement’s tag (a metadata), and you can have multiple tags for a measurement.

By the time, you will have a very huge number of measurement, and usually you don’t want to save them forover. It’s where the concept of retention policy takes place, you can configure the database to delete measurements older than X (X is hours, days, weeks, …, years).

CREATE RETENTION POLICY one_week ON my_database DURATION 1w;

Note	The default retention policy is to keep the data forever.

Moreover, you can define some continious queries to aggregate measurements over time. it will create a new (aggregated) measurement on the same database but with a new retention policy :

CREATE CONTINUOUS QUERY "my_continious_query" ON "my_database"
BEGIN
  SELECT mean("temperature") INTO "my_database"."one_week"."average_temperature" FROM "captor_temperature_humidity" GROUP BY time(1h)
END

So with the help of retention policies and continious queries, you can downsampling the data with somehting similar as that :

keep the fine data for 1 week
aggregate per minute for data between one week and one month
aggregate per 10 minutes for data older than a month

This give those retention policies :

CREATE RETENTION POLICY "1_week" ON "my_database" DURATION 7d REPLICATION 1 DEFAULT
CREATE RETENTION POLICY "1_month" ON "my_database" DURATION 30d REPLICATION 1
CREATE RETENTION POLICY "forever" ON "my_database" DURATION INF REPLICATION 1

And those continuous query :

CREATE CONTINUOUS QUERY "cq_1min_1month" ON "my_database"
BEGIN
  SELECT mean(*) INTO "my_database"."1_month".:MEASUREMENT FROM /.*/ GROUP BY time(60s),*
END;
CREATE CONTINUOUS QUERY "cq_1min_1month" ON "my_database"
BEGIN
  SELECT mean(*) INTO "my_database"."forever".:MEASUREMENT FROM /.*/ GROUP BY time(600s),*
END;

The TICK stack

TICK stack is composed of :

Telegraf : an agent that collects metrics, and write them in an influx database.
Influx : the time series database
Chronograf : a front-end applicatino that allow you to explore your data and to create dashboards.
Kapacitor : an alerting system

It’s a complete stack to collect, manage and exploit your time series data, so it’s perfect for my goal.

The architecture

Generally there is a server where all the metrics are a centralized, and where all the monitoring application are installed. I will call it the Monitoring server.

This server will save all the metrics fron your system, and to do it there is two methods :

push : metrics are directly pushed to the monitoring server by using an agent on each server that you want to monitor.
pull : the monitoring system will query all your servers to collect the metrics.

Generally, the first solution is prefered, and it’s the one that I will put in place.

This is the architecture schema :

In red you have the monitoring server, and in grey the monitored server.

Note	With this kind of architecture, Neo4j send metrics locally, so it’s very fast.

The monitored server

Neo4j

In its Enterprise Edition, Neo4j has a monitoring system. In fact there is four ways to monitor it :

JMX : it’s a standard java functionnality that allow you to retrive some metrics values.
Graphite connector : you just have to configure your Graphana server, and Neo4j will send its metrics regulary.
Prometheus connector : same as for Graphite but for Prometheus time series database.
CSV file : Neo4j dumps all its metrics at a regular time interval

Telegraf is compatible with the Graphite protocol, so I will use it.

The configuration is really simple, just edit your neo4j.conf file and put at the end those lines :

# Setting for enabling all supported metrics.
metrics.enabled=true
# Setting for enabling all Neo4j specific metrics.
metrics.neo4j.enabled=true
# Setting for exposing metrics about transactions; number of transactions started, committed, etc.
metrics.neo4j.tx.enabled=true
# Setting for exposing metrics about the Neo4j page cache; page faults, evictions, flushes and exceptions, etc.
metrics.neo4j.pagecache.enabled=true
# Setting for exposing metrics about approximately entities are in the database; nodes, relationships, properties, etc.
metrics.neo4j.counts.enabled=true
# Setting for exposing metrics about the network usage of the HA cluster component.
metrics.neo4j.network.enabled=true
# Enable the Graphite integration. Default is 'false'.
metrics.graphite.enabled=true
# The IP and port of the Graphite server on the format <hostname or IP address>:<port number>.
# The default port number for Graphite is 2003.
metrics.graphite.server=localhost:2003
# How often to send data. Default is 3 seconds.
metrics.graphite.interval=3s
# Prefix for Neo4j metrics on Graphite server.
metrics.prefix=MyHost

Like you see, I just :

enable the metrics feature and also each familly metric.
enable the graphite integration, and configure its location, time interval and the prefix.

You don’t have to change anything, except the metrics.prefix. This value will be used as the host identifier in metrics.

Telegraf

Installation

There are many ways to install Telegraf on your system, and you can check directly on the documentation.

My prefer OS is debian, so I will show you how to do on it :

add the influxdb repository key
add the repository
perfom an update
install the package telegraf

curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
echo "deb https://repos.influxdata.com/debian jessie stable" | tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install telegraf

Configuration

All Telegraf's configuration is located in the file /etc/telegraf/telegraf.conf.

Firstly we need tell Telegraf to be able to act as a graphite server :

[[inputs.socket_listener]]
  service_address = "tcp://:2003"
  separator = "."
  data_format = "graphite"
  templates = [
    "*.neo4j.*.* host.measurement.measurement.field* name=neo4j,vlan=testing",
    "*.vm.*.* host.measurement.measurement.field* name=neo4j,vlan=testing"
  ]

I think it’s easy to understand, except for the templates part.

In fact, in graphite all metrics follow a schema like this one MyHost.neo4j.bolt.messages_done 10. So we need to tell Telegraf how to parse it to find the field measurement, the value, and tags.

The part *.neo4j.*.* is a filter. If a line match this pattern, then it will parsed with host.measurement.measurement.field*. With the example MyHost.neo4j.bolt.messages_done 10, we will have :

Tags: MyHost
Measurement: neo4j.bolt
Field: messages_done
value: 10

Moreover, at the end of each template you can see this name=neo4j,vlan=testing. It’s a list of static tags that will be added to each metric. This can be really useful if you want to monitor multiple Neo4j server (like a cluster).

Ok, now we have the metrics, but we need to push them to our centralized influx database. For this, you need to configure Telegraf like this :

[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  urls = ["http://10.0.0.12:8086"]

  ## The target database for metrics; will be created as needed.
  database = "telegraf"

  ## Name of existing retention policy to write to.  Empty string writes to
  ## the default retention policy.
  retention_policy = ""

You just have to change the urls property with yours (in my case http://10.0.0.12:8086), and optionnally the database name (by default it’s telegraf) and the rentention policy you want.

Note	In the general section of the configuration, you can configure the batch size if you want.

The monitoring server

Installation

On this server we will install : InfluxDb, Chronograf, Kapacitor and Telegraf (to monitor the monitoring system ^^)

I will follow the same process as explained on the installation of Telegraf : via apt.

curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
echo "deb https://repos.influxdata.com/debian jessie stable" | tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install telegraf influxdb chronograf kapacitor

InfluxDb

I change nothing in the default configuration of Influxdb, The only thing I will do it’s to create a database telegraf with a custom retention policy that keep the data for 3 months.

For this I will use the Influxdb CLI, sudo influx, and typing those commands :

CREATE DATABASE telegraf
USE telegraf
CREATE RETENTION POLICY "3_month" ON "monitoring" DURATION 90d REPLICATION 1

Note	If you want to check, you can type `SHOW RETENTION POLICIES` to display all RPs.

Telegraf

I have installed Telegraf just to monitor the monitoring server (CPU, network, disk, …). You just have to configure it to send all the data to the telegraf (the default value).

[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  # default is localhost with the standard port of influx
  # urls = ["http://10.0.0.12:8086"]

  ## The target database for metrics; will be created as needed.
  database = "telegraf"

Chronograf

By default, Chronograf is listening on the port 8888. So open your browser at http://MONITORING_SERVER_IP:8888/ (change MONITORING_SERVER_IP with the corresponding IP).

You can take a look at the Host List, you should see a list with two items : names of the monitored and monitoring server. Click on one, and should see something like this :

Now you can create a new dashboard for Neo4j with the following widgets :

Name Type Query

Name	Type	Query
Thread Jetty	Line Graph	`SELECT mean("threads.jetty.all") AS "mean_threads.jetty.all", mean("threads.jetty.idle") AS "mean_threads.jetty.idle" FROM "telegraf"."autogen"."neo4j.server" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(null)`
JVM memory	Stacked Graph	`SELECT pool.g1_survivor_space/1000000, pool.metaspace/1000000, pool.g1_eden_space/1000000, pool.g1_old_gen/1000000 FROM "telegraf"."autogen"."vm.memory" WHERE time > :dashboardTime:`
JVM GC Time	Line Graph	`SELECT DIFFERENCE("time.g1_young_generation") AS "mean_time.g1_young_generation", DIFFERENCE("time.g1_old_generation") AS "mean_time.g1_old_generation" FROM "telegraf"."autogen"."vm.gc" WHERE time > :dashboardTime:`
Transactions	Line Graph	`SELECT DIFFERENCE(last("started")) AS "mean_started" FROM "telegraf"."autogen"."neo4j.transaction" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(linear)`
Page cache	Line Graph	`SELECT mean("hits") AS "mean_hits", mean("page_faults") AS "mean_page_faults", mean("flushes") AS "mean_flushes", mean("evictions") AS "mean_evictions", mean("eviction_exceptions") AS "mean_eviction_exceptions" FROM "telegraf"."autogen"."neo4j.page_cache" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(null)`
JVM Threads	Line Graph	`SELECT mean("total") AS "mean_total" FROM "telegraf"."autogen"."vm.thread" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(null)`
Number of Nodes	Line Graph + Single stat	`SELECT max("node") AS "max_node" FROM "telegraf"."autogen"."neo4j.ids_in_use" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(none)`
Number of relationships	Line Graph + Single stat	`SELECT last("relationship") AS "last_relationship" FROM "telegraf"."autogen"."neo4j.ids_in_use" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(none)`
Number of Properties	Line Graph + Single stat	`SELECT last("property") AS "last_property" FROM "telegraf"."autogen"."neo4j.ids_in_use" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(none)`
Number of Relationship Types	Line Graph + Single stat	`SELECT last("relationship_type") AS "last_relationship_type" FROM "telegraf"."autogen"."neo4j.ids_in_use" WHERE time > :dashboardTime: GROUP BY time(:interval:) FILL(none)`
Opened Transactions	Line Graph + Single stat	`SELECT started - committed - rollbacks FROM "telegraf"."autogen"."neo4j.transaction" WHERE time > :dashboardTime:`

Thread Jetty

Line Graph

SELECT mean("threads.jetty.all") AS "mean_threads.jetty.all",
       mean("threads.jetty.idle") AS "mean_threads.jetty.idle"
FROM "telegraf"."autogen"."neo4j.server"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(null)

JVM memory

Stacked Graph

SELECT pool.g1_survivor_space/1000000,
       pool.metaspace/1000000,
       pool.g1_eden_space/1000000,
       pool.g1_old_gen/1000000
FROM "telegraf"."autogen"."vm.memory"
WHERE time > :dashboardTime:

JVM GC Time

Line Graph

SELECT DIFFERENCE("time.g1_young_generation") AS "mean_time.g1_young_generation",
       DIFFERENCE("time.g1_old_generation") AS "mean_time.g1_old_generation"
FROM "telegraf"."autogen"."vm.gc"
WHERE time > :dashboardTime:

Transactions

Line Graph

SELECT DIFFERENCE(last("started")) AS "mean_started"
FROM "telegraf"."autogen"."neo4j.transaction"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(linear)

Page cache

Line Graph

SELECT mean("hits") AS "mean_hits",
       mean("page_faults") AS "mean_page_faults",
       mean("flushes") AS "mean_flushes",
       mean("evictions") AS "mean_evictions",
       mean("eviction_exceptions") AS "mean_eviction_exceptions"
FROM "telegraf"."autogen"."neo4j.page_cache"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(null)

JVM Threads

Line Graph

SELECT mean("total") AS "mean_total"
FROM "telegraf"."autogen"."vm.thread"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(null)

Number of Nodes

Line Graph + Single stat

SELECT max("node") AS "max_node"
FROM "telegraf"."autogen"."neo4j.ids_in_use"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(none)

Number of relationships

Line Graph + Single stat

SELECT last("relationship") AS "last_relationship"
FROM "telegraf"."autogen"."neo4j.ids_in_use"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(none)

Number of Properties

Line Graph + Single stat

SELECT last("property") AS "last_property"
FROM "telegraf"."autogen"."neo4j.ids_in_use"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(none)

Number of Relationship Types

Line Graph + Single stat

SELECT last("relationship_type") AS "last_relationship_type"
FROM "telegraf"."autogen"."neo4j.ids_in_use"
WHERE time > :dashboardTime:
GROUP BY time(:interval:) FILL(none)

Opened Transactions

Line Graph + Single stat

SELECT started  - committed - rollbacks
FROM "telegraf"."autogen"."neo4j.transaction"
WHERE time > :dashboardTime:

And the result :

Kapacitor

Kapacitor is the alerting systen of the stack. You can create some rules for Kapacitor directly in Chronograf :

This alert sends a message on slack as soon as there less than 20% of free space on my disk :)

Conclusion

You see it’s really easy to monitor your infrastructure and Neo4j servers with the TICK stack. But there are some lacks :

Telegraf doesn’t have a JMX plugin
It’s not possible to make a generic continious query that downsample the data with the same field name (you will have an aggregation prefix). This is really annoying when you want to have only one query to make your dashboard (and not one per retention). See this link for more details

I would like to make the same kind of article but this time with Prometheus and Grafana. So if you are interested, please leave a comment, it will motivate me to write it :)

How to use Sigmajs to display your graph ?

Wed, 25 Apr 2018 00:00:00 +0200

Introduction

In this article, I will show how to display and customize a graph in a browser with the help of SigmaJS, a JS library dedicated to graph drawing delivered under the MIT licence.

Initially, SigmaJS has been developed for Gephi to export a graph on the web. Today, sigmajs is a stable, full features and highly configurable graph library. Moreover, the small cherry on the cake is its compatibility with touch screens.

Installation

How-to

There are four steps to follow :

Step 1 : Install sigmajs binaries

sigmajs can be found on :

NPM : you just have to type : npm sigma --save-dev
Github : download the build.zip from a release like this

Step 2 : Import the JavaScript library in your web page

The minimal script you have to load is : <script src="./build/sigma.min.js"></script>.

Personally, I prefer to import all the source files, it helps to lighten the library by removing the unused source files (ex: I don’t need SVG & Webgl renderers if I use canvas).

Step 3 : Create an HTML container

You have to create an HTML element, for example a div with an id like this : <div id='sigma-container'></div>.

Moreover, you also need to give it an height and width, otherwise it will have a size of 0 pixel. This is easily done with some CSS : #sigma-container { width:100%, height:100%}

Step 4: Initialize sigmajs

To initialize Sigmajs, you need to call the function sigma with those arguments :

A render : it contains the HTML element that sigmajs will use to display the graph, plus the type of renderer (SVG, canvas or WebGL)
sigmajs’s settings : this object will override the default settings, so you can pass an empty object for now. The complete settings list is available here

At the end, you should have something like that :

<html>
  <head>
    <title>SigmaJS example</title>
    <script src="./build/sigma.min.js"></script>
    <style>
      html { height:100%;}
      body {height: 100%;}
      #sigma-container { width:100%; height:100%; background-color:#E1E1E1}
    </style>
  </head>
  <body>
    <div id='sigma-container'></div>
    <script>
      // Initialize sigma:
      var s = new sigma(
        {
           renderer: {
             container: document.getElementById('sigma-container'),
             type: 'canvas'
           },
           settings: {}
         }
       );
    </script>
  </body>
</html>

How to choose between SVG, Canvas and Webgl ?

SigmaJS can used one this technology to display your graph : svg, canvas & webgl.

SVG is a markup language, directly interpreted by your browser, so it’s easy to customize nodes & edges style, you just have to add some CSS rules. But when you apply a layout algorithm on the graph, you continuously change the DOM of the SVG, and if this DOM is heavy, the result is slow and can freeze your browser.

Canvas is an HTML5 element compatible with all modern browsers. It defines an area on your web page where you can draw some pixels by using a JavaScript API. The JavaScript code is directly interpreted by your browser and it decides what and when to render. Due to this, Canvas scales more than the SVG, but if you want to change the color of a shape, you need a way to specify it in the code.

On the other hand, there is WebGL, compatible with almost all newest browsers (even if the API is not stable). As in Canvas, you need to write some JavaScript code for rendering your visualisation, but this time, its computation is delegated to OpenGL so to your GPU.

To resume :

	SVG	Canvas	WebGL
Browser compatibility	ALL (even old ones)	ALL	Modern browsers
Render phases	SVG creation > Browser parsing > Browser rendering	JS code > Browser rendering	JS Code > OpenGL rendering
Performance to display a large graph (more than 1000 elements)	Slow	Normal	Performant

SVG

Canvas

WebGL

Browser compatibility

ALL (even old ones)

ALL

Modern browsers

Render phases

SVG creation > Browser parsing > Browser rendering

JS code > Browser rendering

JS Code > OpenGL rendering

Performance
to display a large graph (more than 1000 elements)

Slow

Normal

Performant

The choice will depend on your needs, and also of the SigmaJS plugins you want to use (plugins are not compatible with every format).

I suggest you to choose Canvas or Webgl if you want to really display a big graph.

The SigmaJS graph structure

Definition

Now that sigma is initialized, we need to give it a graph. It structure is simple : an object with an array of nodes and an array of edges.

Table 1. Node definition
Field	Type	Required	Description
`id`	Any	Yes	Node identifier, must be unique across all the node
`label`	String	No
`x` & `y`	Float	Yes	Position of the node in 2D. Can be initialize with `Math.random()`.
`size`	Float	No (0)	Size of the node that is use to render it.
`color`	Color RGB	No (`defaultNodeColor`)	Color used to display the node.

Table 2. Edge definition
Field	Type	Required	Description
`id`	Any	Yes	Edge identifier, must be unique across all the edge.
`source`	Any	Yes	Starting node’s identifier of the edge.
`target`	Any	Yes	Ending node’s identifier of the edge.
`type`	`line`, `curve`, `arrow` or `curvedArrow`	No (line)	Edge type for its render. The choice will depend on if you want to display an oriented graph. If so, you need an arrow type. And if you want to display a high number of edges, it’s faster to use a line type (ie. line or arrow).
`size`	Float	No (`minEdgeSize`)	Size of the edge that is use to render it.
`color`	Color RGB	No (`defaultEdgeColor`)	Color used to display the edge.

Example

var graph = {
  nodes: [
    { id: "n0", label: "A node", x: 0, y: 0, size: 3, color: '#008cc2' },
    { id: "n1", label: "Another node", x: 3, y: 1, size: 2, color: '#008cc2' },
    { id: "n2", label: "And a last one", x: 1, y: 3, size: 1, color: '#E57821' }
  ],
  edges: [
    { id: "e0", source: "n0", target: "n1", color: '#282c34', type:'line', size:0.5 },
    { id: "e1", source: "n1", target: "n2", color: '#282c34', type:'curve', size:1},
    { id: "e2", source: "n2", target: "n0", color: '#FF0000', type:'line', size:2}
  ]
}

Loading graph in sigma

Sigma has a complete API to manage its graph data. I let you see the API documentation.

To load a graph in sigma, you just have to call the method read on the sigma graph instance : s.graph.read(graph)

Once it’s done, we need to tell sigma to draw the graph by calling its refresh function : s.refresh()

Live example

Graph layout algorithm

Definition

What is hard in displaying a graph is to rapidly display it in such a way that we can see all nodes and their edges without overlaps (in fact less as possible). To do it, we need an algorithm that computes the position of each nodes, and the most known for that are the force-directed algorithms.

The principle is simple, you need to consider two forces :

Repulsive	Attractive
Nodes repulse each others. You can consider nodes like particle with the same electric charge.	Two nodes with an edge, attract themselves. You can consider an edge as a spring

Repulsive

Attractive

Nodes repulse each others. You can consider nodes like particle with the same electric charge.

Two nodes with an edge, attract themselves. You can consider an edge as a spring

Then you run an algorithm that compute on each iteration, the sum of the applied forces on each node, and move them in consequence. After a number of iteration, you will see that graph is in a stable state.

Force-Atlas2

SigmaJS include (as a plugin) a forced-directed algorithm called Force-Atlas2.

To use it, you need to :

Step 1 : import the plugin files

<script src="./build/plugins/sigma.layout.forceAtlas2/supervisor.js"></script>
<script src="./build/plugins/sigma.layout.forceAtlas2/worker.js"></script>

Step 2 : Run it

Now that the plugin is loaded, we can directly call it on the sigma instance : s.startForceAtlas2(); This creates a web worker where all the algorithm iterations will be calculated.

Step 3 (al) : Stop it

The algorithm won’t stop by itself, so I recommend you to stop it after a predefined duration (10 seconds in my example) : window.setTimeout(function() {s.killForceAtlas2()}, 10000);

Live example

Plugins

SigmaJS has a lot of plugins, you can see the list here. I will not show you all of them, so I have done a list of my most used plugins.

Edge labels

This plugin allows you to add a label on each edge. I mainly use it to display the Neo4j’s relationship type.

To use it :

Import the needed script (in my case settings.js, sigma.canvas.edges.labels.def.js & sigma.canvas.edges.labels.curvedArrow.js)
Add a label property on yours edges

Edge parallelEdges

If you want parallel edges (ie. to have multiple relationship between two nodes), it’s the plugin you must have.

To use it :

Import the needed script :
- utils.js
- sigma.canvas.edges.curvedArrow.js & sigma.canvas.edgehovers.curvedArrow.js if you have a directed graph.
- sigma.canvas.edgehovers.curve.js & sigma.canvas.edges.curve.js if you want the an undirected graph.
- sigma.canvas.edges.labels.curve.js if you have enabled label on edges
Add a property count that represent the index of the edge in the set of parallel edges. Inversely proportional to the amplitude of the vertex of the edge curve.

Relative size

This plugin is really useful when you want to see which node is most connected. The size of the node depends of its degree, ie. its number of in-going & outgoing edges.

Sigma and Neo4j

Now that you know how to use Sigma, the next step is to build a graph visualisation from Neo4j.

To do this there is two points :

How to query Neo4j in your browser ?
From a query result, how to build a sigma graph structure

I will not explain the first point, Michael has already done this part in this excellent post.

So let see the second one !

Create a graph structure from a Neo4j query

Result of a query is a collection of tuple, ie. composed of rows where each row has some columns. Moreover, each cell is typed, and to display a graph we only want node, relationship and path.

To create our data structure, we need to iterate over rows, then over columns and finnaly check the type. If it’s a node or relationship, we can add it to our sigma graph structure (if it’s not already present).

And what about path ? A path in Neo4j driver types, is an array of segment where each segmet is composed of :

a starting node
a relationship
an ending node

So if we have a path, we need also to iterate over it to add starting & ending node, plus the relationship.

But wait, a Neo4j node (resp. relationship) is not a Sigma node (resp. relationship), so we also need to convert them.

If you code it, finally you should have something like this :

let graph = { nodes:[], edges:[]} ;
this.driver.session().run("MATCH (n)-[r]->(m) RETURN n,r,m LIMIT $limit", {limit:50}).then(
  (result) => {
    // for each rows
    result.records.forEach( record => {
      // for each column
      record.forEach( ( value, key ) => {
        // if it's a node
        if ( value && value.hasOwnProperty( 'labels' ) ) {
          graph.nodes.push(convertionToSigmaNode(value));
        }
        // if it's an edge
        if ( value && value.hasOwnProperty( 'type' ) ) {
          graph.edges.push(convertionToSigmaEdge(value));
        }
        // if it's a path
        if ( value && value.hasOwnProperty( 'segments' ) ) {
          value.segments.forEach( ( seg ) => {
            // add starting & ending nodes + relationship
            graph.nodes.push(convertionToSigmaNode(seg.start));
            graph.nodes.push(convertionToSigmaNode(seg.end));
            graph.edges.push(convertionToSigmaEdge(seg.rel));
          });
        }
      });
    })
  })

And here we go, you have everything to display your graph from a Cypher query with SigmaJS !

It’s a little borring, no ? All this code just to display a graph… And what if I tell you that I have already made the work for you ?

Neo4j + Sigma = NeoSig

To avoid you the complexity of doing all the above work, I have created a library for that : NeoSig.

To use it, you need to import the library + the Neo4j driver :

<script src="https://cdn.jsdelivr.net/npm/neo4j-driver@1.6.0"></script>
<script src="https://cdn.jsdelivr.net/npm/neosig@1.2.2/docs/neosig-1.2.2.js"></script>

Note	The library embed SigmaJS (with some customs code) but not the Neo4j driver.

Then you can call the function Neo4jGraph(neo4jConfig, neo4jStyle, query, queryParams), it returns a promise with the sigma graph object.

Neo4jGraph(neo4jConfig, neo4jStyle, 'MATCH (n)-[r]->(m) RETURN n,r,m LIMIT $limit', {limit:20}).then( function(graph) {
  s.graph.read(graph);
  // enable drag'n'drop
  sigma.plugins.dragNodes(s, s.renderers[0]);
  // start layout
  s.startForceAtlas2();
  setTimeout(() => { s.stopForceAtlas2() }, Math.log(graph.nodes.length*graph.edges.length)*1000);
});

Where neo4jConfig is an object with :

const neoConfig = {
  url:'bolt://localhost:7867',
  user: 'neo4j',
  password: 'letmein',
  driver : {
    // all the driver configuration (optional)
  }
}

And neo4jStyle is :

const neoStyle = {
  labels: { // Map of label
    Person : {
      label: 'name', // The node's property to display as label
      color: '#654321', // Color of the node
      size: 10, // Size of the node
      icon: { // icon object
        name: 'f007', // Fontawesome unicode
        color: '#FFF', // Color of the font
        scale: 1.0 // Scale ratio
      }
    },
    Movie : {
      label: 'title',
      color: '#123456',
      size: 10,
      icon: {
        name: 'f008',
        color: '#FFF',
        scale: 1.0
      }
    }
  },
  edges: { // Map of edges
    ACTED_IN: { // Name of the relationship type
      label: 'roles',
      //color: '#202020',
      // size: 2
    }
  }
};

Note	By default, a node is black, with a size of 5, and the label is its Neo4j’ID ; a relationship is black with a size of 1, and the label is its Neo4j type.

And the final result is :

You can see the code here :

Now have fun !

A big thanks to Jacomyal, Yomguithereal and Astik for the help !