Monday, October 8, 2012

Crazy Social Analytics for C# Nuget

@YaronNaveh

I'm happy to announce that C# nuget projects get some GitMoon love! This means you can get crazy social analytics about nuget projects. Check out your favorite ones:

SignalR
ServiceStack
Mono.Cecil
Facebook
Hammock
sqlite-net

Or check out famous head to head comparisons:

SignalR vs. ServiceStack
Mono.Cecil vs. LibGit2Sharp
sqlite-net vs. FluentMongo
TweetSharp vs. Facebook
Hammock vs. EasyHttp





@YaronNaveh

What's next? get this blog rss updates or register for mail updates!

Thursday, October 4, 2012

10 Caveats Neo4j users should be familiar with

@YaronNaveh

UPDATE: Michael Hunger from neo4j responds to some of my items in a comment.

Recently I used the Neo4j graph database in GitMoon. I have gathered some of the tricky things I learned the hard way and I recommend any Neo4j user to take a look.

1. Execution guard
By default queries can run forever. This means that if you have accidently (or by purpose) sent the server a long running query with many nested relationships, your CPU may be busy for a while. The solution is to configure neo to terminate all queries longer than some threshold. Here's how you do this:

in neo4j.properties add this:

execution_guard_enabled=true

Then in neo4j-server.properties add this:

org.neo4j.server.webserver.limit.executiontime=20000

where the limit value is in milliseconds, so the above will terminate each query that runs over 20 seconds. Your CPU will thank you for it!


2. ID (may) be volatile
Each node has a unique ID assigned to it by neo. so in your cypher you could do something like:

START n=node(1020) RETURN n

START n=node(*) where ID(n)=1020 return n

where both cyphers will return the same node.

Early on I was tempted to use this ID in urls of my app:

/projects/1020/users


This was very convinient since I did not have a numeric natual key for nodes and I did not want the hassle of encoding strings in urls.

Bad idea. IDs are volatile. In theory, when you restart the db all nodes may be assigned with different IDs. IDs of deleted nodes may be reused for new nodes. In practice, I have not seen this happen, and I believe that with the current neo versions this will never happen. However you should not take it as guaranteed and should always come up with your own unique way to identify nodes.

3. ORDER BY lower case
There is no build in function that allows you to return results ordered by some field in lower case. You have to maintain a shadow field with the lower case values. For example:

RETURN n.name ORDER BY n.name_lower

4. Random rows
There is no built in mechanism to return a random row.

The easiest way is to use a two-phase randomization - first select the COUNT of available rows, then SKIP rows until you get to that row:

START n=node(*)
WHERE n.type='project'
RETURN count(*)

// result is 1000
// now in your app code you make a draw and the random number is 512

START n=node(*)
WHERE n.type='project'
RETURN n
SKIP 512
LIMIT 1

An alternative is to use statistical randomization:

START n=node(*)
WHERE n.type='project' AND ID(n)%20=0
RETURN n
LIMIT 1

Where 20 is number you generated in your code. Of course this will never be fully randomized, and also requires some knowledge on the values distribution, but for many cases this may be good enough.


5. Use the WITH clause when cypher has expensive conditions
Take a look at this cypher:

START n=node(...), u=node(...)
MATCH p = shortestPath( n<-[*..5]-u) WHERE n.forks>20 AND length(p)>2
RETURN n, u, p

Here we will calculate the shortest path for all noes. This is a cpu intensive operation. How about separating concerns like this:

START n=node(...), u=node(...)
WHERE n.forks>20 AND length(p)>2
WITH n as n, u as u
MATCH p = shortestPath( n<-[*..5]-u ) WHERE length(p)>2
RETURN n, u, p

now the path is only calculated on relevant nodes which is much cheaper.


6. Arbitrary depth is evil
Always strive to limit the max depth of queries you perform. Each depth level increases the query complexity:

...
MATCH (n)<-[depends_on*0..4]-(x)
...

7. Safe shutdown on windows
When you run Neo4j on windows in interactive mode (e.g. not a service) do not close the console with the x button. Instead, always use CTRL+C and then wait a few seconds until the db is safety closed and the window disappears. If by mistake you did not safely close it then the next start will be slower (can take a few minutes or more) since neo will do recovery. In that case the log (see #8) will show this message:

INFO: Non clean shutdown detected on log [C:\Program Files (x86)\neo4j-community-1.8.M03\data\graph.db\index\lucene.log.1]. Recovery started ...

8. The log is your best friend
When crap hits the fan always turn out to /data/log. Especially if neo does not start you may find out that you have misconfigured some setting or recovery has started (see #7)

9. Prevent cypher injection
Take a look at this code:

"START n=node(*) WHERE n='"+search+"' RETURN n"

if "search" comes from an interactive user then you can imagine what kind of injections are possible. The correct way is to use cypher parameters which any driver should expose an api for. If you use the awesome node-neo4j api by aseemk you could do it like this:

qry = "START n=node(*) WHERE n={search} RETURN n"
db.query qry, {search: "async"}

10. Where to get help
The Neo4j Google group or the community github project are very friendly and responsive.

@YaronNaveh

What's next? get this blog rss updates or register for mail updates!

Wednesday, October 3, 2012

How I Built GitMoon

@YaronNaveh

I got some queries on how I built GitMoon so I decided to come up with this list in BestVendor:

How I Built a Viral Node.js App in Just One Weekend

You can read there about the technology and tool choices I've made and why. Got a cool cover image too. Check it out in BestVendor.


@YaronNaveh

What's next? get this blog rss updates or register for mail updates!

Tuesday, October 2, 2012

MongoDB and Redis go head to head with Node.js social analytics

@YaronNaveh

And not just mongo vs. redis but also jade vs. ejs, azure vs. jitsu and anything else you want! All in today's GitMoon new rollout. Here are some of the amazing geo-social visualizations you get when you compare two projects:






More visualizations are in GitMoon.

Check out some of the popular head-2-head comparisons:

@YaronNaveh

What's next? get this blog rss updates or register for mail updates!