Tuesday, September 18, 2012

How I build the EventStore on Linux.

Yesterday Greg Young's EventStore has its public launch so I couldn't resist and had to check it out. Code's open-sourced on Github, the only problem was that... all my PCs run on Linux and I've never worked with .NET before. Fortunately, EventStore is cross-platform so it turned out not to be hard at all.

First step was obvious (even for a JVM guy like me):
apt-get install mono
Documentation of the project said that I should be using xbuild to build it from source so (with some help from apt-get which is good at finding the packages I'm looking for):
apt-get install mono-xbuild
et voila! Motivated by first easy victory I run a quick
cd src
xbuild EventStore/EventStore.sln
and... the build crashed. That was not unexpected, and examination of error report quickly led me to another apt-get install:
apt-get install mono-dmcs

The build went on but crashed soon after, with some cryptic messages about missing assemblies. Not knowing what to do I just checked what other mono-* packages are available - mono-devel looked promising so yet another package installation and ... a few seconds later I had the EventStore binaries ready! Yay!

Running it is a no brainer and you can see the nice monitoring & management UI at http://127.0.0.1:2113 .

Summing up: except for the fact that I didn't have any mono stuff installed (which you probably need if you're doing any development using it) EventStore was dead simple (and fast!) to build & run. Good job guys!

Can't wait to play with it some more (and maybe port it to JVM?).

Wednesday, May 23, 2012

33rd Degree conference - day 3

Looking at the agenda, I knew exactly which talks I wanted to attend on last day of the conference.

Integrating JVM languages 

by Venkat Subramaniam

Once again, Venkat gave a very entertaining talk, this time about the integration of various languages running on Java Virtual Machine (Java,Groovy,Scala,JRuby) and the problems one can get into when trying to do it. In general, the integration is quite smooth, but there are some edge cases when the solution is not really trivial. From what I've seen Groovy seems to have the most smooth integration, but things don't look bad with Scala or JRuby. While I didn't gained any immediately enlightening knowledge I got a good introduction into the subject and now I know that it's not that hard to do and knowing a few languages that run on JVM one can pick the one most suitable for the problem at hand and don't suffer too much when integrating it with the rest of the application. That's cool, especially that after Venkat's talk about Scala on the 2nd day our manager gave us green light to use it in our project when we learn the language a bit.

MongoDB : Scaling web applications

by Ken Sipe

This one was another eye-opener, not really for me - because I investigated the topic a bit before - but for some of my colleagues for sure. My boss got quite excited about it as he recognized immediately a part of our system that could be easily implemented using MongoDB as storage (and which has been in development for about 2 years now and still very far from being perfect or even production-ready). Ken shown us an example of real web site with Mongo storing it's data. He offered a few useful bits of advice, like modelling the documents based on the specific use cases, so that there's no need to fetch multiple documents for a single thing. He mentioned ways to scale Mongo and to fine tune its performance. Talk was very informative and as a result I was asked to create a proof-of-concept for replacing the mentioned part of our system with Mongo-based implementation (more on that in another post).

The Three Laws of Test Driven Development

by Uncle Bob

I guess I was not the only one really looking forward to this talk. That was the very first time I could meet Uncle Bob live (and I regret I didn't have any of his books for him to sign). As I have read a lot of Uncle's posts and watched a few presentations as well I knew what I could expect, but even then his live performance was an awesome thing to watch, listen & learn from. I guess that almost every phrase could be used as a really good quote.
Introduction of TDD began with some sad numbers, as only about 5% of the industry is doing TDD. This number is however improving rapidly, and in a few years we can expect it to be much better.
Uncle Bob dealt quickly with the myth that developers don't have time to write tests/do TDD. As he said, it's childish to think that we have to make a mess to be quick, even if this mess is going to slow us down in the longer run. When mess in the code base grows, more effort is put into moving it from place to place instead of developing new features. That's not the situation we want to have in our projects as it often ends with a redesign of the whole systems, made by the experienced guys (the "Tiger" team)  - the same who made the whole mess in the first place. Of course, team doing the redesign has to be quick to catch up with the feature set of the system they are going to replace. And to be quick, they need to make some mess of course... See the pattern?
Instead of falling into the trap of the big redesign, we should rather concentrate on clearing thes mess one little bit at a time. That's of course the famous Boy Scout Rule that Uncle Bob often mentions. Each time we touch our code, we should make some improvements. Even small ones. Even very small ones. And of course, when writing new code, we should avoid creating mess. Don't even dare to think to yourself: "I'll fix it later". Later does not exist. We all know it.
"Bad mgmt can be suffered through but the bad code is the anchor that is bringing the company down"
Unfortunately, we are often reluctant to clean up the mess we find in the code we read. We think if we clean it and break it in the process we'll be responsible for it from now on. Or we might be afraid of touching it because we've got no idea what will happen (break) if we do. As Uncle Bob said:


"We have created the monster and now the monster rules us"  
That's unfortunately sad reality for many (most?) software projects. As professionals we should have control over our creations. Our software can't just work (any idiot can do that). It must be continuously flexible, easy to maintain & modify at any point in its life.
"You don't want to be good at debugging. It's not a skill to be desired"
"You don't want to spend time debugging. You want to spend time not bugging"
Debugging is hard. And takes a lot of time. The good news is - we can avoid it. Maybe not 100% but most of the time. How? Use the Force, Luke. I mean, use TDD. With TDD done well (that's the trick - you have to do it well otherwise your code will suck as much as it would if you haven't. But to do it well you have to practice it!).
Additional benefits we get when doing TDD are the formal design docs (yup, your tests), nice & decoupled code. And the possibility to refactor with fear.

Uncle Bob also said more than a few words about fast tests. The suite of unit tests of FitNesse (1892 unit tests) is running on his Mac in under 50s. That's pretty awesome and shows how short feedback loop unit tests give. In our project a suite of 700 tests runs for a few minutes (~3) - most of them are functional tests and it's not very useful for doing TDD. And we have to change our mindset regarding such tests - there's no reason for acceptance tests to be slow. They even can't be slow. It's our job to make them blazing fast. There's no excuse for it.
"If you want control over your code you'd better get yourself a suite of tests"
Another important thing is the trust you are willing to put into your test suite. It's like jumping out of a plane with a parachute - you'd better now have any holes in it. Software can be extremely easily broken, just flip a single bit and it's done (unlike your brain which can loose a couple of thousands of cells in a single drinking session and not be affected by this loss). You need a really good suite of tests, one that you can trust your life to. When it runs and passes, you should be convinced that your software works. If you're not and have to check it manually - what good are your tests for?

And for the end, a few Questions & Answers, which made the whole audience laugh:
Q: But we have a testing department, I don't have to write tests!
A: Screw you!
Q: What about refactorings?
A: It's a good idea to refactor your code. Next question"
Q: How you start TDD culture in company that doesn't have it?
A: You start shooting people at random. Don't get your hopes high regarding developing a TDD culture in your company. And remember - you can always change your organization or change your organization.


Code Craft

by Nat Schutta

Next talk was very nice, going back to the basics - good code, bad code and how to improve it (tip: write less of it, less is more). While nothing earth-shattering, it was a real pleasure to listen to a good speaker, got reminded of many things that we might not think about too often. We got reminded to simplify the code, to talk to each other about it (most of us prefers to write code, not to read it, and it's critical to read code) - but we need to remember that we have to be critical of the code, not the people.
Nat also warned us against thinking too much about code reuse - it should be a byproduct, not a rationale (but too often I participate in meetings when we talk about reusing the code that is not even written yet!).
Nat also touched the topic of tests - we should treat them the same way we treat production code and if a developer is not writing them he's not really doing his job.
And the last advice was where to start improving code - we should let the feat guide us. We should start where the code is really scary. That's where the biggest mess is, just begging us to get it cleaned up.

Demanding Professionalism

by Uncle Bob

I was really sad when this last keynote started, as I knew that I will have to leave early to catch my train back home.
The talk started with a question what is the software crisis? As Uncle Bob put it:
It's so hard to get the damn software done right
One of the major problems is the lack of professionalism in our industry. We see bugs as normal thing that happens rather than "OMG A BUG!". And we live in an illusion that there's no demand for professionalism in software. But thinking about it - how else are we going to deliver features fast, if not by being professional and carefully crafting software?
There's an analogy often made, that building software is a bit like creating buildings. But in fact it's not - a normal architect creates the plans for the building because it's damn expensive to make changes after you started to lay the bricks down. In software, it's cheap so there's no point in having big design up-front. We've got to remember that we're not the masons - our code is not the product, it's the specification the compiler uses to build the product. We're more like architects, preparing plans for the compiler to follow.
Unfortunately, changing your design often works only if the build step is cheap. If we make it expensive (for example long - with slow builds!) the cost of building increases and we have to do more up-front planning. Therefore it's our duty as developers to keep the builds short, so that they don't get in our way when you want to change our design. We cannot submit to the idea that build just takes that much time...

... and unfortunately this is where 33rd degree conference ended for me. I left the presentation and went back home, with 'REST in practice' book I bought on the O'Reilly stand (James Lewis recommended this book and I must admit that it's a really good read).

I'm looking forward to coming back for the next edition of 33rd degree, if it will be possible. In the meantime, I'll probably join Confitura conference in Warsaw, in June.

Monday, May 14, 2012

33rd Degree Conference - Day 2

Okay, a long time since I wrote the first part of my 33rd degree journal and even a longer time since the conference itself. Still, I will try to write down what I learned, just to refresh my own memory.

Agile, Lean and Startups Practice & Principles to create ultimate value creation machine

by Barry O'Reilly

Starting early (not really, but I had some late night discussions with my colleagues) in the morning with a non-technical talk. I'm (unfortunately?) from the corporate world rather than from startup environment so I picked this talk to get a better understanding of how startups work (maybe one day I'll need this knowledge, who knows?). Barry started by introducing (?) waterfall, agile, lean and startup and highlighting some of the differences. He presented how the Deming cycle applies to startups and gave a handful of useful tips:
Ask: What's the smallest thing I can build that will give me a lot of feedback?
"Validate that what you're trying to build is the right thing to build.
In startup you don't have infinite time to do useless things as you might have in corporate world (oh yeah, I know something about it...).
 You need good understanding of the potential users to design a system suiting their needs  
Street surveys: help the team interact with end users to understand their motivations  
Developers don't just write code now, they have to understand their users
Get out of the building. Go to a coffee shop and talk to someone: Hey, I'm building something... 
Barry suggested doing street surveys - the very thought of it scares the s**t out of me but I guess that would be a scary & insightful exercise, as it would take me as far from my comfort zone as possible. That's how we learn after all. There's a few things to remember when doing it:
"Always ask Who? Why? How? What? and that kind of questions" 
"Ask demographic questions first to check if you're talking to the right person"
He also presented a few useful tools that can help people to understand their business (Business Model Canvas, personas, User value stream maps) and make it work better. And he stressed how important for startups is learning how are you doing (various metrics) & learning from mistakes:
Understanding what your customer wants is pure gold
Measure - data is the new oil
Don't waste your failures - don't celebrate failures but what you learned from them

I'll finish here as you can find the slideshow located here. I think that the talk, while concentrating on startups, has a lot of value for people coming from corporate world. I think that many of the patterns/tips applied to such environment would only make it better.

Micro Services - Java, the Unix Way

by James Lewis

Just noticed today - the slides are finally available! Yay!

The title of the talk intrigued me when I first saw it some time before the conference and I knew I'll participate. I think that was the best talk I've been to in the conference - maybe not the best show but I think most valuable from my point of view.
James shared with us the story of a project he participated in - they've build a huge application out of tiny, independent services (divided along the business capabilities of the system - one or more service per capability), implementing using various technologies (individual teams was free to choose whatever technology worked for them), all communicating in the same way - via HTTP.

"Micro service - small enough to fit in your head. Small enough so you can throw it away"
The services were small enough so that they could be discarded & rimplemented at any time. In fact, rewritting them would be even faster & cheaper than applying bigger modifications to them, should the need arise ("Rewrite over Maintain").

"Organize your teams around your business capabilities"
Such divide & conquer approach allowed them to share the workload among many teams, not conflicting with one another (and remember the Conway's law!). Otherwise, they wouldn't be able to meet tight deadlines they had. Additional benefit of this architecture:

  • can adjust the architecture of individual services according to the specific requirements (like huge data sets or low latency)
  • adhering to SRP (each application doing only one thing)
"If the class on the screen is bigger than by head it's too big"
  • individual services fits in one's head (simplicity!)

"Be of the web, not behind the web"
The services were talking to each other via REST APIs, using regular request-response calls and publishing ATOM-based event logs.
The services were distributes as standalone JAR files (with Unix rc.d scripts), along with their configuration. To prevent developers from seeing the similarities in the code & extracting tons of common code, they were put in different VCS roots (with some library/infrastructure code being shared as separate artifacts, maintained in a open-source-like way).
Later, James explained how such system scales (see the presentation), what tools have they used. He also stressed that the uniform interface of the services is critical and how to deal with eventual consistency in this case. He didn't manage to talk about versioning but he was asked about it after the talk and explained that they were using automated tests as contracts between the systems - as long as the tests were passing the applications could communicate with their collaborators. I find this a wonderfully simple (conceptually) way to ensure that systems can change & still work together.

The presentation was a great example how huge things can built from small parts, using well-known, open technologies, without a need to spend years to develop an internal framework that nobody will be familiar with. Additionally, I found it interesting to see the same concepts Greg Young was talking about during his DDD/CQRS training were coming out in this talk (eventual consistency, bounded contexts, simplicity!). I guess great minds think alike and those techniques are not something new, that people could be afraid to use or have troubles with understanding.

After the talk, I bought the 'REST in Practice' book that James recommended. I'm reading it right now and I see it will make me smarter, even if we're not doing REST in our projects at work.

Concurrency Without Pain in Pure Java

by Venkat Subramaniam

I arrived really late for this talk (stayed a bit longer near the TouK room to listen & talk to James Lewis after his presentation). As for every talk Venkat gave, the room was suffering from a heavy participants overflow so I wasn't really able to see/hear everything. I only saw Venkat showing how to use STM (using Refs) in Java (and Clojure, IIRC). While entertaining (as always!) this talk didn't bring much value to me. Maybe if I had participated from the beginning.

What's new in Groovy 2.0?

by Guillaume Laforge

This was one of those presentation I wasn't paying 100% of my attention to. I don't know Groovy too well so learning what's new wasn't of much use to me but what I've seen looks like a really nice language. I'm a bit surprised that Groovy developers are going in the direction of allowing Groovy code to be statically compiled - I know performance might be an issue (and it certainly is for Groovy 1.x) but that's the tradeoff you make when using a dynamic language - sacrificing performance & compiler checks for reduced development friction & more expressiveness. But anyway, I like the dot- and parentheses-less code you can write with Groovy.

And of course - attending this presentation enabled to get me a seat for the next talk:

Scala for the Intrigued

by Venkat Subramaniam

This one I was really looking for. First - you know the show will be great if Venkat is the presenter (no free seats 15 minutes before the presentation start). Second - from what I have seen so far, Scala looked like a nice language. Maybe not entirely familar, but kind of luring me to give it a try. Therefore I wanted to see what cool features of the language Venkat will show us.
First class support for XML, no primitives, implicits are only some of the goodies you get if you run away from Java language to embrace Scala. As Venkat said - Scala is Java with its flaws fixed. An expressive, low-ceremony language that doesn't need an IDE that will produce hundreds of lines of code for getters, setters etc for you:
"Java: I just declare fields and my IDE magically vomit the rest of the getters, setters and constructors"
Additionally, Scala has traits (~ mixins) - something I was often missing when hacking Java code.
Of course, the show was as good as expected and this presentation was a good ending for the second day of the presenatation (at least for me). Scala seems to be really promising and we even get support from our manager to learn Scala and use it (as a proof of concept initially) at work! Good news indeed.

Thursday, April 12, 2012

33rd Degree Conference - day 1

Thanks to my company, I was able to attend 33rd Degree conference in Kraków a few weeks ago. It was 3 days of great fun & enormous profit (my knowledge growth). Looking at the agenda I didn't suspect it will be such an awesome event.
Let me summarize the talks I attended to and maybe some insights I gained.

Twitter : From Ruby on Rails to the JVM by Raffi Krikorian

Opening keynote brought some impressive numbers about Twitter: each tweet causes emails being sent, Blackberry/iPhone/whatever pushes, resulting in 2.5E9 deliveries/day. After this introduction Raffi explained how Twitter, initially a RoR application ended up migrating some parts to the JVM. It turns out that at their scale Ruby's performance was definitely not sufficient. In their search for better performance they turned over to the JVM, but didn't pick its flagship, Java, as their language of choice. Instead, they picked Scala which felt more familiar to developers used to Ruby (they are also using Clojure in some parts!). Additionally, JVM's concurrency model was what they needed (seems that concurrency in Ruby really sucks).
However, it turns out that out-of-the-box JVM was still not perfect for the real-time, massive scale application that Twitter is. Twitter applications are running on custom JVM builds, having custom garbage collection. Ugh, that seems like challenges that would really scare an average developer. I'm glad I know how others are approaching those issues so I can keep calm if  I ever face such problems.
Next Raffi gave us a short introduction to Finagle, Twitter's framework for protocol-agnostic RPC, handling a lot of issues for developers: retries, connection management, transient error handling, load-balancing, distributed tracing, service discovery etc. Code samples looked really nice, with concise Scala syntax and all issues not directly related to the business logic hidden by the framework (which doesn't look invasive at all).
What I found interesting is that at the end Raffi stated that the fact they migrated from RoR to JVM doesn't mean that picking RoR in the first place was a mistake. With RoR you can get your application working really soon, and if you find out you need to scale to the point it's not enough you can look for alternatives. And that's the way he'd suggest startups to proceed.
I really enjoyed the talk, but felt a bit sad that I don't get to deal with such large scale problems in my work.

Complexity of Complexity by Ken Sipe

Next keynote was less technical, but interesting nonetheless. Some of the things discussed were already well known for me, such as the difference between essential and accidental complexity but Ken also offered a great number of advises really worth remembering. As he says, many things (idempotency for example) seem are hard for us to understand just because we're not familiar with them. Therefore it's hard to say that some programming language is easy and another one is hard. He stated (it was some quote IIRC) that you should pick up the most powerful language of those available and suitable for the problem at hand. Doing otherwise is a mistake. 
Ken also mentioned CAP theorem and shared some lessons learned by eBay developers such as embracing inconsistency and minimizing and controlling dependencies. He talked about the wise choices they've made - keeping the core business components highly available and the non-core only mostly available. Such trade-offs should be made to do the right thing and not waste resources on problems that you don't really have to care that much about. That's something people tend to forget or ignore and try to solve every tiny problem (ignoring the ROI).
I really liked one of the last sentences in the presentation: it's better/cheaper to develop product with good people than with cheap people.

Pointy haired bosses and pragmatic programmers: Facts and fallacies of software development by Venkat Subramaniam

I never before seen Venkat live. I have seen some of this talks online, but that was quite long ago and I didn't exactly remember what I should expect. The presentation was awesome. Venkat is a true rockstar and an awesome speaker, able to get your attention even if he doesn't say anything really new. He started with comparing software development to surfing - unpredictible, dynamic and really fun (but beware the sharks!). He offered a simple advice how to win software projects - with passionate, competent and responsible people because the thing that affects us most is the people (like Romans affecting the diameter of rocket boosters). Also we got warned of best practices (and advised to run whenever we hear those words).
Of course, as the title of the talk suggests, Venkat also went through a list of fallacies of software development:

  • More money and time will solve our problems (In fact, it turns out that projects with far deadlines and huge budgets tend to fail). I have already experienced it.
  • It gotta be good because it's from that large vendor.
  • Dynamic languages aren't safe (yeah, and Java's static type system is - riiiiiight - about as safe as a monkey using a computer)
A few memorable, fun and/or inspiring quotes from Venkat:
  • "Standardization before innovation is a bad idea"
  • "One thing you should never do is arguing with Java compiler"
  • "Don't fear to try new things, to make mistakes, to push the boundaries"
  • "A professional who doesn't learn to fail, fails to learn"
  • "Be unemotional in technical decisions, give things unbiased consideration"
Venkat finished his great show with a really good quote from Robert F. Kennedy, a quote every software professional should know (and agree with):
“Every time we turn our heads the other way when we see the law flouted, when we tolerate what we know to be wrong, when we close our eyes and ears to the corrupt because we are too busy or too frightened, when we fail to speak up and speak out, we strike a blow against freedom and decency and justice.” 
Howgh!

Build trust in your build to deployment flow by Frederic Simon

This talk was a bit disappointing for me. First, I arrived late because we wasted a lot of time waiting for our pizza in a nearby fast-food (for some unknown reasons we got reservation without lunch at the conference...). Then, the part of the presentation I've seen seemed to be an ad for Artifactory. The only thing I've learned from the talk is that Artifactory looks like a far more sophisticated tool than Nexus we're using at the company (which only seems to have a really slow search capability).

Being unsure what to do next I decided to take a look at Play framework. The presenter warned us at the very beginning that he was badly jetlagged - and that was clearly visible, but I enjoyed the presentation. Scala's (which is the language Play 2.0 is written in) syntax seemed nice again, and the way that the framework handles chunks of data looks interesting. A short demo of the Typesafe Console made me go really Wow! I guess the talk convinced me to take a look at the framework when I get some spare time (and take a look at many other, even more interesting things first).

Every rose has its thorn : Taming the automated tests beast by Wojtek Seliga

For this talk the room was overflowing with people. Wojtek is a leader in one of the Atlassian's Jira development teams and shared with us his experiences related to automated tests. Atlassian developers are serious about automated testing and Jira has an enormous tests suite, from unit tests to functional to integration tests using HtmlUnit to Selenium tests. The whole suite is about (if I'm not wrong) 13k tests that take an awful lot of time to run, slowing down development and causing fear of committing code (with distributed teams in different time zones, when a build broken by one team can prevent other teams from doing anything useful).
The presenter was very honest and clear about the mistakes they've made, such as putting all tests into a single project (actually a project per type of tests - unit/functional/etc) which makes the feedback loop extremally long. He shared a tip how to make their integration tests (going through the web UI) easier to maintain using what he called Page Objects (objects exposing actions you can take on given page but  hiding the ugly details of parsing HTML etc from the test itself).
The presentation was really awesome but it unfortunately ended with a sad accident involving a really ugly troll who came out shouting that Jira guys don't know how to write tests at all. When asked what would he do differently he yelled that we'd change the job if he were them and he left. Every time I saw him rising hand on different presentations later on I was expecting another flame but luckily it didn't happen. I regret I couldn't find traces of the troll online (though I remember his name well) - I would like to know where he works (if at all) just to be sure never to join such company ;)

Summary of day 1

If I had any doubts about coming to the conference before, they were all gone after the first day. It was a great experience and learning. I've spent the evening (up to late in the night) discussing what we've learned with my colleagues who were there with me.
(Oh, yeah, there was the beer party sponsored by ZeroTurnaround but as I don't drink beer it wasn't something I could get excited about.)

Tuesday, February 14, 2012

Open sourcing Surveys project

In late 2010 and first half of 2011 me and a few other friends (Rafał Jamróz, Jacek Bryła and Małgorzata Popiołek) have created a small Java+Flex application for creating, filling and evaluating surveys. It was intended to be used at the local university but AFAIK was never really put into use. That's not a big deal, really, because it was an awesome opportunity to learn, away from everyday job's problems and deadlines.
We have spent a lot of time doing refactoring (my main focus was on Java tests), testing new ideas. We even got a major re-write of the user interface when it turned out that our initial concept was not really usable by the target users (on backend the changes were major but not that hard to fit into existing design).


Quite a lot of time has passed since we have finished working on it (in fact - abandoned it). From the very beginning we intended to open-source it but were not sure whether our contract will allow us to do it. In the end we were able to keep all the rights to the code (in exchange for 2 years of support period - but we don't spend a lot of time supporting the released version ;) and now the big day has come - Surveys and it's helper projects (ActionScript code generator and some commons) go available on the web - hosted on github.

Initially we did host it on a private SVN repository on Xp-dev.com . We've done all of our development using the old, good SVN but decided to migrate it to git before public release. I used a short tutorial on how to migrate a repo along with its history to git and it worked great (we've lost some of the history while still on SVN, while splitting projects, renaming svn repos etc.).

To end this short self-advertisement I would like to say thanks to my collaborators - you were great! I really liked our "stand-up" breakfasts at work and our "iteration" reviews. I had a great time working with you on that project.

P.S.Head revision might not be stable (I hope it still builds correctly!). To build Flex you might need some dependencies that are not (or maybe were not at that time) available in public Maven repos.