I’m sharing with you my impressions on one of the best conferences I’ve ever attended.
It is one of the best, due to the technology leap that Docker is introducing now and will introduce in the future.
Cloud is impossible without containers.
DockerCon EU 2015 Barcelona
November 29, 2015Repair Canon EF 16-35mm F/2.8L II USM
January 6, 2015In a recent month I visited Canada and it was very cold, lower than -30C. I took many pictures with my Canon 16-35mm F/2.8L 2 usm, it is my favorite lens. In some day when I was taking pictures, I had noticed that the lens is not sitting firmly in its mount, but it appeared that the lens was broken some how and was wobbling. I searched the internet and found the following blog page http://psy-factor.livejournal.com/752711.html showing the broken lens part and explaining how to disassemble the lens and replace the part. I had ordered a part from ebay and replaced the part.
Many thanks to the owner of the blog http://psy-factor.livejournal.com/ Vladimir, he was kindly helping me with the disassembly and answered my questions. Thanks a lot Vladimir.
Here are disassembly pictures:
This was the broken part which I replaced.
I think the plastic handles were broken due to big temperature differences (about 55C degrees). Probably Canon has used cheap plastic parts on high-end lenses. Shame on you Canon!
Performance Tests on different versions of Jackson Scala Module 2_10 libraries
November 11, 2014Recently I performed performance tests on one of our server side components and saw very significant impact on performance. After some investigations I have managed to isolate the culprit of the performance degradation, it is the Jackson library (for json parsing), here are the results:
In my tests I use some relatively big json string ~11KBytes.
The test performs deserialization from json to object 10000 times, repeats it 10 times and makes an average.
def measure(repeat : Int, times : Int) = {
var start = 0L
var end = 0L
var diff = 0L
var sum = 0L
(1 to repeat).foreach { rep =>
start = System.currentTimeMillis()
(1 to times).par.foreach { i =>
val hh = JsonLayer.JSON_MAPPER.readValue(householdString("hh" + i * rep), classOf[Household])
//val st = JsonLayer.JSON_MAPPER.writeValueAsString(hh)
}
end = System.currentTimeMillis()
diff = (end - start)
sum += diff
logger.debug("Round " + rep + " took " + diff + " millis")
}
sum
}
@Test
def householdJsonPerformance() =
{
val repeat = 10
val times = 10000
var start = 0L
var end = 0L
var sum = 0L
var diff = 0L
// warmup
diff = measure(repeat, times)
logger.debug("Round warm up took " + diff + " millis")
Measure.time(this.logger, "performance household deserialize/serialize") {
sum = measure(repeat, times)
}
logger.debug("Average per run " + (sum/repeat) + " millis")
}
Jackson libraries version: 2.4.3
jackson-core-2.4.3.jar
jackson-annotations-2.4.3.jar
jackson-databind-2.4.3.jar
The only difference between the tests is following jar were updated each time:
jackson-module-scala_2.10-{version}.jar
The results of the test above with different versions of the jars:
2.2.2
Round 1 took 2921 millis
Round 2 took 842 millis
Round 3 took 677 millis
Round 4 took 597 millis
Round 5 took 556 millis
Round 6 took 590 millis
Round 7 took 697 millis
Round 8 took 597 millis
Round 9 took 577 millis
Round 10 took 548 millis
Round warm up took 8602 millis
[ Executing performance household deserialize/serialize ...]
Round 1 took 474 millis
Round 2 took 458 millis
Round 3 took 465 millis
Round 4 took 493 millis
Round 5 took 519 millis
Round 6 took 445 millis
Round 7 took 446 millis
Round 8 took 485 millis
Round 9 took 485 millis
Round 10 took 456 millis
[ performance household deserialize/serialize took 4731 ms ]
Average per run 472 millis millis
2.2.3
Round 1 took 8228 millis
Round 2 took 5428 millis
Round 3 took 3771 millis
Round 4 took 3798 millis
Round 5 took 3560 millis
Round 6 took 3558 millis
Round 7 took 3847 millis
Round 8 took 3709 millis
Round 9 took 3496 millis
Round 10 took 3836 millis
Round warm up took 43231 millis
[ Executing performance household deserialize/serialize ...]
Round 1 took 5355 millis
Round 2 took 6018 millis
Round 3 took 3962 millis
Round 4 took 3481 millis
Round 5 took 3821 millis
Round 6 took 3739 millis
Round 7 took 3545 millis
Round 8 took 3594 millis
Round 9 took 3615 millis
Round 10 took 3595 millis
[ performance household deserialize/serialize took 40726 ms ]
Average per run 4072 millis
2.3.3
Round 1 took 8631 millis
Round 2 took 5342 millis
Round 3 took 3617 millis
Round 4 took 3578 millis
Round 5 took 3633 millis
Round 6 took 3740 millis
Round 7 took 3541 millis
Round 8 took 3833 millis
Round 9 took 3678 millis
Round 10 took 3596 millis
Round warm up took 43189 millis
[ Executing performance household deserialize/serialize ...]
Round 1 took 3753 millis
Round 2 took 3408 millis
Round 3 took 4728 millis
Round 4 took 4270 millis
Round 5 took 3565 millis
Round 6 took 3521 millis
Round 7 took 3492 millis
Round 8 took 3573 millis
Round 9 took 3959 millis
Round 10 took 4094 millis
[ performance household deserialize/serialize took 38365 ms ]
Average per run 3836 millis
2.4.3
Round 1 took 8331 millis
Round 2 took 4903 millis
Round 3 took 3568 millis
Round 4 took 3545 millis
Round 5 took 3494 millis
Round 6 took 3597 millis
Round 7 took 3637 millis
Round 8 took 3629 millis
Round 9 took 3612 millis
Round 10 took 3550 millis
Round warm up took 41866 millis
[ Executing performance household deserialize/serialize ...]
Round 1 took 3611 millis
Round 2 took 3510 millis
Round 3 took 3508 millis
Round 4 took 3923 millis
Round 5 took 4848 millis
Round 6 took 4819 millis
Round 7 took 3960 millis
Round 8 took 3580 millis
Round 9 took 4092 millis
Round 10 took 4480 millis
[ performance household deserialize/serialize took 40332 ms ]
Average per run 4033 millis
To summarize:
There is performance degradation in jackson scala module 2_10 libraries starting from 2.2.3 version (2.2.2 version works fine)
In my tests above version jackson-module-scala_2.10-2.2.2 work ~7.5 times faster that jackson-module-scala_2.10-2.2.3
Jackson module scala 2_10 version Average latency
2.2.2 ——————————– 472 ms
2.2.3 ——————————– 4072 ms
2.3.3 ——————————– 3836 ms
2.4.3 ——————————– 4033 ms
There is an open ticket for that issue since May 2014
https://github.com/FasterXML/jackson-module-scala/issues/148
Spring 3 – SPEL Expression
September 17, 2014To use the configuration parameter in SPEL expressions use the following syntax:
<property name="someProperty" value="#{ '${config.parameter}' == 'some.string.to.compare.to' ? false : true }" />
Socket: Connection Refused – No Timeout
May 18, 2014Some one has asked me: Your server failed connecting to third party server, it retries and immediately gets a response: Connection Refused. Why is not there a time-out?
Here is capture from a log:
DEBUG: Creating a new socket to '10.38.6.62:1119'... ERROR: Failed reconnecting: Connection refused DEBUG: Creating a new socket to '10.38.6.62:1119'... ERROR: Error - Connection refused [ socket = null ] Going to retry for the '3' time. DEBUG: Creating a new socket to '10.38.6.62:1119'... ERROR: Failed reconnecting: Connection refused ERROR: Could not communicate with 10.38.6.62:1119. '3' retries are ended. Total request time '11' ms
The answer is:
Connection Refused means that there is no program at the other end of the connection that listens to incoming connections. Connection time-outs are for the cases where there is no response at all. A connection refusal is a valid response, so the time-out doesn’t apply in such case.
Spray vs Scalatra
May 15, 2014During last few evenings I have evaluated two of the well known frameworks:
Spray http://spray.io/
AND
Scalatra – http://www.scalatra.org/
————————————————————————-
My goal was creating few simple web pages to receive (GET/POST) requests, go to third-party http server for results and return response to the client.
I have started from Spray framework:
The main class looks like this:
It creates Http server and connects it with Akka
Akka IO will receive all the communication and redirect it to Spray’s Http layer, which in his turn redirects it to service actor of type HttpRoutingService shown below
import spray.can.Http
import akka.actor.{Props, ActorRef, ActorSystem}
import akka.io.IO
/**
* Main Class
*
*/
object Main extends App {
try {
// we need an ActorSystem to host our application in
implicit val system = ActorSystem("on-spray-can")
// create and start our service actor
val service = system.actorOf(Props[HttpRoutingService], "routing-service")
// start a new HTTP server on port 8080 with our service actor as the handler
IO(Http) ! Http.Bind(service, "localhost", port = 8080)
}
catch
{
case t : Throwable => println(t)
}
}
Let’s look at HttpRoutingService class
import scala.concurrent.duration._
import akka.pattern.ask
import akka.util.Timeout
import akka.actor._
import spray.can.Http
import spray.can.server.Stats
import spray.util._
import spray.http._
import HttpMethods._
import MediaTypes._
import spray.can.Http.RegisterChunkHandler
import java.io.File
import org.parboiled.common.FileUtils
import scala.concurrent.duration._
import akka.actor._
import akka.pattern.ask
import spray.routing.{HttpService, RequestContext}
import spray.routing.directives.CachingDirectives
import spray.can.server.Stats
import spray.can.Http
import spray.httpx.marshalling.Marshaller
import spray.httpx.encoding.Gzip
import spray.util._
import spray.http._
import MediaTypes._
import CachingDirectives._
// we don't implement our route structure directly in the service actor because
// we want to be able to test it independently, without having to spin up an actor
class HttpRoutingService extends Actor with RoutingService {
// the HttpService trait defines only one abstract member, which
// connects the services environment to the enclosing actor or test
def actorRefFactory = context
// this actor only runs our route, but you could add
// other things here, like request stream processing,
// timeout handling or alternative handler registration
def receive = runRoute(httpRoutes)
}
// this trait defines our service behavior independently from the service actor
trait RoutingService extends HttpService {
// we use the enclosing ActorContext's or ActorSystem's dispatcher for our Futures and Scheduler
implicit def executionContext = actorRefFactory.dispatcher
val loginActor = ActorSystem().actorOf(Props[LoginActor])
val httpRoutes =
get
{
pathSingleSlash {
complete(index)
} ~
path("login") {
complete(loginForm)
}
} ~
post {
path("loginForm") {
//entity(as[Login]){ login =>
entity(as[FormData]) { fields =>
println("Fields = " + fields)
val username = fields.fields(0)._2
val password = fields.fields(1)._2
val login = Login(username, password)
println("Login = " + login)
loginActor! login
complete("Done")
}
}
}
lazy val index =
<html>
<body>
<h1>Say hello to <i>spray-routing</i> on <i>spray-can</i>!</h1>
<p>Defined resources:</p>
<ul>
<li><a href="/login">/login</a></li>
</ul>
</body>
</html>
lazy val loginForm =
<div class="login_block">
<form name="LoginForm" action="/loginForm" method="post">
<table class="login">
<tbody>
<tr>
<td>
<input type="text" name="username" id="login_email" />
</td>
</tr>
<tr>
<td>
<input type="text" name="password" id="Irealpass" />
</td>
</tr>
<tr>
<td align="left"><input type="submit" name="login" /></td>
</tr>
</tbody>
</table>
</form>
</div>
}
The class defines few Http routes:
- HTTP GET / -> serves the default page (index page)
- HTTP GET /login -> shows a simple submit form
- HTTP POST /loginForm -> receives a submit form, takes out form fields and redirects request to other actor
I would like to say that I have spent a lot of time to make this routing service to work. I had trouble compiling the class, due to some errors with tildas and problems with getting out the fields from a form (this was a nightmare… I do not know why it should so complex) The syntax is not easy nor intuitive for understanding at all (at least for me). There are some tildas… I cannot understand why do we need them?! We don’t have them in Scalatra.
Here is the code of additional actor – LoginActor
This actor receives a login message with username and password, sends http request for login to some third-party server.
It was important to me to get the hands on the spray http client and try it out. It also was not so straight forward working with it. I had some trouble with HttpHeaders and its huge hierarchy…
Anyway, I find the dispatch http client the easiest of all to use.
Dispatch Http Client is a Scala wrapper for Java Async Http Client
import akka.actor.{ActorSystem, ActorLogging, Actor}
import akka.util.Timeout
import java.util.concurrent.TimeUnit
import spray.http._
import scala.concurrent.Future
import akka.io.IO
import spray.can._
import spray.client.pipelining._
import HttpMethods._
import spray.http.HttpRequest
import spray.http.HttpResponse
import spray.http.HttpHeaders.{RawHeader, Accept}
import scala.util.{Success, Failure}
import spray.httpx.marshalling.Marshaller
/**
* Created by roma on 5/13/14.
*/
class LoginActor extends Actor with ActorLogging {
implicit val timeout: Timeout = Timeout(15, TimeUnit.SECONDS)
implicit val system: ActorSystem = ActorSystem()
import system.dispatcher // implicit execution context
def receive() =
{
case Login(username, password) => login(username, password)
}
def login(username : String, password : String) =
{
log.info("Logging in with Username = " + username + " ...")
val headers = List[HttpHeader](
RawHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"),
RawHeader("Origin", "http://www.server.com"),
RawHeader("Content-Type", "application/x-www-form-urlencoded"),
RawHeader("Referer", "http://www.server.com/"),
RawHeader("Accept-Encoding", "gzip,deflate,sdch"),
RawHeader("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,he;q=0.4,ru;q=0.2"))
val formData = FormData(List(("Username",username), ("Password", password)))
val entity = HttpEntity.NonEmpty(ContentType(MediaTypes.`application/x-www-form-urlencoded`), HttpData(formData))
val pipeline: HttpRequest => Future[HttpResponse] = sendReceive
val response: Future[HttpResponse] = pipeline(Get("http://www.server.com/login.php").withHeaders(headers).withEntity(formData))
response onComplete{
case Failure(ex) => log.error("Failed logging in - " + ex.getMessage)
case Success(resp) =>
resp.status.intValue match
{
case 200 =>
log.debug("Body: \n" + resp.entity.asString)
log.info("Successfully logged in as '" + username + "'")
case _ =>
log.error("Failed logging in - " + resp.status)
}
}
}
}
Scalatra
After Spray I found Scalatra very easy to use and work with. Intuitive APIs without any complex syntax (remember tildas in Spray?)
Scalatra requires a specific project structure (actually the same as all other web projects):
I just copied the webapp directory from one of the Scalatra’s examples.
In my set up I used embedded jetty as Http serlvet container for Scalatra, here is how it is done:
The main class is a JettyLauncher class:
A simple class which starts up a Jetty Server and passes it Scalatra’s listener servlet
import akka.actor.{Props, ActorRef, ActorSystem}
import akka.io.IO
import org.eclipse.jetty.servlet.DefaultServlet
import org.scalatra.servlet.ScalatraListener
import org.eclipse.jetty.webapp.WebAppContext
import org.eclipse.jetty.server.Server
import com.typesafe.scalalogging.slf4j.LazyLogging
/**
* Main Class
*
*/
object JettyLauncher extends App with LazyLogging {
try {
logger.info("Starting up ...")
val port = if(System.getenv("PORT") != null) System.getenv("PORT").toInt else 8080
logger.info("Using port " + port)
val server = new Server(port)
val context = new WebAppContext()
context setContextPath "/"
context.setResourceBase("src/main/webapp")
context.addEventListener(new ScalatraListener)
context.addServlet(classOf[DefaultServlet], "/")
server.setHandler(context)
server.start
logger.info("Running.")
server.join
}
catch
{
case t : Throwable => logger.error("ERROR", t)
}
}
ScalatraBootstrap class bind Scalatra’s Listener with the ScalatraServlet. In our case it’s called HttpHandlerService
import org.scalatra._
import javax.servlet.ServletContext
class ScalatraBootstrap extends LifeCycle {
override def init(context: ServletContext) {
context.mount(new HttpHandlerService, "/*")
}
}
import akka.util.Timeout
import akka.actor._
import akka.pattern.ask
import org.scalatra.ScalatraServlet
import com.typesafe.scalalogging.slf4j.LazyLogging
import java.util.concurrent.TimeUnit
import dispatch.Defaults.executor
import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import com.ning.http.client.Response
class HttpHandlerService extends ScalatraServlet with LazyLogging {
implicit val timeout = Timeout(10, TimeUnit.SECONDS)
logger.debug("Creating actors ...")
val loginActor= ActorSystem().actorOf(Props[LoginActor])
logger.debug("done.")
get("/")
{
index
}
get("/login") {
<html><body>{loginForm}</body></html>
}
post("/loginForm") {
val username = params("username")
val password = params("password")
val login = Login(username, password)
logger.debug("Sending to loginActor...")
val futureData = loginActor.ask(login)
logger.debug("Done. Waiting for response ...")
val responseData = futureData map { data => }
logger.debug("Received response.")
responseData match
{
case ResponseData(response) => logger.debug(response.getResponseBody); <html><body>"Logged in as " {username}</body></html>
case other @ _ => logger.debug("Other data type: " + other); <html><body>"Other Data Type received: " { other }</body></html>
}
}
lazy val index =
<html>
<body>
<h1>Welcome to <i>bouncer</i> on <i>scalatra</i>!</h1>
<p>Defined resources:</p>
<ul>
<li><a href="/login">Login</a></li>
</ul>
</body>
</html>
lazy val loginForm =
<section class="loginform cf">
<form name="login" action="index_submit" method="get" accept-charset="utf-8">
<ul>
<li>
<label for="usermail">Email</label>
<input type="email" name="usermail" placeholder="yourname@email.com" required="true" />
</li>
<li>
<label for="password">Password</label>
<input type="password" name="password" placeholder="password" required="true" />
</li>
<li>
<input type="submit" value="login" />
</li>
</ul>
</form>
</section>
}
I really liked the way the Scalatra defines routes. Very easy and clear, without additional noisy declarations. It took me 10 minutes to start the Scalatra’s service. Well Done Scalatra guys!
Here is the LoginActor class with the usage of Dispatch http client:
import akka.actor.{ActorRef, ActorSystem, ActorLogging, Actor}
import akka.util.Timeout
import java.util.concurrent.{ExecutorService, Executors, TimeUnit}
import scala.util.{Random, Success, Failure}
import dispatch._
import com.typesafe.scalalogging.slf4j.LazyLogging
import com.ning.http.client._
import com.ning.http.client.AsyncHandler.STATE
import com.ning.http.client.cookie.Cookie
import scala.collection.JavaConversions._
/**
* Created by roma on 5/13/14.
*/
class LoginActor extends Actor with LazyLogging {
implicit val timeout: Timeout = Timeout(15, TimeUnit.SECONDS)
implicit val system: ActorSystem = ActorSystem()
import system.dispatcher // implicit execution context
private var isRunning : Boolean = false
private var lastRun : Long = 0
def receive() =
{
case Login(username, password) => sender ! Data(login(username, password))
}
def login(username : String, password : String) =
{
logger.info("Logging in with Username = " + username + " ...")
// create request
val request = url("""http://www.server.com/login.php""").POST
// add headers
request.addHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
request.addHeader("Origin", "http://www.yad2.co.il")
request.addHeader("Content-Type", "application/x-www-form-urlencoded")
request.addHeader("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36")
request.addHeader("Referer", "http://www.server.com/")
request.addHeader("Accept-Encoding", "gzip,deflate,sdch")
request.addHeader("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,he;q=0.4,ru;q=0.2")
request.addParameter("Username", username)
request.addParameter("Password", password)
val futureResponse = Http(request OK as.String)
futureResponse
}
}
I really liked how the Dispatch framework works. Recommend you to try it out – link
I used maven so here are the maven dependencies:
<dependency>
<groupId>org.scalatra</groupId>
<artifactId>scalatra_2.10</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.10</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.scalatra.scalate</groupId>
<artifactId>scalate-web_2.10</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>org.scalatra.scalate</groupId>
<artifactId>scalate-project_2.10</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-webapp</artifactId>
<version>8.1.15.v20140411</version>
</dependency>
<dependency>
<groupId>com.typesafe.scala-logging</groupId>
<artifactId>scala-logging-slf4j_2.10</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.6.6</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.9</version>
</dependency>
<dependency>
<groupId>net.databinder.dispatch</groupId>
<artifactId>dispatch-core_2.11</artifactId>
<version>0.11.1</version>
</dependency>
Bug in AWS Image of Red Hat Linux 6.4
March 27, 2014I was stuck a couple of hours with the following problem when using an Amazon Red Hat Enterprise Linux 6.4 (RHEL 6.4):
Configure and start the RHEL 6.4 Instance
Stop the instance, I was unable to connect again to that machine.
The problem is that the /etc/rc.local file inserts some configuration lines to /etc/ssh/sshd_config file and it has an error, so next time when machine restarts it fails to start up the ssh daemon.
The solution is to fix or remove the ssh lines from /etc/rc.local and fix the /etc/ssh/sshd_config file
Thanks to my colleagues which solved this issue.
Whatsapp – Very impressive backend performance
February 20, 2014Whatsapp Performance:
- 571000 pkts/sec, >200k dist msgs/sec
- Whatsapp servers are doing 1 million tcp connections on a single box: http://blog.whatsapp.com/?p=170
- $ netstat -an | grep -c EST
1016313
Whatsapp backend uses:
- Erlang language – http://en.wikipedia.org/wiki/Erlang_(programming_language)
- Dual Westmere Hex-core (24 logical CPUs)
- 100GB RAM, SSD
- Dual NIC (user-facing, back-end/distribution)
- FreeBSD 8.3
- OTP R14B03 (Erlang Open Telephone Platform)
http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf
https://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/
PS:
Akka and reactive programming is the way to go in server side!
Http Server Mock
February 13, 2014Today I found nice open source framework mocking an Http Server
Really a nice one.
But I fought with the following exception when using it from Scala class:
java.lang.Exception: The @Rule 'wireMockRule' must be public.
And finally found a solution here:
http://randomallsorts.blogspot.co.il/2012/11/junit-411-whats-new-rules.html

































