Scala extension methods using implicits

What are extension methods?

Implicits are one of the fundamental language features in Scala. Some of its uses are in implicit method parameters, implicit object conversions and described in this article – implicit extension methods. This methodology is also called “pimp my library pattern”.

Extension methods allow you to add new capabilities to existing types / classes without the need of extending original type. This is especially useful when we want to enrich some class we do not have control over (e.g. external library).

A good example of this is in Apache Spark when adding support for new data sources for read write, e.g. Avro IO support. All we need to do is to add avro dependency to our project, add an import in the scope and magically our Dataframe/Dataset API has a new method available, e.g:

val df = spark.load.avro(...)

You can see implementation of this here:

Continue reading “Scala extension methods using implicits” »

Finding which tables reference column in given table in MySQL

When dealing with schema updates in MySQL I often have to check what foreign key constraints are in place. How can we find which table have columns that reference the primary column on the table we’re modifying. Here’s the SQL statement that will help us find those references.


Difference between functions and methods in Scala

Scala has a concept of functions and method definitions. These two concepts are often confused and it’s not always clear when to use which.

Consider this example.

Simple method:

def m1(x: Int) = x + x
m1(2)  // 4

Simple function:

val f1 = (x: Int) => x + x
f1(2)  // 4

Both m1 and f1 are called the same way and will produce the same result but when you look closer you will see that these are two different things.

f1  // Int => Int = <function1>
m1  // error: missing argument list for method m1...

Calling f1 without argument will give us the signature of anonymous function.
Our anonymous functions is actually an instance of Function1[Int, Int] that means a function with 1 parameter of type Int and return value of type Int.

f1.isInstanceOf[Function1[Int, Int]]  // Boolean = true  

When calling m1 we will get an error. This is because m1 is not a value, it is a method, something that will produce a value when you call it by providing all required arguments.

Converting method into a function

Method can be converted into a proper function (often referred to as lifting) by calling method with underscore “_” after method name.

val f2 = m1 _  // Int => Int = <function1>

Alternatively you can supply a type and compiler will know what to do.

val f2: Int => Int = m1  // Int => Int = <function1>

Above method to function conversions will always create a new instance of Function1[Int, Int] which is an important observation.

Let’s consider following example. We are creating a sequence of tuples with type (Int, Int => Int). For each tuple we decided to convert our previously defined method into a function. The following will create 10 instances of a function that does the same thing.

for (i <- 0 until 10) yield i -> m1 _

Better approach would be to pass the same instance of the function for each tuple and avoid allocating memory for each instance as in previous approach.

for (i <- 0 until 10) yield i -> f1

When to use methods and when functions

– use functions if you need to pass them around as parameters
– use functions if you want to act on their instances, e.g. f1.compose(f2)(2)
– use methods if you want to use default values for parameters e.g. def m1(age: Int = 2) = ...
– use methods if you just need to compute and return

How to create git like command line interface in Scala

Recently I’ve been building a small application that exposed only command line interface to the user. While I found some solutions that suggested to use plain Scala for this task (which was pretty straighforward) I wanted something that is more suitable. For example I wanted to make sure that –help will be auto-generated so I don’t have to define it manually.

After little research I found scopt which had all the features I needed. Syntax may be a bit confusing at first especially for people new to Scala (myself included) but after a while you will appreciate it’s simplicity.

As an example of what scopt can do I decided to implement a demo app resembling GIT CLI with branch and commit child commands. This will allow for executing commands like git commit [options] or git checkout <branch> [options].

Continue reading “How to create git like command line interface in Scala” »

Finding shortest path using Dijkstra’s algorithm and weighed directed graph

One of the first known uses of shortest path algorithms in technology was in telephony in the 1950’s. The problem was in finding alternate routing in case shortest route became blocked. In today’s world shortest path algorithms are used in plethora of applications. You can find them when calculating driving directions, airline travel routing, network communications, linguistics, social networks – they all use graphs to represent relationships between object.

In this particular post, I wanted to implement a weighed directed graph data structure. Because of its properties, weighed digraph is commonly used in mapping applications. I will use this graph in conjunction with the well-known Dijkstra’s algorithm to find shortest path between two points on a map. In order to do this, I will map intersection points on a small section of city map, then calculate shortest distance between those sets of points using our algorithm. Finally, I will use Google Maps driving directions on the same point sets to validate our experiment.

Continue reading “Finding shortest path using Dijkstra’s algorithm and weighed directed graph” »

Basic Data Structures and Their Properties

Knowledge of data structures is fundamental for any software engineer.
Why data structures are so important? The goal of data structures is to organize and store information in an efficient manner. But there is no one perfect data structure that can/should be used everywhere. You have to pick the right tool for the job. While you can use a hammer to drive both nails and screws into a piece of wood, using a screw driver for latter would most likely be a better choice. In this post I wanted to focus on most basic data structures and their properties. In later articles I will go over implementation of those basic data structures.
Continue reading “Basic Data Structures and Their Properties” »

How to use IPython notebook with Dropbox

Recently I fell in love with IPython notebook. I installed it on my PC, laptop and my station at work. Very quickly I realized that sometimes I need to access notebook files from work pc at home and vice versa. Since all notebook files are saved locally you have to copy it over to another machine in order to access it.
One way to fix the problem is to setup IPython notebook on remote server and that way all your notebooks are in one place. Another way is to tell your local notebook to look into remote location for files.
Since Dropbox integrates well with Linux by mounting its resource into filesystem it is and ideal service for this purpose. It may be possible to use Google Drive as well but as far as I know it is a little bit more involved.
Continue reading “How to use IPython notebook with Dropbox” »

How to export MySQL database or single table into a file and load it back

Recently I was working on optimising our internal application at work that is heavily MySQL driven. I was focusing on reworking SQL statements and making sure we are utilizing indexes properly. Modyfing and toying around with tables on live server is less than ideal so I needed a way to reliably and quickly move the data elsewhere so I can steadily experiment and fine tune sql statements and tables. Instead of using available phpmyadmin export options I decided to work directly with mysql which gives you more controll.
Continue reading “How to export MySQL database or single table into a file and load it back” »

Editing your website files live in Linux/Ubuntu without FTP client

Whenever I work on small web projects I tend to edit files live to speed up the process. I used to use Filezilla FTP client for this purpose although every time you’re saving the file, Filezilla asks you if you want to overwrite the file. This becomes a little bit annoying over time.
On another note, If you’re developing on Windows I suggest you look at WinSCP free FTP client that will let you edit file live and won’t bug you every time you’re saving it.
Continue reading “Editing your website files live in Linux/Ubuntu without FTP client” »