Scala extension methods using implicits

What are extension methods?

Implicits are one of the fundamental language features in Scala. Some of its uses are in implicit method parameters, implicit object conversions and described in this article – implicit extension methods. This methodology is also called “pimp my library pattern”.

Extension methods allow you to add new capabilities to existing types / classes without the need of extending original type. This is especially useful when we want to enrich some class we do not have control over (e.g. external library).

A good example of this is in Apache Spark when adding support for new data sources for read write, e.g. Avro IO support. All we need to do is to add avro dependency to our project, add an import in the scope and magically our Dataframe/Dataset API has a new method available, e.g:

val df = spark.load.avro(...)

You can see implementation of this here: https://github.com/databricks/spark-avro/blob/master/src/main/scala/com/databricks/spark/avro/package.scala

Continue reading “Scala extension methods using implicits” »

How to export MySQL database or single table into a file and load it back in

Recently I was working on optimising our internal application at work that is heavily MySQL driven. I was focusing on reworking SQL statements and making sure we are utilizing indexes properly. Modyfing and toying around with tables on live server is less than ideal so I needed a way to reliably and quickly move the data elsewhere so I can steadily experiment and fine tune sql statements and tables. Instead of using available phpmyadmin export options I decided to work directly with mysql which gives you more controll.
Continue reading “How to export MySQL database or single table into a file and load it back in” »