# Pig to Scalding¶

This pages intends to help Pig users to learn Scalding by listing corresponding statements and basic Scala knowledge. You should also take a look at the tutorial.

Pig:

A = LOAD 'foo'


Scalding:

// The TextLine source splits the input by lines.
val textSource = TextLine(args("input"))
// Create a type-safe pipe from the TextLine.
val lines: TypedPipe[String] = TypedPipe.from[String](textSource)


## STORE¶

Pig:

STORE B INTO 'bar'


Scalding:

b.write(TypedTsv[String](args("output")))


## FOREACH¶

Pig:

B = FOREACH A GENERATE /* expression */


Scalding:

val b = a.map((t) => /* expression */)


## FILTER¶

Pig:

B = FILTER A BY foo == 0


Scalding:

a.filter{ case (foo, bar) => foo == 0 } // using pattern matching to name elements of a tuple
// if you don't need to name an element you can use the _ wildcard instead
a.filter{ case (foo, _) => foo == 0 }


## FOREACH A GENERATE FLATTEN(...)¶

in Scalding the use of flatMap is similar to the following in Pig:

B = FOREACH A GENERATE FLATTEN(Tokenize(text))


in Scalding:

def tokenize(s: String) = s.split("\\s+").toList
b = a.flatMap(tokenize(_))
// which produces the same result as:
b = a.map(tokenize(_)).flatten()
// and the same as
b = a.map(tokenize(_)).flatten // empty parens are usually omitted


Pig:

B = FOREACH (GROUP A BY $0) GENERATE COUNT(A)  Scalding: val b = a.groupBy(_._1).size  notice the _ shorthand used here. ## Join¶ Pig: C = JOIN A BY$0, B BY \$0


Scalding: assuming a and b are both a Pipe[(K,V)], you can join them as follows

val c = a.join(b)


## Scala cheat sheet¶

It is recommended to know the basics of Scala when trying out Scalding. Here are some common things Scala noobs may become confused about coming from Java and Pig.

### Primitive types¶

Scala uses the java primitive type names but with the first letter capitalized. (Scala uses the boxed type automatically when needed.)

For example:

Java:

final int a = 1


Scala:

val a = 1  // (val means it's a constant. Type is inferred. use var for variables)
val a: Int = 1 // same thing with explicit type declaration


### Functions¶

def f(x:Int) = x * 2 // return type inferred

def f(x:Int): Int = x * 2 // same thing with explicit return type


### Common types¶

#### Case classes¶

A case class is an immutable data class that can be used in pattern matching. For example:

case class User(val firstname: String, val lastname: String)


is kind of similar to the following Java code (plus the added benefit of pattern matching):

final class User {
// these are immutable so it's fine to make them public
public final String firstname;
public final String lastname;
public User(String firstname, String lastname) {
this.firstname = firstname;
this.lastname = lastname;
}
}


#### Tuples¶

fixed size with type assigned to each field ex:

val t = (1, "foo") // the type of t is Tuple2[Int, String]
t._1 // => 1
t._2 // => "foo"


assigning the members of t to a and b:

val (a, b) = t
a // => 1
b // => 2


it is the same as:

val a = t._1
val b = t._2


it is not the same as:

val a, b = t
// which is:
val a = t
val b = t


### Pattern matching¶

example:

t match {
case (a, b) => a
}


Which translates to: if t is a Tuple2, assign t._1 to a and t._2 to b and return a.

You don’t need to name things you don’t use. The _ wildcard can be used:

t match {
case (a, _) => a
}


Similarly with case classes:

// This is the same as User.apply("Jack", "Jackson"). Not a constructor call
val u = User("Jack", "Jackson")

val v = u match {
case User(firstname, lastname) => firstname
... // other cases
}


case class Name(first: String, middle: String, last: String)
case class Address(street: String, zip: String, city: String)

val p = Person(Name("Bob", "E.", "Roberts"), 42, Address("23 colorado st.", "99999", "Las Vegas"))

// unwrap Person
p match { case Person(a,b,c) => (a,b,c) }

// unwrap Person and Name
p match { case Person(Name(f,m,l), b, c) => (f, m, l, b, c) }

// multiple case statements (anonymizing minors not in the "Roberts" family)
p match  {
// matches only when lastname in Name is "Roberts"
case Person(Name(first, _,"Roberts"), _, _) => first

// predicate can be applied as well
case Person(Name(first, _, _), age, _) if (age > 21) => first

// default case if none of the above applies
case _ => "anonymous"
}
// just extracting age
val age = p match { case Person(_, age, _) => age }

// The previous line does the same thing as
val age = p.age

// flattening the entire structure
p match {
case Person(Name(f,m,l), age, Address(street, zip, city)) =>
(f, m, l, age, street, zip, city)
}


### Typed pipes basics¶

#### Map¶

If we have the following:

• p1 of type TypedPipe[T]
• f of type Function1[T,U]

then we can do

val p2: Pipe[U] = p1.map(f)


p2 is of type Pipe[U]

#### Lambda syntax¶

When defining a function inline we use the following syntax:

(param1, param2, ...) => /* expression */


which can be used in map

p.map( (a) => a + 1 )


Here we are defining a function that takes one parameter named a and apply it to all elements of p

#### Map variations¶

With p1 of type Pipe[(Int, String)] (a Pipe of Tuple2[Int, String]) mapping elements in p1:

p1.map( (t) => t._1 )


When a function takes only one parameter and is extremely simple, we can use the following shorthand:

p1.map( _._1 )


This syntax defines a function that takes one parameter on which we call ._1 (get the first element of the tuple)

WARNING: _ expands only to the expression directly around it. _._1._2 works but (_._1)._2 does not. (It turns into ((t) => (t._1))._2 which does not compile.) Always fallback to the full syntax when in doubt: (t) => (t._1)._2 works.

In Scala the syntax for getting a field is the same as for calling a parameter-less method (parens are omitted). In fact getting a fields is calling a parameter-less methods.

#### Operator notation to call a function¶

 p1 map f


is the same as

 p1.map(f)


In Scala every method can be used as an operator. In fact, this is how operators are implemented as symbols are allowed in method names.

p1 filter { _._1 == 0 } map { _._2 }


also:

p1.map { (t) => t._1 }


Notice the curly braces, we’re executing a block of code that returns a function. The result (last statement) of { } is passed to map

// "foo" is printed once (before passing the function to map)
p1.map { println("foo"); (t) => t._1 }

// "foo" is printed once (before passing the function to map)
p1.map { println("foo"); _._1 }

// "foo" is printed for each element
p1.map{ (t) => { println("foo"); t._1 } }


#### Pattern matching shorthand¶

p1.map { case (a,b) => a }


Passing a block of code that returns a partial function is a short hand for:

p1.map( (t) => t match { case (a,b) => a } )