TDD or not to Be

Developers journey to clean code.

Using Multiple Ssh-ids With Github

| Comments

Sometimes I need to interact with github using multiple ssh-keys. This is how I configure git to do it:

Lets create new rsa key

1

ssh-keygen

Then configure new key in github

1

  cat ~/.ssh/new_id_rsa.pub

When you try to push ssh-key-agent will use your old key. Create/Modify the file ~/.ssh/config

1
2
3
4

Host github.com-new_id_rsa
  HostName github.com
  User git
  IdentityFile ~/.ssh/new_id_rsa

Next open .git/config in your project and change url accordingly:

1
2


url = git@github.com-new-id-rsa:YOUR_USERNAME/YOUR_REPOSITORY.github.git

Now in your project directory you should be able to git push

Consuming Google AdWords Api With Akka Spray

| Comments

Google AdWords Api is a very tricky beast when under heavy use. When you send too many requests per minute you get banned for 30s. Moreover the amount of queries that you cam perform varies durning the day. I found consuming Google AdWords Api a perfect use case for Scala Akka and Spray frameworks. Please take into account I’m beginer Scala programmer and it’s one of my very first projects in that language. We will build Spray Api that will accept json list of keywords, adWords locationId and languageId and we return the keywords enchanced with targetingIdea and trafficEstiamtor service data. We use throttle actor(http://doc.akka.io/docs/akka/snapshot/contrib/throttle.html) to limit amount of queries to adWords Api.

  1. Start new Spray project like described at: https://github.com/spray/spray-template/tree/on_spray-can_1.3
1

 git clone git://github.com/spray/spray-template.git adwords-project
  1. Add the following libraries to build.sbt
1
2
3
4
5
6
7
8
9

 "com.typesafe.akka" %% "akka-contrib" % akkaV,
 "io.spray" %%  "spray-json" % "1.3.2",
 "com.google.api-ads" % "ads-lib" % "2.11.0" exclude("commons-beanutils", "commons-beanutils"),
 "com.google.api-ads" % "adwords-axis" % "2.11.0" exclude("commons-beanutils", "commons-beanutils"),
 "commons-beanutils" % "commons-beanutils" % "1.9.2",
 "com.google.http-client" % "google-http-client-gson" % "1.21.0",
 "com.google.inject" % "guice" % "4.0",
 "org.slf4j" % "slf4j-simple" % "1.7.5",
 "javax.activation"  % "activation"  % "1.1.1"
  1. Write JSON converter We want to return json, and we need to convert adWords api response to json In order to do that we write json converter:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

implicit object AnyJsonFormat extends JsonFormat[Any] {
    def write(x: Any): JsValue = x match {
      case n: Int => JsNumber(n)
      case l: Long => JsNumber(l)
      case d: Double => JsNumber(d)
      case f: Float => JsNumber(f.toDouble)
      case s: String => JsString(s)
      case x: Seq[_] => seqFormat[Any].write(x)
      case m: Map[_, _] if m.isEmpty => JsObject(Map[String, JsValue]())
      // Get the type of map keys from the first key, translate the rest the same way
      case m: Map[_, _] => m.keys.head match {
        case sym: Symbol =>
          val map = m.asInstanceOf[Map[Symbol, _]]
          val pairs = map.map { case (sym, v) => (sym.name -> write(v)) }
          JsObject(pairs)
        case s: String => mapFormat[String, Any].write(m.asInstanceOf[Map[String, Any]])
        case a: Any =>
          val map = m.asInstanceOf[Map[Any, _]]
          val pairs = map.map { case (sym, v) => (sym.toString -> write(v)) }
          JsObject(pairs)
      }
      case a: Array[_] => seqFormat[Any].write(a.toSeq)
      case true        => JsTrue
      case false       => JsFalse
      case p: Product  => seqFormat[Any].write(p.productIterator.toSeq)
      case null        => JsNull
      case x           => JsString(x.toString)
    }

    def read(value: JsValue) = value match {
      case JsNumber(n) => n.intValue()
      case JsString(s) => s
      case JsTrue      => true
      case JsFalse     => false
    }
  }

We add the code above the routes into HttpService Trait 4. Add a route Our only action will accept list of keywords and respond with enchanced data from targetingIdeaService

1
2
3
4
5
6
7
8

post {
    requestInstance { req =>
      val request = req.entity.asString.parseJson.convertTo[KeywordsRequest]
      complete {
        AdwordsCaller.handler(request.keywords, request.location_id, request.language_id).mapTo[Map[String, Map[String, Any]]]
      }
    }
  }
  1. Implement an action Adwords Caller implements Akka actor system that throttles requests to google limited to specific timeframe
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

case class CallAdwords(keywords: Array[String], location_id: Integer, langauge_id: Integer, session: AdWordsSession, AdWordsServices: AdWordsServices)
case object CallAdwords

class AdwordsCaller extends Actor {
  def receive = {
    case CallAdwords(keywords, location_id, language_id, session, adWordsServices) => sender ! GetAdwordsStats.run(keywords, location_id, language_id, session, adWordsServices)
  }
}

object AdwordsCaller {

  val system = ActorSystem()
  import system.dispatcher
  implicit val timeout = Timeout(60.second)

  val adwordsCaller = system.actorOf(Props[AdwordsCaller])
  val throttler = system.actorOf(Props(classOf[TimerBasedThrottler], 200 msgsPer 60.seconds))
  throttler ! SetTarget(Some(adwordsCaller))

  def handler(keywords: Array[String], location_id: Integer, language_id: Integer): Future[Any] = {
    val result = (throttler ? CallAdwords(keywords, location_id, language_id, null, null))
    return result
  }
}
  1. Call AdWords Api Finally GetAdwordsStats calls GoogleAdwords Api IdeaQuery and TrafficEstimation Query and combines results for each keyword
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

 def run(keywords: Array[String], location_id: Integer, language_id: Integer, session: AdWordsSession, adWordsServices: AdWordsServices): Map[String, Map[String, Any]] = {
      val oAuth2Credential = new OfflineCredentials.Builder().forApi(Api.ADWORDS)
    .fromFile("./ads.properties")
    .build()
    .generateCredential()
  val logger = LoggerFactory.getLogger(classOf[AdWordsServices])
  val session = new AdWordsSession.Builder().fromFile().withOAuth2Credential(oAuth2Credential)
    .build()
  val adWordsServices = new AdWordsServices()
    
    val ideaResults = runTargetingIdeaQuery(adWordsServices, session, keywords, location_id, language_id)
    val traficEstimatorResults = runTrafficEstimatorQuery(adWordsServices, session, keywords, location_id, language_id)
    var resultKeywords = Map[String, Map[String, Any]]()
    for (keyword <- keywords) {
      resultKeywords = resultKeywords + (keyword.toLowerCase() -> Map())
      for ((ideaKey, ideaValue) <- ideaResults(keyword.toLowerCase())) {
        var newVal = resultKeywords(keyword.toLowerCase()) + (ideaKey -> ideaValue)
        resultKeywords = resultKeywords + (keyword.toLowerCase() -> newVal)
      }
      for ((ideaKey, ideaValue) <- traficEstimatorResults(keyword)) {
        var newVal = resultKeywords(keyword.toLowerCase()) + (ideaKey -> ideaValue)
        resultKeywords = resultKeywords + (keyword.toLowerCase() -> newVal)
      }
    }

    return resultKeywords
  }

  def runTargetingIdeaQuery(adWordsServices: AdWordsServices, session: AdWordsSession, keywords: Array[String], location_id: Integer, language_id: Integer): Map[String, Map[String, Any]] = {

    val targetingIdeaService = adWordsServices.get(session, classOf[TargetingIdeaServiceInterface])
    val selector = new TargetingIdeaSelector()
    selector.setRequestType(RequestType.STATS)
    selector.setIdeaType(IdeaType.KEYWORD)

    selector.setRequestedAttributeTypes(Array(AttributeType.KEYWORD_TEXT, AttributeType.SEARCH_VOLUME,
      AttributeType.CATEGORY_PRODUCTS_AND_SERVICES, AttributeType.AVERAGE_CPC, AttributeType.COMPETITION))
    val paging = new Paging()
    paging.setStartIndex(0)
    paging.setNumberResults(2500)
    selector.setPaging(paging)
    val relatedToQuerySearchParameter = new RelatedToQuerySearchParameter()
    relatedToQuerySearchParameter.setQueries(keywords)

    val languageParameter = new LanguageSearchParameter()
    val language = new Language()
    language.setId(language_id.toLong)
    languageParameter.setLanguages(Array(language))
    val locationParameter = new LocationSearchParameter()
    val location = new Location()
    location.setId(location_id.toLong)
    locationParameter.setLocations(Array(location))

    selector.setSearchParameters(Array(relatedToQuerySearchParameter, languageParameter, locationParameter))
    val page = targetingIdeaService.get(selector)
    var resultKeywords = Map[String, Map[String, Any]]()
    if (page.getEntries != null && page.getEntries.length > 0) {

      for (targetingIdea <- page.getEntries) {
        val data = Maps.toMap(targetingIdea.getData)
        val keyword = data.get(AttributeType.KEYWORD_TEXT).asInstanceOf[StringAttribute]
        val categories = data.get(AttributeType.CATEGORY_PRODUCTS_AND_SERVICES).asInstanceOf[IntegerSetAttribute]
        var categoriesString = "(none)"
        if (categories != null && categories.getValue != null) {
          categoriesString = categories.getValue.toJson.prettyPrint
        }
        val averageMonthlySearches = data.get(AttributeType.SEARCH_VOLUME).asInstanceOf[LongAttribute]
          .getValue
        val cpc = data.get(AttributeType.AVERAGE_CPC).asInstanceOf[MoneyAttribute]

        val money = cpc.getValue
        var averageCpc = 0.0
        if (money != null) {
          averageCpc = money.getMicroAmount.toDouble / 1000000.0
        }
        var result: Map[String, Any] = Map("search_volume" -> averageMonthlySearches.toString(), "categories" -> categoriesString, "average_cpc" -> averageCpc.toString())
        resultKeywords = resultKeywords + (keyword.getValue() -> result)
      }
    } else {
      println("No related keywords were found.")
    }
    return resultKeywords
  }
  def runTrafficEstimatorQuery(adWordsServices: AdWordsServices, session: AdWordsSession, keywords: Array[String], location_id: Integer, language_id: Integer): Map[String, Map[String, Any]] = {

    val keywordEstimateRequests = new ArrayList[KeywordEstimateRequest]()

    for (keywordText <- keywords) {
      val keyword = new Keyword();
      keyword.setText(keywordText);
      keyword.setMatchType(KeywordMatchType.BROAD);
      val keywordEstimateRequest = new KeywordEstimateRequest();
      keywordEstimateRequest.setKeyword(keyword);
      keywordEstimateRequests.add(keywordEstimateRequest);
    }
    val adGroupEstimateRequests = new ArrayList[AdGroupEstimateRequest]()
    val adGroupEstimateRequest = new AdGroupEstimateRequest()
    adGroupEstimateRequest.setKeywordEstimateRequests(keywordEstimateRequests.toArray(Array()))
    adGroupEstimateRequest.setMaxCpc(new Money(null, 600000L))
    adGroupEstimateRequests.add(adGroupEstimateRequest)

    var campaignEstimateRequests = Array[CampaignEstimateRequest]()
    val campaignEstimateRequest = new CampaignEstimateRequest()
    campaignEstimateRequest.setAdGroupEstimateRequests(adGroupEstimateRequests.toArray(Array()))

    val location = new Location()
    location.setId(location_id.toLong)
    val language = new Language()
    language.setId(language_id.toLong)
    campaignEstimateRequests = campaignEstimateRequests :+ campaignEstimateRequest
    val selector = new TrafficEstimatorSelector()
    selector.setCampaignEstimateRequests(campaignEstimateRequests)

    val trafficEstimatorService = adWordsServices.get(session, classOf[TrafficEstimatorServiceInterface])
    val result = trafficEstimatorService.get(selector)

    var resultKeywords = Map[String, Map[String, Any]]()
    var index = 0
    for (
      campaignEstimate <- result.getCampaignEstimates; adGroupEstimate <- campaignEstimate.getAdGroupEstimates;
      keywordWithEstimate <- adGroupEstimate.getKeywordEstimates.zip(keywords)
    ) {
      var keyword = keywordWithEstimate._2
      var keywordEstimate = keywordWithEstimate._1
      val min = keywordEstimate.getMin
      val max = keywordEstimate.getMax
      val avg_total_cost = (max.getTotalCost.getMicroAmount + min.getTotalCost.getMicroAmount) / 2000000.0
      var result: Map[String, Any] = Map("avg_ctr" -> (min.getClickThroughRate + max.getClickThroughRate) / 2,
        "keyword" -> keyword, "avg_total_cost" -> avg_total_cost.toString(),
        "clicks_per_day" -> Array(min.getClicksPerDay, max.getClicksPerDay),
        "total_cost" -> Array(min.getTotalCost.getMicroAmount / 1000000, max.getTotalCost.getMicroAmount / 1000000),
        "average_position" -> Array(min.getAveragePosition, max.getAveragePosition))
      resultKeywords = resultKeywords + (keyword -> result)
    }
    return resultKeywords
  }

Logging With Docker

| Comments

There is great article how to configure centralized logging for docker with cloudwatch:

1

https://blogs.aws.amazon.com/application-management/post/TxFRDMTMILAA8X/Send-ECS-Container-Logs-to-CloudWatch-Logs-for-Centralized-Monitoring

It assumes that the application can send logs to other container running rsyslog. We will extend the example to work with locally saved logs. In order to make it work we need to:

1) Create one container that will configure cloudwatch and rsyslog that listen to the events.

2) Configure rsyslog on second devise that will listen to file changes in logs folder and will send logs to second rsyslog service running on another machine

I decided to create another rsyslog agent running together with my application because I’m reusing the docker cloudwatch container.

First lets remove /dev/xconsole from rsyslog config. It breaks the rsyslog because most docker images does not have dev/xconsole. Open aws cloudwatch ecs project and edit Dockerfile

1

RUN rm /etc/rsyslog.d/50-default.conf

Next we need to configure rsyslog on our app machine. We need rsyslog version > 8.15.0 to use wildcard imfile as a log source(see rsyslog.conf):

1
2
3
4
5
6
7
8
9

RUN apt-get update
RUN apt-get -y install libestr-dev  liblogging-stdlog-dev  libgcrypt-dev uuid-dev libjson0-dev  libz-dev
RUN wget http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.16.0.tar.gz
RUN tar xvzf rsyslog-8.16.0.tar.gz
RUN cd rsyslog-8.16.0 && ./configure --enable-imfile && make && make install
COPY rsyslog.conf /etc/rsyslog.conf
RUN pip install supervisor
COPY supervisord.conf /usr/local/etc/supervisord.conf
CMD ["/usr/local/bin/supervisord"]

supervisord.conf is pretty simply:

1
2
3
4
5
6
7
8

[supervisord]
nodaemon=true

[program:rsyslogd]
command=/usr/local/sbin/rsyslogd -n

[program:awslogs]
command=your-app-with-logs

rsyslog.conf sends logs written to file to rsyslog on machine running cloudwatch agent

1
2

module(load="imfile")
input(type="imfile" file="/usr/src/app/logs/location/*.log" tag="spider_locations" facility="local6")

Modifing AWS Gateway Output Parameters

| Comments

Maybe you want to hide unnecessery parameters from api or you want to modify the keys. It’s straightforawrd with amazon gateway but it took me some time to find how to do it. Go to AWS Gateway -> Your api -> Integration Response -> Response template and change “passthru” to sth like

1
2
3
4

#set($inputRoot = $input.path('$'))
{
  "listings" : $inputRoot.items
  }

This will strip all other keys in a response dict and replace it with one key that’s the value of original ‘items’ key. Whoila!

Raspberry PI as a Home Hotspot Powered by Telekom Hotspot

| Comments

If you are living in Germany and you have phone plan covering free Telekom Hotspot usage probably you wondered, like me if its possible to broadcast signal from hotspot to home. One function that prohibits us from using simple reapeter is the fact that Telekom hotspot has html username/password authentication. The hotspot deuthenticates the user when inactive and when token expires. The idea is to put 2 WIFI cards into raspberry PI. One will receive signal from hotspot. On another we will configure hostapd and dhcp server to broadacast signal at home. We will write simple script that authenticates user in Telekom hotspot using curl.

  1. Step is to get raspberry PI. I’m using the B+ model that has 4 USB exists and microSD slot.
  2. Install Raspbian OS
  3. Configure raspberry PI
1
2

sudo apt-get update
sudo apt-get -y install hostapd isc-dhcp-server iptables wpa_supplicant

edit the following files: /etc/hostapd/hostapd.conf, /etc/dhcp/dhcpd.conf, /etc/default/isc-dhcp-server, /etc/network/interfaces: 4. Enable packet forwarding on start by adding the line “net.ipv4.ip_forward=1” to “ /etc/sysctl.conf” 5. Configure IP-Tables(I assume wlan1 is a interface connected to Telekom Hotspot.

1

sudo iptables -t nat -A POSTROUTING -o wlan1 -j MASQUERADE
1

sudo iptables -A FORWARD -i wlan1 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
1

sudo iptables -A FORWARD -i wlan0 -o wlan1 -j ACCEPT
  1. Save the rules:
1

sh -c iptables-save > /etc/iptables.ipv4.nat
  1. Run dhcp server and hostapd:
1

sudo update-rc.d hostapd enable
1

sudo update-rc.d isc-dhcp-server enable
  1. You should be able now to connect to raspberry PI wifi network(wlan0) from home. But no internet yet.
  2. Now connect to telekom hotspot:
1

sudo iwconfig wlan1 essid Telekom; sudo dhcpclient wlan1
  1. Move to your laptop and perform http Login with your Telekom username and password. In Chrome open console and in a network tab find the post to Telekom gateway. Click on that request and click “save as curl command” paste the command to a script and save on raspberry. Then write short script that will check internet connection and if there is no it will reauthenticate. Sth like this
1
2
3
4
5

#!/bin/sh
while [ 1 ]; do
    sudo ping www.google.com -c 1 && echo "OK" || curl -v 'https://hotspot.t-mobile.net/wlan/rest/login' -H 'Origin: https://hotspot.t-mobile.net' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, */*' -H 'Referer: https://hotspot.t-mobile.net/TD/hotspot/de_DE/index.html?origurl=http%3A%2F%2Fwww.kimeta.de%2FOfferDetail.aspx&ts=1451396580457' -H 'Cookie: 06847EADCEBFD2E920C1F12378924815.P2; oam.Flash.RENDERMAP.TOKEN=118ogwrec4' -H 'Connection: keep-alive' --data-binary '{"username":"phone-number@t-mobile.de","password":"password","rememberMe":true}' --compressed
    sleep 1
    done

Put the script in monit or run with nohup in raspberry pi

AWS Lambda. The Bad Parts

| Comments

Amazon about their Lambda service: “AWS Lambda is a zero-administration compute platform for back-end web developers that runs your code for you in the AWS cloud and provides you with a fine-grained pricing structure” I spend some time with a microservice-awslambda architecture. Here I share numerous problems I encountered: 1) Testing/Staging env requires seperate lambda function. We can not deploy same lambda function with different version to different APIs. 2) The Lambda environment is archaic: Node v0.9 without ECMA6 support and Python 2.7. 3) Difficult to deploy Python with shared libraries. When deploying python function with precompiled shared library we have to build it on same architecture like AWS Lambda, otherwise it won’t work. We can not simply deploy liblxml build on MacOSX. In order to compile linked libraries it’s necessery to build them on ec2 instance and then deploy to Lambda. Seems like too much work for simple task.

Conclusion

There are great benefits in an architecture build on Lambda functions. Unfortunately it comes in a costs. It might work well for very simple services but when we need to create anything more complicated than ‘hello world’ its wise to move to other solution. Maybe Dockeris a thing to go. If you want to share your thoughts about Lambda AWS architecture please comment.

Comparison of PDF Generation Libraries

| Comments

Recently I investigated the possibilities of PDF generation from doc(x) documents. It came up that its not that trivial. The goal is to have separate application that will have api to perform conversions.

Example Usage:

Post: docx2pdf.tmh.io/templates/create?filename=name.docx # status: ok Get: doc2pdf.tmh.io/templates/1/pdf?placeholder1=value1&placeholder2=value2 #document.pdf