mbeddr.arduino
Today a gave a little talk about my spare time project ‘mbeddr.arduino’. I gave brief overview what’s the goal and it current state. The slides can be found here.
Installing Mbeddr on Ubuntu 13.04
This guide covers how to build and install mbeddr from source, it does not include how to install the fancy model checking tools.
After you have installed Ubuntu 13.04 you will notice that it already ships with Java 7. Since MPS only works with Java 6 you need to downgrade it. The problem is that the official package repository of Ubuntu does not contain a Oracle Java version anymore, there is OpenJDK 6 but it will not work with properly MPS. You can go to Oracles download page and install it on your own but that is pain in the ass. Luckily there is a unofficial repository that can do all the that for you. To add the repository and install Java run this commands:
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java6-installer
This gives you a working Java 6 but you need to make it the default Java on your machine. This is done with:
sudo apt-get install oracle-java6-set-default
Now you can install MPS from the JetBrains page. Extract the tar.gz to your favorite location. In order to be able to build and run mbeddr you will need some additional stuff: ant, gcc, git, make and graphvis. To install them run:
sudo apt-get install ant git-core graphviz build-essential
After that go to the location where you want to put the mbeddr source code and clone the git repository with:
git clone https://github.com/mbeddr/mbeddr.core.git
Once that is finished set up the build properties file in /path/to/mbeddr.core/code/languages/ there is an example file in the same location that should give you a good point to start from. The finial configuration should look something like this:
mbeddr.github.core.home=/build/mbeddr.core/
mps.platform.caches=/build/mps-ant-caches/
# MPS installation
mps.home=/build/mps/
To build mbeddr run:
./rebuildLanguagesAfterCheckout.sh
in the same location.
When that is finished start your MPS and navigate to File -> Settings, there you select GlobalLibraries and create a new variable called mbeddr.core which points to path/to/mbeddr.core/code, save the setting and you are done.
This should give you a working basic mbeddr setup, to verify that it works you can open the tutorial project from the mbeddr page and run some of the samples.
Note: You will not be able to run any of the formal verification stuff, since we did not install them yet. I will extend the guide in the near future to cover that too.
Trainmonitor: The API and Why It Sucks
Recently I have published my Trainmonitor app, that is using a backend server to get the train data and do the notifications. To communicate with the server I have an HTTP API, some people might call it a RESTful API, but we will see later that this is not the case. There are several issues with the API, part of the done intentionally done wrong and others not. So why would I do stuff wrong if I knew it was wrong? Do demonstrate what’s wrong and to show what the benefit is when you do a real RESTful HTTP API with hypermedia. The idea about this came to me when I ret this blog post about HTTP APIs at XING. What the API currently is, is some kind of remote method invocation over HTTP.
So what’s wrong with the API?
Let’s have a look at some URIs:
http://trainmonitor.logv.ws/api/v1/germany/stations
http://trainmonitor.logv.ws/api/v1/germany/trains
At first you can see there is some kind of version in the URI. In a RESTful world this should not be needed. The data the API responses should guide the client through the resources. There should just be a single URI you need to know to discover the hole API. We will talk about this later, so lets have a look at a typical request - response: For testing purpose I am issuing a request from Chrome to http://trainmonitor.logv.ws/api/v1/germany/trains:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
User-Agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
The server response headers looks like this:
Cache-Control:public, max-age=105
Content-Encoding:gzip
Content-Length:28515
Content-Type:application/json
Date:Tue, 08 Jan 2013 20:41:44 GMT
Keep-Alive:timeout=15,max=100
Server:Mono-HTTPAPI/1.0
[{
"train_nr": "ICE 576",
"status": 4,
"stations": null,
"started": null,
"nextrun": "2013-01-10T12:21:00Z",
"finished": null
}, {
"train_nr": "ICE 37",
"status": 4,
"stations": null,
"started": null,
"nextrun": "2013-01-10T14:23:00Z",
"finished": null
}, {
"train_nr": "ICE 909",
"status": 1,
"stations": null,
"started": "2012-12-08T09:30:01.742Z",
"nextrun": "2012-12-08T15:49:00Z",
"finished": "2012-12-08T15:56:52.916Z"
}]
According to the header we got some gzip compressed JSON that can be cached for the next 105 seconds. But we don’t really know what the response contains, except it is some kind of JSON. This means the consumer needs more out of band knowledge to be aware how to process the data. When we look at the payload, the JSON, we see there are no links, but some IDs. If the client now wants to access other resources that have a relationshilt to this one the client needs to know where to find this resource. Once again out of band knowledge is needed.
How to fix the mess?
Lets start with the problem that the client needs to know the complete URI layout of the server. Wouldn’t it be nice if the client only needs to know a single URI?. This URI might be http://trainmonitor.logv.ws/api/ and the server responds with something like this:
Cache-Control:public, max-age=86400
Content-Encoding:gzip
Content-Length:28515
Content-Type:application/vnd.logv.trainmonitor-discovery+json
{"links":[{"rel":"germany","href":"http://trainmonitor.logv.ws/api/germany"}]}
As you can see the Content-Type is now application/vnd.logv.trainmonitor-discovery+json and you might notice that I am using the non standard +json suffix. This is currently in draft status to update the +xml RFC. My mediatype vnd.logv.trainmonitor-discovery is a ripoff of the html link element, extended with some more semantic for the rel element. The client now knows that the data is formatted as JSON and what this JSON means. If the client wants to get more information about germany it knows that it should issue a GET request to http://trainmonitor.logv.ws/api/germany. And the server should answer with:
Cache-Control:public, max-age=86400
Content-Encoding:gzip
Content-Length:28515
Content-Type:application/vnd.logv.trainmonitor-discovery+json
{
"links": [{
"href": "http://trainmonitor.logv.ws/api/germany/staions",
"rel": "stations"
}, {
"href": "http://trainmonitor.logv.ws/api/germany/trains",
"rel": "trains"
}]
}
As before the Content-Type is application/vnd.logv.trainmonitor-discovery+json, but we see some new rel data. When the client parses the response data it finds the information where to find the stations and trains for germany. All responses contain a Cache-Control header with a value of 864000 seconds, this allows the client or any other cache on the way from the origin server to the client to cache this resource for a day. This way the origin server is not hammered with requests for discovering the resource layout and the clients can reuse a fresh resource without contacting the origin server, but after at least 24 h they will ask you again. Why is this good instead of hardcoding the URIs? Once your services grows you might experience more load on the server and single server might not be able to answer all the requests in reasonable time. So you need to scale horizontally. To do this you can reroute all the request for a single country to another server. Or you can have single server for stations and one for trains or mixing it all together. E.g. having a single server for the stations of all countries but one server for each countries trains, because this data changes much more often and is more interesting to the clients. To do so no change on the clients is required, all of them will use the new infrastructure without touching them.
What about Links?
After that we have talked about the URIs and how the client gets to know them, lets have a look at the resources and its representation. We start with the stations from http://trainmonitor.logv.ws/api/v1/germany/stations. A typical response looks like this:
Cache-Control:public, max-age=86400
Content-Encoding:gzip
Content-Length:28515
Content-Type:application/json
[{
"lat": 47.5500209,
"lon": 14.3122344,
"name": "Selzthal",
"id": 380
}, {
"lat": 52.5104217,
"lon": 13.4969506,
"name": "Berlin-Lichtenberg",
"id": 394
}]
What’s wrong here? The Content-Type! From the header the client just knows that it is some kind of JSON, but has no clue whats the containing JSON means. The client would need to know that this particular URI returns a list station objects. To fix this, the Content-Type might look like this application/vnd.logv.trainmonitor-stations+json. And what about the Cache-Control? 24 hours freshness??? Yes, this is valid, because of the fact that the backend server only refreshes the station list, from its data source, once a day this resource can be cashed a full day.
Now lets move on to the more interesting trains. A response from http://trainmonitor.logv.ws/api/v1/germany/trains looks like this:
Cache-Control:public, max-age=105
Content-Encoding:gzip
Content-Length:28515
Content-Type:application/json
[{
"train_nr": "ICE 576",
"status": 4,
"stations": null,
"started": null,
"nextrun": "2013-01-10T12:21:00Z",
"finished": null
}, {
"train_nr": "ICE 37",
"status": 4,
"stations": null,
"started": null,
"nextrun": "2013-01-10T14:23:00Z",
"finished": null
}, {
"train_nr": "ICE 909",
"status": 1,
"stations": null,
"started": "2012-12-08T09:30:01.742Z",
"nextrun": "2012-12-08T15:49:00Z",
"finished": "2012-12-08T15:56:52.916Z"
}]
Once again the Content-Type is too general and should be more precise like application/vnd.logv.trainmonitor-trains+json. The Cache-Control is now much shorter, why? The server refreshes trains on schedule and calculates the time to the next refresh when a client issues a GET request. Then it sets the Cache-Control header according to this value. Now a closer look at the JSON content and why this is clearly not a hypermedia type. Why is the stations property always null? This requires some explanation … In the domain data model the stations property is a list off all stations of each train with arrival, departure and delay times. A complete list of all trains contains about 1300 trains. Also the client is not interested in all the trains and its delays. That is why I decided to omit this value in the train list and only include it when a single train is requested. But how do we get this data? Good question, once again the client need out of band information that is not part of the conversation with the server. The URI template for a train looks like this http://trainmonitor.logv.ws/api/v1/germany/trains/{train_nr}. For the ICE 909 this would be http://trainmonitor.logv.ws/api/v1/germany/trains/ICE%20909. In order to solve this issue we need to include this information in the reposen. A response might look like this:
{
"train_nr": "ICE 909",
"link": {
"href": "http://trainmonitor.logv.ws/api/v1/germany/trains/ICE%20909",
"rel": "train"
}
"status": 1,
"stations": null,
"started": "2012-12-08T09:30:01.742Z",
"nextrun": "2012-12-08T15:49:00Z",
"finished": "2012-12-08T15:56:52.916Z"
}
Simply by addeing a link attribute the client knows where to find the resource. But still the stations attribute is null, so lets fix that in a similar way:
{
"train_nr": "ICE 909",
"link": {
"href": "http://trainmonitor.logv.ws/api/v1/germany/trains/ICE%20909",
"rel": "train"
}
"status": 1,
"stations": {
"href": "http://trainmonitor.logv.ws/api/v1/germany/trains/ICE%20909/stations",
"rel": "stationdetails"
}
"started": "2012-12-08T09:30:01.742Z",
"nextrun": "2012-12-08T15:49:00Z",
"finished": "2012-12-08T15:56:52.916Z"
}
Just like before a link element lets client know where to find it. The other resources contain similar fails, so I will not discuss them here, it would get boring for you to read the same over and over again.
Why should I use hypermedia?
This is easy! Because you gain a lot of flexibility and get a loosely coupled system. But is it worth to do so? Isn’t the client getting much more complex?
I will compare which information the client need before and after using hypermedia.
Before
- data type for trains
- data type for stations
- data type for stationdetails
- URI template for trains
- URI template for stations
- URI template for a single train
- URI template for stations of a train
- connection between each URI (template) and its datatype
To visualize this I drew a little diagram with the direct relations between the resources.
After
- data type for trains
- data type for stations
- data type for stationdetails
- data type for discovery
- a single URI
And that is how the diagram now looks:
As you can see all the resources now have explicit connections, instead of beeing just two independant islands, that were connected with out of band knowledg.
Some people might argue that implementing a client that can follow links and parse a slightly more complex data type requires more work to. That might be true! But is it really that much more work to do? Using a URI from an resources attribute or formatting a attribute of a resource into a template is not that much different!
But you must put more work into the design of your resources and decouple them from their representation. The task is not: how do publish my domain model as JSON or XML It is: how should the resources for my domain model look like and how is their relationship to each other. After you have an abstract model of your resources you can think of how to represent them. Depending on your needs you might have a single data type for your whole domain. Which is often the case when you use XML, because XML offers more semantics than, e.g. JSON. But you can either find ways to do so in JSON. On the other hand you can use a data type for each kind of resource.
OK, but where is the API?
While I was writing this post I decided not to publish a API documentation, right now! As you can see there are several issues with it. I wrote this post to show other people in which traps I have gone and how fix / avoid them. I will redesign the API in the next weeks and reimplemt the client side completly. After that I will publish a complete API documentation and open the API for everyone. In the meantime you can have a look at the trainmonitor android app which just got open sourced!
Trainmonitor Goes Open Source
The trainmonitor / Zugmonitor Android App is now open source. Like I have planned from the beginning. Currently I only open sourceed the Android App. The server side will follow as soon as I have done some code clean ups! But it can take a while!
The App is licensed under the Apache 2 license and the source code can be found on Github.
I am developing with IntelliJ and the project has the folowing dependencies that are not part of the source code, but all open source or available from the Android SDK:
- basic-http-client
- Googles GCM client library
- Google Play Services
- GSON
- ormlite
- EventBut
- android-support-v4
If you would like to contribute feel free to cantact me of fork the app on Github.
EventBus for Android
The new version of trainmonitor adds some new features like syncing multiple devices. Which was one of the most challenging features to implement. The app uses a lot of threading to do nearly everything except UI drawing off the main thread. This required a lot of callbacks and it really started to get messy when the same action should be triggered from different sources. To illustrate some of the issues I had we can have a look at the start up of my app. To illustrate it I have drawn some state diagram: 
As you can see there are several way through the application. And here is a typical peace of code:
private void chooseAccount(final MainActivity that)
{
if(!Installation.wasChooseAccountShown(this))
{
new ChooseAccountFragment(new Runnable() {
@Override
public void run() {
Installation.setChooseAccountShown(that);
init();
}
}).show(getSupportFragmentManager(), "choose_account");
}
else {
init();
}
}
Both paths though the code call the same init function. The init function is the point where the menu is set up, the check for a valid Google Cloud Messaging registration and if need the registration is started. Now there nothing really wrong with it but it tends to get a bit messy when I wanted to extend the logic that is performed when the disclaimer was shown. Since I am doing a lot of Servicebus related work in my daily work for a client I searched if there is something similar for android. And indeed there is a android solution for it, EventBus. After some refactoring the code now looks like this:
@SuppressWarnings("UnusedDeclaration")
public void onEvent(DisclaimerAcceptedEvent event)
{
if(!Installation.wasChooseAccountShown(this))
{
final MainActivity that = this;
new ChooseAccountFragment(new Runnable() {
@Override
public void run() {
Installation.setChooseAccountShown(that);
mBus.post(new AccountChoosnEvent());
}
}).show(getSupportFragmentManager(), "choose_account");
}
else
{
mBus.post(new AccountChoosnEvent());
}
}
Now the code does not look that much different, the explicit calls to ‘init’ where replaced with a post on the EventBus and the function itself just listens to a Event and is then called by the EventBus. But you can now add easily new functionality that is called when the event occurs. This especially useful when triggering the events from different sources, e.g. refreshing a list of subscriptions from the UI or by a message from the cloud.
Introducing logv.http - a Event Driven Webserver for .net
While I was working on the trainmonitor project I need a back-end server that could handle the HTTP API. The normal .Net way would be to use ASP.Net and do it with MVC, but since my back-end should run on Mono I don’t wanted to go that painful road with Mono and decided to do some stuff on my own.
That’s when logv.http was born a simple webserver written in .Net. It is not fully written by my own it is mostly a wrapper around the HttpListener class in the framework but it takes away many of the annoying work from you with a simple API and does not have the large footprint like using ASP.
Creating a “hello world” looks like this
var server = new Server("localhost", 13337);
server.Get("http://localhost:13337" ,(req, res) => res.Write("Hello World!"));
server.Start();
But the project contains many more stuff like, helpers for setting caching responding with JSON serialized objects. To do so simply replace the second line with:
server.Get("http://localhost:13337" ,(req, res) => res.WriteAsJson(new {Hello = "world"}));
And the server will response with a JSON string. There are also helpers for all kinds of error messages and other response codes, simply try it out!
You can find the NuGet Package here. And of course everything is open source under the Apache 2.0 license. The source code is on Github. If you want to report bugs or implement features just sent me a pull request or create an issue there.
Event Driven Webserver With F#
I am always open to experiemt with new stuff, so this time I did something I wanted to do for a long time: play a bit with F#. I read some book about it in the past, but never had the chance to really do something with it. I was always fascinated by the expressiv code that you are able to create with it. Since my current spare time poject is using a webserver I thought it would be a nice idea to implement some stuff for it with F#.
I am now rewriting the hole server component of the trainmonitor in F# to get a feeling for it. Here is the first part of code I wrote that is nothing more that a wrapper around the HttpListener of the .Net framework. I have written similar stuff in C# to get a async event based server for C#, but what I alway felt ugly about was the way the controlflow was distibuted all over the place. Wouldn’t it be nice to just write code like this:
server.Get |> Observable.add greeting
server.Get |> Observable.add (fun(ctx) -> response ctx |> write "hello world" |> close)
open logv.http
[<EntryPoint>]
let main argv =
let server = new Server("localhost", 13337)
server.Start()
server.Get |> Observable.add (fun(ctx) -> response ctx |> write "hello world" |> close)
System.Console.ReadLine() |> ignore
server.Stop()
namespace logv
module http =
open System.Net
/// <summary>Extracts the response from HttpListenerContext</summary>
/// <param name="ctx">The HttpListenerContext to extract the response from</param>
///<returns>The HttpListenerResponse from the Context</returns>
let response (ctx : HttpListenerContext) = ctx.Response
/// <summary>Extracts the reqeust from HttpListenerContext</summary>
/// <param name="ctx">The HttpListenerContext to extract the request from</param>
///<returns>The HttpListenerRequest from the Context</returns>
let request (ctx : HttpListenerContext) = ctx.Request
let body (req : HttpListenerRequest) = async {
let decode b = req.ContentEncoding.GetString(b)
let stream = req.InputStream
let! bytes = stream.AsyncRead((int)stream.Length)
return decode bytes
}
let writeCode501 (res : HttpListenerResponse) =
res.StatusCode <- 501
res
let write (data : string) (res : HttpListenerResponse) =
let bytes = System.Text.Encoding.UTF8.GetBytes(data)
let stream = res.OutputStream
stream.Write(bytes, 0 , bytes.Length)
res
let requestUrl (ctx :HttpListenerContext) = ctx.Request.RawUrl
let endsWith(pattern : string) (target : string) = target.EndsWith(pattern)
let startsWith(pattern : string) (target : string) = target.StartsWith(pattern)
let contains (pattern : string) (target : string) = target.Contains(pattern)
let matches (pattern : string) (target : string) = System.Text.RegularExpressions.Regex.IsMatch(target, pattern)
let close (res : HttpListenerResponse) = res.Close()
type Server(host : string, port : int) = class
let cts = new System.Threading.CancellationTokenSource()
let listener = new HttpListener()
let getEvent = new Event<_>()
let postEvent = new Event<_>()
let putEvent = new Event<_>()
let deleteEvent = new Event<_>()
let patchEvent = new Event<_>()
let handleContext(ctx :HttpListenerContext) =
match ctx.Request.HttpMethod with
| "GET" -> getEvent.Trigger(ctx)
| "POST" -> postEvent.Trigger(ctx)
| "PUT" -> putEvent.Trigger(ctx)
| "DELETE" -> deleteEvent.Trigger(ctx)
| "PATCH" -> patchEvent.Trigger(ctx)
| _ -> response ctx |> writeCode501 |> close
let mailbox = MailboxProcessor.Start(fun inbox ->
let loop =
async {
while true do
let! ctx = inbox.Receive()
handleContext ctx
}
loop
)
let rec requestLoop(token : System.Threading.CancellationToken) = async {
let! context = Async.FromBeginEnd(listener.BeginGetContext, listener.EndGetContext)
mailbox.Post(context)
if token.IsCancellationRequested = false then
return! requestLoop(token)
}
member val Host = host with get
member val Port = port with get
///Starts the webserver
member this.Start() =
listener.Prefixes.Add("http://" + host + ":" + port.ToString() + "/")
listener.Start()
Async.Start(requestLoop(cts.Token))
///Stops the webserver
member x.Stop() =
cts.Cancel()
listener.Stop()
///Event for handling a GET request
member x.Get = getEvent.Publish
///Event for handling a POST request
member x.Post = postEvent.Publish
///Event for handling a PUT request
member x.Put = putEvent.Publish
///Event for handling a DELETE request
member x.Delete = deleteEvent.Publish
///Event for handling a PATCH request
member x.Patch = patchEvent.Publish
end
Trainmonitor
In meiner Freizeit habe ich ein kleine Projekt gestartet das ich nun endlich in die Freiheit entlassen möchte. Es richtet sich vorallem an deutschsprachige Pendler die die Deutsche Bahn nutzen. Mehr dazu findet ihr auf einer eigenen Seite hier im Blog, auf der ich den Status des Projekts immer aktuell halten werden.
Zu Trainmonitor geht es hier.
Forcing a HttpWebRequest to IPv6
In my current project I have a use case where I want to connect to a HTTP server via IPv6, but the client and server also have IPv4 connectivity. Which leads to the problem that the .Net framework chooses the IPv4 address over the IPv6 address. I could enable round robin in the ServicePointManger config, but that would only cycle through all the addresses returned by the DNS request.
So I need a another way to solve the problem. I stumbled up on a post on stackoverflow about how to control the address from which the connection is established. Starting from there I had a look at the framework code which establishes the connection to the ServicePoint which is located in the System.Net.ServicePoint class method “ConnectSocketInternal” this method loops through all the IP Addresses returned by the DNS subsystem. The code skips to the next ip if something goes wrong while connecting. As the post on stackoverflow points out we can add our own delegate to the ServicePoint to intercept this connection request. Enforcing IPv6 is quite easy to do:
var req = HttpWebRequest.Create("http://google.com") as HttpWebRequest;
req.ServicePoint.BindIPEndPointDelegate = (servicePount, remoteEndPoint, retryCount) =>
{
if (remoteEndPoint.AddressFamily == System.Net.Sockets.AddressFamily.InterNetworkV6)
{
return new IPEndPoint(IPAddress.IPv6Any, 0);
}
throw new InvalidOperationException("no IPv6 address");
};
This code simply throws if a IPv4 address is passed to connect and the ServicePoint class will try the next address. But be aware it only works in use casses where both, the server and the client, are reachable via IPv6 and the server provides a AAAA record for the hostname.
Zugmonitor CSharp Bindings
Da sich der Inhalt dieses Posts hauptsächlich an deutschsprachige Entwickler richtet verfasse ich diesen Post auf Deutsch. Es wird einer von mehreren sein, da ich dieses Thema in nacher Zukunft etwas ausführlicher bearbeiten werde.
Durch Zufall bin ich bei der suche nach einer API für die Zugdaten der Deutschen Bahn auf ein sehr interessantes Projekt der Süddeutschen Zeitung (SZ) gestoßen. Es handelt sich um den Zugmonitor, der Verspätungen in nahe zu Echtzeit visualisiert und sehr erfreulicherweise auch eine API bereitstellt um mit diesen Daten zu arbeiten. Also habe ich schnell ein paar Minuten Freizeit geopfert und eine CSharp Assembly geschrieben die einen einfachen Zugriff auf die Daten ermöglicht. Das ganze ist eine Portable Class Library, die mit .Net 4, Silverlight 5 oder WinRT funktioniert. Da die Datenmenge pro Abfrage der Züge relativ groß (>1MB) sein kann habe ich ganz bewusst eine Windows Phone Kompatibilität weggelassen. Für eine Reduzierung der Datenmenge und noch einiges mehr habe ich ein anderes Projekt in der Pipeline, aber dazu mehr wenn es fertig ist ;-). Der Quellcode ist auf Github unter Apache 2 Lizenz verfügbar und die Binaries gibt es bei dort auch. Einzige Abhängigkeit ist der JSON Serialisierer von Json.Net.
Bitte vergesst nicht das die API nur nicht kommerzielle Nutzung bereit gestellt wird und das die Datenquelle als “Zugmonitor von süddeutsche.de/OpenDataCity” genannt werden soll. Außerdem bittet das Team darum dass Ihr in einer kurzen Mail den Link zu eurem Projekt kundtut.