I was a big fan of "Rent a bike"-offers in germany for the longest time. Nowadays it's not as comfortable as it was a couple of years ago, because of the fee you have to pay if you park a bike at a location other than the predefined ones. Nonetheless it's a fun project for me. (BTW: It seems that the rest of the world uses a service from CityBikes which is already visualized by Ramnath Vaidyanathan)
In a night shift, in mid 2012, I played around with a couple of apps on my mobile, a http proxy (mitmproxy) and tcpdump to find a way to MITM some of these apps.
After a dozen of apps (most of them failed because of strict certificate checks on the HTTPS front) and way to much coffee, my shift was going to an end. So I started my favorite bike-renting app and looked for the nearest bike which should bring me home. After finding one, I noticed that I still had a proxy configured on my phone and that the app didn't even remotely complain about the wrong certificate (this is fixed now). Jackpot ... I thought ... and went home to bed because it was 8am and I was tired enough. But that should be a good starting point for some "data collection and visualization".
The next night started with some traffic analyzing and a fast discovery of a pair of credentials for the (more or less) public API endpoint. "More or less", because it needs some very basic authentication, based on your mobile device. You'll know what I mean if you look at this snippit from the Ik4ru5/pyABike repository.
userData.User = 't_cab_android' #from android app
userData.Password = 'DELETED TO PREVENT PROBLEMS' #from android app
Source: Github
And since I have an iPhone, the credentials on my phone are a bit different. Here is a SOAP-Request I use to gather my data. And since I only need one API endpoint for my queries, I didn't start bothering with a python SOAP library and just copy/paste the query from my phones traffic.
def createRequest(self, lati, longi, radius, amount):
""" Returns an XML-Set of all the bikes around the specific coordinates """
data = """<?xml version="1.0" encoding="utf-8"?>
<SOAP:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP:Body>
<foo.listFreeBikes xmlns="https://xml.dbcarsharing-buchung.de/hal2_cabserver/">
<CommonParams>
<UserData>
<User>t_iphone_cab</User>
<Password>DELETED TO PREVENT PROBLEMS</Password>
</UserData>
<LanguageUID>1</LanguageUID>
<RequestTime>%s</RequestTime>
<Version>2</Version>
</CommonParams>
<SearchPosition>
<Longitude>%.6f</Longitude>
<Latitude>%.6f</Latitude>
</SearchPosition>
<maxResults>%i</maxResults>
<searchRadius>%i</searchRadius>
</foo.listFreeBikes>
</SOAP:Body>
</SOAP:Envelope>"""% (datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"), longi, lati, amount, radius)
return data
Do you see the professional consistency of the usernames? "t_cab_android" vs "t_iphone_cab". Amazing ...
To make it clear: These credentials are only useful to query the backend for bike-data and not user-data.
Nonetheless, I used a couple selfmade python modules to gather this data periodically and save it to a database.
The application logic is very simple:
Look at a certian position (longitude, latitude) for bikes in a range of n meters. If a bike is found, check the database for an entry of that bike. If we don't have an entry for that bike, we've never seen it before and create a new entry. If - on the other hand - we know that bike already, we set the current position and add the last position to the history.
With this little logic we can keep track of every bike for however long we're interested in it and it's path through the city.
And here is an entry of a bike in the database:
> db.bikes.findOne()
{
"_id" : ObjectId("500c6b57ea0f273df592c45a"),
"bikenr" : 6935,
"currentPos" : [
50.10824,
8.672381
],
"history" : [
{
"pos" : [
50.121815,
8.614588
],
"time" : ISODate("2012-07-23T00:16:10.997Z")
},
{
"pos" : [
50.121986,
8.614226
],
"time" : ISODate("2012-07-23T07:30:47.639Z")
},
{
"pos" : [
50.1220567603,
8.6142222359
],
"time" : ISODate("2012-07-23T07:55:46.248Z")
}
],
"lastseen" : ISODate("2012-07-23T10:47:06.573Z")
}
> db.bikes.count()
745
}
As you can see, the positions are sometimes nearly the same, so I guess this is the 24h theft-protection of the bike, which sends it's position to the backend once every day if it hasn't moved.
Finaly, a OpenStreetMap-leaflet combination to visualize the data:
Watch it here in full. The Boroughs are colored by the amount of bikes that are currently available and the heatmap shows some data about the requests I made against the Bike-API. This data is static, as I don't want to have my unsecure REST-API available on the internet.
Credit for the GeoJSON of the Frankfurt boroughs goes to the folks at Frankfurt Gestalten and DailyMo. Thanks also to Ik4ru5, because without the knowledge of other people playing with the Bike-API, I'd never have started to visualize the data.
If I'll ever find fun in this type of data again, I'd definitely use ElasticSearch as a replacement for MongoDB and Kibana3 as a replacement for my DIY-Leaflet map. Just to try it out.
EDIT: I deleted all signs of brands and so on to save myself from problems.