Open Data and the Personalisation of Experience

Earlier in July at SMC_MCR, a monthly digital and social technology meet up in Manchester UK,  BBC R&D demonstrated a new approach to personalised entertainment called Perceptive Media. It is something that BBC producer Ian Forrester had been talking about for some time, being revealed at SMC_MCR in February. At that point it was hard to understand what the concept entailed. It was explained as a way of delivering media that was tailored to individual preference and environment but little else.

 

On their return in July the team showcased a short radio play demonstrating some of the concepts of Perceptive Media. The play can be found here http://futurebroadcasts.com/ At first listening the play seems to follow the traditional radio play form, but within the play there are certain personalisations that are based upon the location of the listener. After a couple of listenings it is quite obvious where the personalisations exist. As Ian Forrester stated in the Q&A, it was a fairly basic demonstration of the technology pointing to the challenges of narrative personalisation and the ability to create these personalisations ‘on the fly’, in the browser. Even with such a short and somewhat basic demonstration of Perceptive Media it is easy to see how it could develop into a more complex form cutting across platforms.

 

The personalisation aspect of Perceptive Media comes from the creation of a narrative framework that allows certain variables to be inserted, with these variables influenced from the data that the Perceptive Media storyteller has access to. In the case of ‘Breaking Out’ – the play in the demonstration – the data accessed was local weather, listings and local news. As more data is made available it is easy to see how it could be integrated into a Perceptive Media framework. The demonstration offers a glimpse into a new form of story telling based on an individuals location and environment and if coupled with personal data – preference and situation.

 

In 2009 at FutureEverything there was a presentation by Philip Trippenbach, then at the BBC, about the construction of narrative in games especially first person games. He highlighted a game called ‘Six days in Fallujah’ which he writes about here http://trippenbach.com/2009/06/09/six-days-in-fallujah-and-the-dirty-g-word/ What I find of interest is the possible use of the form to be educational, to disseminate news and information in a way that many would be uncomfortable with. What Trippenbach talks about is a personalisation of experience, a certain ‘being there’. The use of real situations to create realistic gaming experience is not new but a concerted attempt to create ultra-realistic gaming through streaming of real-time positional and telemetry data from Formula 1 Grand Prix was attempted in 2006. http://news.bbc.co.uk/1/hi/technology/7440658.stm Although as the article states it would probably only be of interest to hardcore gamers, it offers fascinating possibilities about could be achieved at this intersection of gaming, personalisation and data.

 

Although not using open data, a great example of this was demonstrated at FutureEverything in 2011. Arcade Fire’s – We Used To Wait scores a personalised film called The Wilderness Downtown by Chris Milk developed in association with Google Labs. It invites the user to enter the address of where they grew up and then the HTML5 based experience literally flies. You can try it here: http://www.thewildernessdowntown.com/

 

Data both open and personal is at the centre of the personalised experience whether it be local weather, what food we like, position of racing cars, location of where we once lived or the environment in which real-life situations were played out. We are starting to see a new world where the way information is delivered to us is adaptive, often in real-time and just for us. It might not be to everyone’s liking but it is happening, just look what Google are doing: http://www.google.com/landing/now/

 

Disclosure: Julian Tait is a co-founder of SMC_MCR and content programmer for FutureEverything

Transport Special – ODM May

Transportation holds a lot of interest for many in the open data community. The availability of transport open data and realtime transport open data offers the potential to create diverse, innovative applications and services, as well as a greater understanding of how transportation systems work.

The meeting was an opportunity to get an update from Craig Berry and Dave Busby from TfGM as to the progress they were making in opening up the data within TfGM. Back in 2010 TfGM made a commitment to start making available open data. This began with the release of the ATCO-CIF Timetable data in July 2010.

In January, Dave and Craig outlined what data they were trying to release and some of the technical, organisational and contractual obstacles that needed to be overcome, they were encouraged to join the Open Data Manchester Google Group and use it as a means of finding out what data developers were interested in and in what form. This they did and the update they gave seemed to reflect this.

The presentation is attached. What was apparent from the presentation was that there was a real willingness to involve the ODM community in the process of release and it was hoped that through this engagement, a greater awareness of need and organisational intelligence would develop.

TfGM Presentation

Following on from the TfGM update Nathan Day, Business Development Manager of Rockshore gave a preview of the soon to be released Network Rail realtime data APIs at the moment it is in Beta and Network Rail are only allowing 100 people access. It is due to be launched at the end of June.

The specifications and structure of the data are contained within the developer pack Developer pack for Network Rail data feeds

Although the structure of the data is described there is little context to understand what the data is describing and it will be up to the developer community to create this.

It is hoped that there will be another transport update later in the year.

Co-operating on Open Data

The new co-op HQ
The view of the new Co-op HQ from the 24th floor…

This week, way up on the 24th floor of the CIS Building in Manchester, we facilitated an event to look at how and why the Co-operative movement could engage with open data.

Alongside Co-operatives UK and The Co-operative News, a group of open data and co-op people gathered to hear from Chris Taggart of OpenCorporates, and then begin the discussion of how this movement could evolve.

We learnt that the Co-operative movement is vast and diverse – ranging from banking to funeral care, from software to snake catchers!  Equally, there is not *one* type of co-op. Variations such as worker, consumer, retail and volunteer co-operatives are just some – although all share the same ethos and relation to the guiding principles of mutal benefit for members and wider society.

Chris took us through some of the philosophical and practical issues around open data, summized in the classic line:

“Open data is the new democracy”

.. which certainly got people thinking.

I’ve picked up on two main themes that stood out from the workshop.  The Co-operative News have ably started to document and publish materials from the workshop – well worth a look.

1 – Open Data on Co-operatives

This was our starting point.  If co-ops were to openly publish data about their activities, what would represent their “added value” and “point of difference”.  Would this be data on membership, community activities and other measures? How would this be achieved?  What could be the “quick-win” datasets that co-ops could push out to engage people?

We discussed how an data standard for co-operatives could be one way to facilitate this – but this event was only two hours…

2 – Open Data by Co-operatives

This theme was something that struck to the heart if many it seemed.  One of the key values of the co-operative movement was knowing how to run successful businesses and organsations in a collaborative manner.  How could this be applied to open data?  Could open data projects be governed in a co-operative way?   There was some discussion around the notion of “prosumer co-operatives” in terms of data producers and consumers working in joint.  Certainly, there was an appetite for exploring this further…

What Next?

Open Data Manchester were proud to be a part of kicking off this discussion, and many thanks to all those that attended, and our collaborators at Co-operatives UK and The Co-operative News.  This October/November, the co-op world will arrive in Manchester to celebrate the end of the United Nations International Year of Co-operatives.

Please add your thoughts on how we can further stimulate the conversation in the run-up and during this event…

Open Data Manchester March meeting

March’s meeting was an opportunity to help shape Manchester City Council’s forthcoming open data Hackathon. Stuart Baldwin – an ODM regular – spoke about Manchester’s plans for an event in October to coincide with the Manchester Science Festival.

The driver behind this is the recently announced Manchester Digital Strategy and a recent trip that Chief Executive of MCC, Sir Howard Bernstein made to New York. Whilst a guest of Mayor Bloomberg, Sir Howard was apparently impressed with what New York was doing with their open data initiatives such as 311 and App Challenges.

Open Data Manchester and MDDA advised MCC, that for a Hackathon to work it needed to work with the developer community to make the event relevant and developer friendly.

The conversation was mainly focussed on the types of data that developers wanted releasing and there is a list from Duncun Hull @dullhunk here

What was notable was the willingness to listen to what the community wanted and by suggestions from MCC itself, such as Contaminated Land data which has traditionally been contentious.

[vimeo http://www.vimeo.com/36540620 w=400&h=300]
Visualisation by Jonathan Fisher more details here

After the Hackathon discussion attention focussed on Road Traffic Collision data and the work that Steven Flower, Jonathan Fisher and Jonathan S. – There has been discussion about forming a sub-group around RTC data and its use. So if people want to get involved in that contact Steven Flower on the Google Group. Jonathan Fisher’s visualisations where discussed and also the variation in data quality that exists. It was noted that although data was provided to TfGM who collated the data for the Department of Transport. Different flavours of the data existed in different places. TfGM upload monthly data to DataGM which lacked detail on casualties and vehicles involved. The complete RTC data gets forwarded it to the DfT who then make it available via the DfT website and and data.gov.uk with more detail but in two different versions. We are trying to find out why DataGM only holds a less detailed version.

T-Shirts

There was a discussion a while back around Open Data Manchester t-shirts. It was probably after an Open Data Manchester meeting in Common Bar, but in a moment of procrastination I have designed a t-shirt.

It will probably never see the light of day but it would be cool to see if other people can design one too. You never know one day we might get some printed. This is the template T Shirt Blank

It doesn’t matter how bizarre or strange – as long as it isn’t offensive, libellous or irrelevant. Send it to opendatamcr [at] littlestar.tv and I’ll post it up

Julian

January meeting with TfGM

January’s Open Data Manchester was a transport special, with Craig Berry and Dave Busby from TfGM giving an update as to the types of data that TfGM hold, and what they are trying to release. Open Data Manchester people may already know of Craig Berry as the Information Manager who has been tasked with identifying and releasing open data. Dave Busby’s brief is for integrated ticketing and real-time information.

TfGM reinforced its position with regard to open data at the meeting. There has been a number of rumours over the past twelve months as to what the organisation was trying to release to DataGM – Greater Manchester’s open data portal . TfGM are currently releasing data with regard to bus schedules, NaPTAN stop locations, fixed and mobile speed camera locations and monthly Road Traffic Collision updates. There had been mooted some realtime data would be released.

Greater Manchester has been crying out for an intelligent integrated ticketing system. To many a lack of such system has made travel by public transport around Greater Manchester more difficult than it should be. To this end TfGM are developing a specification that will go to tender in the 1st half of 2012. The system will initially cover Metrolink and then encompass Greater Manchester buses. The system will use contactless technologies in a similar vein to TfL’s Oyster Card but with the added functionality of being able to use contactless bankcards and NFC phones. It was interesting to note the certainty that NFC will be adopted, by most handset companies within the next year. Paying by Google Wallet was also mentioned as a possibility. The ticketing system will also have fare rules that will calculate the best price for journeys undertaken.

Although getting Integrated ticketing to work with Metrolink would be a relatively easy task and a useful test bed to prove the utility of the system, getting Greater Manchester’s 40+ independent commercial bus operators to adopt the system maybe more challenging and may need a certain amount of political will. Anonymised journey data from the system or personal access to journey history wasn’t discussed in detail, although the later seems to be fairly standard in smart ticketing systems, access to anonymised data could offer huge potential for applications and services that look at gate loading on routes, passenger density etc.

The advent of the oft mooted, realtime data from TfGM looks closer – although there was no specific timescale mentioned. There will be access to the Metrolink Passenger Information Displays data, although how this will manifest itself is uncertain. Developers present at the meeting suggested that JSON would be preferable. The main challenge with accessing real-time Metrolink location data is that the Tram Management System currently being implemented isn’t currently functioning throughout the network. The initial release of data will cover the South Manchester line and Eccles lines.

Although it doesn’t look like there will be any real-time bus data soon, TfGM would like to release the location information of the free Centreline buses that are being operated on TfGM’s behalf. This data will be location data that won’t identify the actual service the bus is running. It was suggested that as there are only three distinct Centreline routes it wouldn’t be that complicated to identify, even where the routes overlap. There is also an Informed Personal Traveller pilot that is being run in Bury by Logica, ACIS and First Bus. It uses a number of technologies including an AVL system that has been fitted to approximately 100 of their buses. The IPT application hasn’t been released yet and there are indications that the system is closed.

TfGM recently submitted a bid to the Local Sustainable Transport Fund and written into it is the provision of open data and the development of an intelligent multi-modal Journey Planner pulling all relevant data that TfGM has at it’s disposal, how developers could access the Journey Planner was discussed and whether it would exclude the provision of other types of journey data.

There is a move to make other data available through the LSTF, these include Car Park updates, real-time disruption data, journey down roads data and feeds off TfGM’s SCOOT adaptive traffic control system. SCOOT controls half of the approximately 2000 traffic control signals in Greater Manchester.

The lack of transparency with regard to bus fare structures within Greater Manchester has been a subject that has come up many times, especially regarding anecdotal evidence that dependant communities are charged more per mile than others having viable transport alternatives. TfGM stated that Greater Manchester is one of the few places where bus travel is generally more expensive than rail. To this end TfGM are interested in developing a project similar to one that Open Data Manchester was developing over a year ago that encouraged travelers to submit the details of their journey and price.

At the close of the discussion TfGM were encouraged to use the Open Data Manchester Google Group as a resource to ask questions and to highlight initiatives and challenges.

Making Open Data Real consultation results published.

The results of the last year’s Making Open Data Real consultation have been published. Open Data Manchester submitted a response as did 246 others.

These responses will be used to define the governments approach to Open Data and will hopefully bring about a meaningful push from both central and local government.

The following Greater Manchester based organisations responded:

  • North West e Government Group
  • Open Data Manchester
  • Rochdale Council
  • Swirrl IT Ltd
  • Trafford Council
  • Transport for Greater Manchester
  • So far play to the above for the above for engaging.

    Summaries of the consultation can be read here:

    Full responses can be downloaded here:

    SOPA and PIPA

    What and why?

    The SOPA and PIPA legislation currently going through the American congress has impact and ramifications far beyond the shores of the US and piracy, and is a threat to the Free and Open internet that we have today. It iS a blunt, badly constructed piece of legislation that seeks to prop up an industry based on old business models by clamping down on new and innovative ways of producing and disseminating content. As with most bad pieces of legislation it will be open to creep and opportunism, where arbitrary take down and closure of sites pointing to or serving content would prevail. It offers a world where the winners will be the established players with large legal teams and vested interests and the losers will be everyone else. It is an outcome which we, as part of the Open movement, are fighting against.

    What could it do?

    Would it effect Open Data? We don’t know, as open data is made available by data owners under open licensing. It might constrain the publishing of data that has been scraped, but the legal arguments as to whether data is protected by copyright and IP law are complex.

    It could constrain search, if you fall foul of its broad and ill-defined terms. You may have a site but it might as well not exist if it doesn’t show up in search results.

    It can effect free expression – just imagine the internet awash with copyright bots who can automatically close down or de link sites with out any recourse to the law. You can get a flavour of this on on some sites that have content sniffing technology.

    Who decides what is or isn’t copyrighted or IP protected material? In the SOPA world you will be found guilty and then have to fight to prove your innocence. How will this effect the ability to expose wrongdoing if the evidence is subject to SOPA and PIPA?

    Many people have written about why these laws are a bad thing and some links are below. From the UK we can only watch and hope.

    [vimeo http://www.vimeo.com/31100268 w=640&h=480]

    Technical overview

    Why SOPA is bad for business

    Guardian article on effect

    Added 19/1/12
    Excellent article on Al-Jazeera by Jonathan Zitrain et al.

    Update 19/1/12

    Last night several members of congress, who originally supported the SOPA/PIPA bill withdrew their support due to the massive international campaign that saw many thousands of websites, including Open Data Manchester’s, go dark and people ‘block the switchboards’ of the American congress. It was a victory in the battle to protect our free and open internet but there is too much money at stake and we are sure we will see a revised, more subtle form of SOPA/PIPA in the not too distant future.

    Open Data Manchester – November Meeting

    November’s Open Data Manchester.

    Paul Gallagher – Head of Online for Manchester Evening News gave a presentation regarding the role of the MEN during the Manchester Riots. He described how the Manchester Evening News had used Social Media during the riots and how his team had started to collect data regarding the riots and the subsequent court cases to give insight into some of the possible causes of the riots.

    Most interesting was the resources that the MEN had put into reporting on the court cases following the riots and by having court reporters sitting in on each of the trials they created a schema and dataset show the areas that people lived in, mitigating circumstances, age, type of offence, sentence etc. This is data that can only be created if you attend the trail. This allowed them to map offences against depravation indices and changes in the way that sentencing was delivered over the course of the trials.

    The discussion also touched on news organisations becoming huge archives of sentencing data and how this can effect people’s lives even after their convictions have been struck off. MEN does have a policy where certain details are redacted from the historical archive but this is done on a case by case basis.

    There was also an update as to the preparations for the International Open Data Hackday and the responses to the Governments Open Data and Public Data Corporation consultations.

    ODM Response to the Public Data Corporation consultation

    Charging for Public Data Corporation information

    1. How do you think Government should best balance its objectives around increasing access to data and providing more freely available data for re-use year on year within the constraints of affordability? Please provide evidence to support your answer where possible.

    This question is framed incorrectly. For open data to be truly sustainable there has to be a shift away from the notion of affordability and access. Open Data is part of a transformation of how services are delivered within and by government and how government relates to people and business. What we should be moving to is the notion of Government as platform where the data that the government uses for its own purposes is also seamlessly available for reuse.

    2. Are there particular datasets or information that you believe would create particular economic or social benefits if they were available free for use and re-use? Who would these benefit and how? Please provide evidence to support your answer where possible.

    We see that there are a number of core ‘infrastructure’ datasets that have allowed systems to be developed within the UK. The majority being run by trading funds. Consolidating their charging position within the PDC will have a chilling effect not only on the direct creation of applications and services but on an underlying data ecosystem that will create social and economic value. It has impact on future technological developments where applications need to be aware of their relation to core data infrastructure. This is particularly important with the emerging development of the Internet of Things and pervasive technologies.
    Whilst developing the Open Data Cities project in 2009 and DataGM – The Greater Manchester Datastore with Trafford Council it became apparent that local authority and community access to certain data such as Land Registry data was creating problems. Anecdotally it had been suggested that easy and open access to Land Registry data would help combat cross boundary housing benefit fraud and would of eliminated the MPs second home scandal.

    3. What do you think the impacts of the three options would be for you and/or other groups outlined above? Please provide evidence to support your answer where possible.
    The charging options outlined will all have impact on the development of open data services/applications and future technologies where open data is an enabler.
    All three models are flawed in that they are trying to predict and extract value from an emergent field. They fail to take into account what is needed to create a sustainable, innovative and disruptive data ecosystem. Disruptive innovation in emerging fields needs to have a low barrier to entry and the creation of an ecosystem where ideas can be tested, fail and succeed with marginal cost.

    4. A further variation of any of the options could be to encourage PDC and its constituent parts to make better use of the flexibility to develop commercial data products and services outside of their public task. What do you think the impacts of this might be?
    By encouraging public organisations to develop services outside the public task has the potential to distort an emerging market and should be treated with caution. The knowledge that many public organisations hold in regard to their task is unique and could be encouraged as long as the underlying raw data resources are available to all.

    5. Are there any alternative options that might balance Government’s objectives which are not covered here? Please provide details and evidence to support your response where possible.

    5. There needs to be an appraisal of the wider value and impact of releasing public data. This impact should not just be seen as a simple transactional value but a broader impact on the engagement and wellbeing of society.

    Licensing
     
    1. To what extent do you agree that there should be greater consistency, clarity and simplicity in the licensing regime adopted by a PDC?
    It is understood that having multiple licensing regimes can create confusion and hence hinder the development of interpretations, applications and services. The danger of ‘double licensing’ is real especially as products become more complex. The adoption of OGL should be seen as a default position for raw open public data. At the moment within public datastores such as DataGM there are numerous licensing options most with a potential to cause confusion and contaminate downstream data usage. This confusion has also been used as an excuse for not releasing data.

    2. To what extent do you think each of the options set out would address those issues (or any others)? Please provide evidence to support your comments where possible.
    The potential impact of different organisations within the PDC to define their own licenses to suit different uses of data usage presupposes that the data provider has an appreciation of the potential uses of the data. This may work in an environment where products are developed in one specific domain but when innovation is cross cutting the need for standardisation and clarity becomes clear. Whilst the third option of a single PDC licence with adapted schedules of use would seem easiest. The question fails to recognise that raw open public data should be free by default with exemptions being rigorously justified.

    3. What do you think the advantages and disadvantages of each of the options would be? Please provide evidence to support your comments
    Please see above

    4. Will the benefits of changing the models from those in use across Government outweigh the impacts of taking out new or replacement licences?
    Yes, as the current licensing regime is opaque and hinders innovation and innovation drives the economy.

    Oversight

    1. To what extent is the current regulatory environment appropriate to deliver the vision for a PDC?
    You cant have a system of oversight which fails to engage users. It is necessary to have one robust and representative regulatory environment that has real powers to make PDC based organisations compliant. The representation should be a balance of suppliers and users of data.

    2. Are there any additional oversight activities needed to deliver the vision for a PDC and if so what are they?
    Apart from making sure that raw public data is made open and freely available, No

    3. What would be an appropriate timescale for reviewing a PDC or its constituent parts public task(s)?
    Six monthly initially then after the initiative becomes embedded less often