Who’s afraid of the big bad data?

There’s been an explosion of news around big data over the last month but nothing much actually seems to have happened, perhaps its newsworthiness is the result of a growing realisation that it’s here to stay?

The fact that data is being collected, en masse and used to influence almost every aspect of your daily life, is nothing new.  The stark realisation just how some of this data has been collected in the past has created some stirs and caused the odd person to flee the country but still, for many, it’s old news, or just a validation point for a lot people who said “I told you so”.

For many, their credit score is something to be nurtured and tended in order for it to provide a lifestyle that we otherwise could not afford and we tend to welcome data collection that reflects well on our salary, ability to borrow and payback and generally shows us to be individuals worthy of inhabiting this planet of ours, so what’s good data and what’s bad data or is there no such thing as either?

In many ways we are on the verge of a new information era as can been seen by the changes of Internet2, currently we have the means to capture unheard of amounts of data relating to almost anything we wish, using it effectively however, is still proving tricky.  Google’s flu-tracking database has not had a good time of it lately, completely mis-calculating the ’11-’12 and ’12-’13 flu outbreaks, not great for what had become big data’s poster child.  Never has the old adage, rubbish in – rubbish out, been so true, much of the feeds for this area of development, known as “computational social science”, are from social networks such as twitter and facebook, which in themselves are proving to be unreliable platforms when it comes to reporting numbers.

However, in more closed systems is big data doing any better?  IBM lists hundreds of examples where big data analysis has yielded positive results, ranging from preventing athletes injuries to managing transport infrastructure, but these generally seem to be big data being applied to solve small problems.

googles brain

Googles brain

There has been a certain amount of bandwagoning in the agricultural sector after some big money was spent on big data companies by even bigger agricultural companies but big data is far from new in this sector.  The NSF funded iplant platform, was developed nearly a decade ago to help manage animal and plant life sciences data and is deeply engaged in Internet2 via universities and research hubs.

Modern farming techniques already harness huge amounts of data, helping to make decisions on how much fertiliser to use, levels of irrigation, when to harvest and methods of crop rotation.  Expensive machinery can now be monitored and pre-emptively serviced before costly breakdowns occur and with machine to machine communication, product usage and optimisation can be done in real time, saving money and potentially increasing profits.  Many people argue that big data is just a logical progression from the latest in a continuing stream of innovations that began with the mechanisation of agriculture in the early 1900s, followed by hybrid corn in the late 1920s.

Roberts Steam Tractor

Roberts Steam Tractor

With a predicted 47% growth in population by 2050, many people think that agriculture cannot serve the coming needs of the planet without the use of big data but does this throw up a potentially new conundrum for the sector?  Big data can help optimise the production levels of crops and also tell the a farmer the best time to harvest and go to market, based on the current market price for

Automated harvesting

Automated harvesting

their products, but who actually owns this potentially sensitive data and how private is it?  Could this mean the age of free market competition in agriculture is about to end?  With more and more information available on almost every acre of land in the US can a farmer keep this information private or indeed, should they, given the remit for big data to do more, with less?

You can read more at http://www.farmers-exchange.net/detailPage.aspx?articleID=13580

IBM’s success list can be found here (http://www-01.ibm.com/software/success/cssdb.nsf/solutionareaL2VW?OpenView&Count=30&RestrictToCategory=default_BigData).

Matt Willis, Director, International Markets.

Once in a 100 years?

I’m British so I love talking about the weather, it’s the ultimate conversation gap filler, benign, comforting and always there for you to fall back on.  Often there is no need for a preamble, a simple “cold today” will illicit 3 to 5 minutes of inane chat whilst waiting for a bus or a pint.  Everyone in the UK is born with this ability as part of their birth right, people who move here gain the skill in a matter of seconds and visitors to our fair isle access this skill shortly after clearing passport control.  However, I also travel a lot, both for work and personally and I have noticed a dark and foreboding trend, this quintessentially Britishism has spread to the four corners of the globe.

Bushfires at Grampians National Park, Victoria, Australia - 18 Feb 2013

Australian bush fires 2013

Given what has occurred just in 2013, this is unsurprising, the point about the British weather and what used to make it such a constant topic of conversation, was the fact that it was unpredictable but also essentially unremarkable, even boring.  We had some rain but not as much as really wet areas, we have some snow, granted enough to bring little local airports like Heathrow and Gatwick to a complete standstill and to render the public transport system inoperable, but still only a few inches in reality, winds were unremarkable (we had a hurricane in 1984 but it was a small one), floods were rare, fires even rarer… you get the picture.  Granted, the further you travel from London the greater the chance was of encountering more severe weather, but still cataclysmic extreme weather events were rare, so rare in fact that they have a name… “a one in one hundred year event”.  Insurance companies used to use this term and derivatives of it, to help classify risk for insuring businesses.  However, these events are now occurring at an increasingly frequent rate.

The UK has had some bizarre weather over the past 12 months with severe flooding, heatwaves, snow and month vs month temperature differentials, between 2012 and 2013, which are almost unheard of however, globally it was far from quiet…

Some talking points of 2013:

–        Monsoon rains cause severe flooding in Malaysia, 23,000 people evacuated, Jan ‘13

–        Flooding in Mozambique, 250,000 people have to abandon their homes. Jan ‘13

–        Flooding in Indonesia, 100,000 people left homeless, Jan ‘13

–        Hottest month ever experienced in Australia, Jan ‘13

–        Largest snowfall from a single storm ever recorded, NE USA, Feb ‘13

–        Tropical storm Haruna hits Mozambique & Madagascar, Feb ‘13

–        Sever winter storms hit central USA, highest snowfall ever recorded in Kansas, Feb ‘13

–        Flash floods and crops destroyed in Uganda, Mar ‘13

–        New Zealand, worst drought in 30 years, crops fail, livestock slaughtered, Mar ‘13

–        Second warmest March on record in China (warmest was 2008), Mar ‘13

–        Wettest March on record, Spain, ‘13

–        Brazils worst drought for 50 years, Apr ‘13

–        Driest year to date on record, California, Apr ‘13

–        Shortest period between last ice day and summer ever, Austria, Apr ‘13

–        Record flooding in Central US, Apr ‘13

–        China, wettest May in 40 years, ‘13

–        Smallest snow covering in Eurasia since records began, May ‘13

–        1m+ evacuated in Bangladesh due to Cyclone Mahasen, May ‘13

–        Widest observed tornado ever hit Oklahoma, May ‘13

–        Over a 1000 people killed by flash floods, India, Jun ‘13

–        Hottest June temperature ever, Death Valley, Jun ‘13

–        Massive flooding in Europe, $18 Billion in damages, Jun ‘13

–        California Yosemite forest fires, took 2 months to contain, Aug ‘13

–        Colarado’s one in a thousand years flood, Sept ‘13

–        Cyclone Phailin, potentially strongest storm ever observed, Oct ‘13

–        Typhoon Haiyan, 195mph winds ravage the Philippines, Nov, ‘13

Strangely, 2013 was actually a quiet year when compared with 2012 which was really an  extreme year, in fact the last decade has seen some of the hottest, driest, wettest and coldest years since records began.  A recent report by the World Bank estimates losses to extreme weather to be more than $200 billion a year.

Extreme weather is having a massive detrimental effect on farmers worldwide, the high rainfall in China in May turned the world’s biggest corn producer into a net importer in one swift blow.  Droughts in the US will cut USA beef production to below 1994 levels.  Wheat production in the UK has been severely effected by some of the wettest years on record.


Farmers can adapt slowly to changes in climate but dealing with extreme events leaves most farmers at a loss.  It’s unclear if extreme weather will stop being classed as extreme as these events start to occur more and more frequently, will they simply become “weather”.  Farming practices are changing to try to adapt to these events, but also the choice of crops will also have to change.

Although these extreme weather events can be devastating the most difficult to deal with, as a farmer, is drought.  A drought is one of the worst natural disasters because it is able to cover wide areas and continue for extended periods of time. Droughts not only affect the direct area, but also the nearby cities that rely on that food and service income. 14% of the United States is affected, on average, by severe to extreme drought annually and this figure is rising steadily each decade.  Food crops are being adapted by a combination of breeding and years of experience however, some crops are already adapted to these environments.  As land becomes degraded through over farming, mismanagement and continually changing environmental factors, making the wrong choice of crop could have devastating effects on some farmers.Map of extreme weathers event that hit Australia during Summer 2012/2013

 TerViva’s goal is to help farmers do more, with less, by addressing the fundamental issues around the changing environment and enabling farmers to continue to farm land that would otherwise be unproductive.

Matt Willis is TerViva’s Director of international markets.

Are you fracking crazy?

The global debate on fracking continues to dominate much of the world’s energy press.  The US is currently benefitting from a boom in shale gas production which some commentators claim is one of the biggest reasons the American Eagle is still managing to dominate the Chinese Dragon.  In just a few years the US has gone from being a net importer to net exporter of gas.  This cheap liquefied gas is driving Brent crude prices down and breaking the US’s reliance on the OPEC countries output and price controls.  The US is currently opening up more than 10,000 new fracking wells each year and gas prices are three and half times lower than in the UK.

Fracking is far from new technology, Canada has been doing it since the 60’s and Germany since the mid 70’s.  France and Bulgaria have banned it, but the Netherlands and China are actively pursuing it.  China’s addressable reserves are claimed to be 50% higher than the current largest production centre, the US.


Some large scale shipping companies have started to convert their fleets to gas engines

The debate in the UK is fierce with many doubters claiming that the sheer physical difference between the landscapes of the US and the UK mean that a revolution of this scale is simply not feasible.  Many of the suitable fracking sites will be in urban centres or near potential vulnerable areas such as rivers.  However, cheap, instant energy, is a hard argument to resist, indeed PM David Cameron has already gone from stating that renewables are the way forward for the UK’s energy requirements, to fracking being the new solution, in less than a year.

As usual, all is not clear, the endorsement in the UK reeks of biased corporate interest with many of the Governments advisors having strong links in the burgeoning fracking industry, Lord Browne, ex Chief Exec of BP and now chairman of Cuadrilla, one the largest operating fracking companies in the UK, being the most prominent.  Is it a coincidence that The Royal Academy of Engineering had Lord Browne as its president while producing the final report on shale gas extraction, which is used as the backdrop to justify the safety of fracking, by the UK government?

US Fracking

Fracking on the Pinedale Anticline formation in Pinedale, Wyoming, where there are serious water shortages being reported

However, the current US gas prices are a big incentive unfortunately, there is no guarantee that the UK’s prices will follow the same pattern, or that the trend in the US is even sustainable.  Indeed, some researchers claim that the US bubble is already bursting.  Michael McElroy, who writes for Harvard Magazine, states that: “the economic momentum of the shale-gas industry can be sustained for the long term only by decreasing production (ultimately causing prices to adjust—a process that may be under way…) or by increasing sales of its product”.  Veteran Petroleum Geologist, Arthur Berman, wrote in 2011: “Facts indicate that most wells are not commercial at current gas prices and require prices at least in the range of $8.00 to $9.00/mcf to break even on full-cycle prices, and $5.00 to $6.00/mcf on point-forward prices. Our price forecasts ($4.00-4.55/mcf average through 2012) are below $8.00/mcf for the next 18 months. It is, therefore, possible that some producers will be unable to maintain present drilling levels from cash flow, joint ventures, asset sales and stock offerings.”

Do low gas prices provide an opportunity to reduce Carbon emissions?  Most claims, that shale gas will significantly reduce US carbon emissions in the future, are not currently based on any solid facts but rather on a certain amount of wishful thinking. That’s because those claims assume natural gas is replacing coal only, rather than replacing some combination of coal, renewables, nuclear power, and energy efficiency — which is obviously what will happen in the real world.  A separate study released this week from the International Energy Agency found“low natural gas prices will hamper the U.S.’s incentive to continue spending on energy-efficiency projects.”


A new study, from Stanfords energy modelling forum, “Changing the Game? Emissions and Market Implications of New Natural Gas Supplies Report.” suggests that use of Shale gas will have little or no effect on the reduction of CO2.

Is the lure of cheap natural gas just too great to resist and should it be pursued at the cost of other more sustainable technologies?  Is this the start of a sustainable movement towards better use of our natural resources or simply another boom and bust gold-rush to be exploited?

Matt Willis is Terviva’s Director, International Markets.

History Repeating – Old Habits Die Hard.

The early 1990’s saw Cuba enter a severe crisis due to the collapse of the Soviet Union and its associated economies in Eastern Europe. These countries supported around 80% of Cuba’s economy, which was based on intensive single-crop agriculture and was highly dependent on imports of agricultural sub-products, such as fertilisers and pesticides. Up until the collapse, the Soviet Union had been paying much higher prices for sugar than the true price in the international markets, providing much needed income for Cuba.  In a very short period Cuba’s agricultural system collapsed, posing a serious threat to something that had been taken for granted, its supply of food.  This heralded what is known as the “special period” where many Cubans were starved of basic food supplies while their whole countries approach to Agriculture went through some radical changes.

However, during the 1980’s Cuba’s government and agriculture-related professionals, had been developing alternative technologies and processes for agriculture, anticipating the vulnerability of their economy being so dependant on a few agricultural products.  When the crisis arrived they were relatively prepared, even so the crisis was tougher than anyone expected. The response from the government was to attempt to instil nothing short of basic survival strategies into urban communities. Agriculture, in a few years, moved from the large export-oriented, chemical dependant monoculture, that is the norm for most of the world, to a small size, local urban-based, organic food production model. Small scale farms and orchards at home or in disused plots, became the standard.  The government supported this new approach of urban agriculture and it soon became part of national policy.

urban farm

Cuba’s achievements in urban agriculture are truly remarkable, there are now over 380,000 urban farms, covering 50,000 hectares of otherwise unused or marginal land, producing more than 1.5 million tons of vegetables with top urban farms reaching yields of 20 kg/m2 per year of edible plant material using no synthetic chemicals or the equivalent to a hundred tons per hectare. Urban farms now supply 70% or more of all the fresh vegetables consumed in Cuba’s main cities.  However, this change has come at a price with Cuba’s agriculture sector contributing an estimated 4% of GDP in 2010, but comprising of 20% of Cuba’s 5.1 million labour force.

Under this new scenario the importance of the contributions being made by peasant farms, in reducing food imports, should not be underestimated.  However, despite these notable advances of sustainable agriculture, and evidence of the effectiveness as an alternative to the monoculture model, Cuban Government interest persists in promoting high external input systems based models. Under the pretext of increasing food security and reducing reliance on food imports, these specific programs pursue the old paradigm of large scale crop and livestock production and insist on going back to the previous monoculture methods, in turn increasing dependence on synthetic chemical inputs, large scale machinery, and irrigation, despite proven energy inefficiency and technological fragility, in short leading Cuba right back to where they started.

Chemical useMany of the resources that are being provided by international partners, such as Venezuela and Brazil, are dedicated to protecting or boosting agricultural areas where a more traditional, intensive, agriculture is practiced for crops like potatoes, rice, soybean, and vegetables. Currently these areas used for large-scale, industrial-style agricultural production still represent less than 10% of the cultivated land, however, millions of dollars are being invested in pivot irrigation systems, machinery, and other industrial agricultural technologies, a “quick fix” model which increases short-term production but generates high long-term environmental and socioeconomic costs, while replicating a model that was failing even before 1990.  This cyclical mindset would seem to strongly undermine the advances achieved by Cuba’s agricultural organic farming since the economic collapse in 1990.

However, since 2008 Cuba’s agricultural sector, in general, has started to underperform, and the authorities have acknowledged the heavy cost of importing food to fill the gap. Interestingly the worst affected sectors are the traditional, large-scale production crops, Plantains dropped by 44.2 percent, potatoes by 36 percent and citrus by 33.9 percent. The category of “other tubers” plunged by 58.4 percent. Corn production fell by 22.5 percent, beans by 7 percent and fruit by 13.9 percent, according to the report by the National Statistics Office (ONE).  Increases in production were reported in garden vegetables, up 8.4%, and rice, up by 2.5%.

Cubans are only too aware of the implications, which translate into high retail prices for foodstuffs.  As President Raúl Castro told a recent cabinet meeting, every time the production quota is missed, the cost to the state runs into millions of US dollars.  Official figures show that Cuba spent 1.7 billion dollars on food imports in 2012, up from 1.5 billion in 2010. The projection for 2013 is another 200-million-dollar increase, to 1.9 billion dollars.  In early December, state television reported that annual production of beans, a Cuban staple, was running at 20,000 tons a year, when consumption was 100,000 tons.  Despite President Castro’s focus on raising farm output since 2008, the sector has consistently failed to fulfil the Communist Party’s stated plan of growing enough rice, beans, maize, soya and other crops to allow a “gradual reduction of imports”.

the normCuban agriculture currently experiences two extreme food-production models: an intensive model with high inputs, and another, beginning at the onset of the special period, oriented towards local small scale farms based on low inputs. The experience accumulated from these initiatives in thousands of small-and-medium scale farms constitutes a valuable starting point in the definition of national policies to support sustainable agriculture, thus potentially displacing a monoculture model that has reigned supreme for almost four hundred years.  However, is the drive for short terms gains combined with the external pressures being put on Cuba, by “partner countries” ensuring that that their sustainable agriculture model was nothing more than an interesting experiment developed out of necessity and if so, what chance is there for a wider, world, adoption of such techniques as we try to loosen our dependency on fossil fuels?

Matt Willis is TerViva’s Director of International Markets.

What is marginal land? Goodbye topsoil, my old friend…

Author: Matt Willis

The food versus fuel debate in the biofuels development world has prompted some policy makers to propose that agrofuel crops should only be planted on land that is considered marginal or degraded, but what exactly does this mean?

Marginal land is defined as arid and generally inhospitable land that usually has little or no potential for profit, and often has poor soil or other undesirable characteristics. This land is often located at the edge of deserts or other desolate or degraded areas.  However, much of the worlds prime agriculture land would be classified as marginal were it not for the advent of cheap, nitrogen based fertilisers and large scale irrigation, to keep it productive.

In fact more and more land is becoming degraded as intensive farming and bad land use practice leaves prime agriculture land vulnerable to severe topsoil erosion. Around the world, including the USA and the UK, soil is being swept and washed away 10 to 40 times faster than it is being replenished, destroying cropland the size of Indiana every year, reports a Cornell University study. The vast majority, 99.7 percent, of human food comes from cropland, which is shrinking by more than 10 million hectares (almost 37,000 square miles) a year due to soil erosion, while more people than ever, more than 3.7 billion, are malnourished. More pessimistic reports state that there may be as little as 70 years left before all the worlds topsoil is eroded.


Photograph by Lynn Betts, NRCS

Heavy rains in northwest Iowa washed away soil, leaving this scarred tableau. This type of erosion, termed sheet-and-rill erosion, occurs when there is insufficient vegetation to hold soil in place. As rain falls, it forms sheets of surface water that transport soil away. As more water accumulates, it forms runoff channels called rills, which further displace soil.

Excessive use of nitrogen based fertilisers also has a price.  Around 60% of nitrate in English waters originate from agricultural land. Elevated levels of these nutrients are of concern because they can cause eutrophication, which harms the water environment. Also, excess nitrate has to be removed before water can be supplied to consumers, raising the cost of supplying fresh water.

Agriculture accounts for 70% of our fresh water use, we pour most of our water straight onto the ground. If soil is not fit for purpose, that water will be wasted, because it washes right through degraded soil and past the root system.

So, although the definition of marginal land may seem obvious, what actually is considered marginal land can change within a relatively short period of time.

Are lands that require huge amounts of irrigation and fertilisers to remain productive, under the current intensive agriculture model, only not marginal due to these inputs and if these inputs become too expensive to maintain the productivity of the land, does it then become marginal?  Citrus greening disease is currently turning once productive, non marginal land, into unproductive acreage that, given time, may lay fallow and be susceptible to soil erosion creating huge swathes of marginal land in States such as Florida.  Options exist to utilise this land now to grow oil seed crops before this acreage becomes truly marginal. Would policy makers prefer the land to become unproductive first before seeing suitable crops such as Pongamia cultivated in these areas?

In the developing world the identification of marginal land is also not so easy.  If you believe what you read, then in continents such as Africa and Latin America, 1000’s upon 1000’s of acres lie unused and waiting to be developed however, it is necessary to draw the distinction between unused, be it commercially or otherwise, and marginal.  Unused acreage is often, in fact, being used by small communities. One example of how estimates for “abandoned cropland” useable for bioenergy are derived is a 2008 study by Christopher Field et al, who suggests that 386 million hectares of such land exist. Any land believed to have been used as cropland at any time since 1700, and which satellite images don’t show as being “cropland” today is classed as “abandoned” unless it is currently forested or part of urban settlements. There has been no critical review to assess whether such satellite-based mapping ignores small-scale mixed farming by communities, but it is clear that other community uses, including the use of land for pasture, are ignored when “abandoned cropland” is defined.

Policy makers need to take into account a far larger range of issues when considering land use for biofuels.  Intelligent use of crops that can actually thrive on truly marginal land are required to develop sustainable agriculture programs that can potentially create a new source of income for struggling farmers, such as in Florida and also help to develop new markets for burgeoning economies in Africa and Latin America.  However, a “one size fits all” approach to these policies simply will not work.

TerViva works responsibly with farmers and landowners to create crops that truly thrive and rehabilitate degraded land, transforming barren landscapes into healthy, profitable environments.Image

Matt Willis is TerViva’s Director of International Markets.