What if My Data Quality Is Not Good Enough for Analytics or Situational Intelligence?


Spreadsheet 01


You may feel that the quality of your data is insufficient for driving decisions and actions using analytics or situational intelligence solutions.  Or, you may in fact know that there are data quality issues with some or all of your data.  Based on such feelings or knowledge, you may be inclined to delay an analytics or situational intelligence implementation until you complete a data quality improvement project.

However, consider not only the impact of delaying the benefits and value of analytics , but also that you can actually move forward with your current data and achieve early and ongoing successes. Data quality and analytics projects can be done holistically or in parallel.

“How?” you ask. Consider these points:

  • Some analytics identify anomalies and irregularities in the input data. This, in turn, helps you in your efforts to cleanse your data.
  • Some analytics, whether in a point solution or within a situational intelligence solution, recognize and disregard anomalous data. In other words, data that is suspect or blatantly erroneous will not be used, so the output and results will not be skewed or tainted (see this related post for a discussion about: “The Relationship Between Analytics and Situational Intelligence“). This ability renders data quality a moot point.
  • It is a best practice to pilot an analytics solution prior to actual production use. This allows you to review and validate the output and results of analytics before widespread implementation and adoption. Pilot output or results that are suspect or nonsensical can then be used to trace irregularities in the input data.  This process can  play an integral part in cleansing your data.
  • Some analytics not only identify data quality issues but also calculate a data quality score that relates to the accuracy and confidence of the output and results of the analytics. End-users can therefore apply judgement if and how to use the output, results, recommendations, etc. Results with low data quality scores point to where data quality can and should be improved.
  • Visualization is a powerful tool within analytics to spot erroneous data. Errors and outliers that are buried in tables of data stand out when place in a chart, map or other intuitive visualization.

You can be pleasantly surprised at how much success you can achieve using data that has not been reviewed, scrubbed or cleansed. So set aside your concerns and fears that your analytics or situational intelligence implementation will fail or have limited success if you do not first resolve data quality issues.

Instead, flip such thinking around and use analytics as one of the methods to review and rectify data quality.  In other words, integrating analytics into your efforts to assess and cleans your data is a great way to leverage your investment in analytics and get started sooner rather than later.

What are you waiting for?  Get started exploring and deriving value from your data no matter the status of its quality.


Situational Intelligence and Smart Cities: Crisis Response


So far in this series on situational intelligence and smart cities, we’ve looked at services in constant use such as transportation, water, and energy. How might the concept of progressive use cases help evolve smart infrastructure for more intermittent services, such as crisis response?

As a first step, situational intelligence supports integrating all first responder communications and locations onto a single, citywide GIS system, possibly on a video wall in a city operations center. By knowing where first responders are located and using route optimization analysis, city officials can dispatch the closest and best-suited people in time of crisis. Quick response saves costs, property and lives.

Operational savings can be applied towards increasing coordination between city agencies in response to crisis. With all first response systems shown on a single map, the city operators can more easily dispatch appropriate teams to the scene. For instance, a large apartment building fire might require fire fighters, police for crowd and traffic control, shelter workers and counselors for displaced residents, and even animal control to help rescue and reunite pets and families separated in the fire.

With a history of crisis responses to analyze with other information about weather, building inspection reports, crime trends, and even social media, city officials can expand situational intelligence to crisis prediction models, early warning systems, and crises prevention programs.


Situational Intelligence and Smart Cities: Energy, Part 1


The previous entry in this smart cities series examined how situational intelligence could be used to improve and optimize municipal water systems. Let’s look at how progressive use cases can help evolve smart infrastructure for city energy services and consumption.

As a first step, situational intelligence supports aggregating the energy consumption of city facilities into a single view of energy usage across city operations. Aggregation leads to efficiency in accounts payable and provides leverage for negotiating better energy rates with suppliers.

With the savings from accounting efficiencies and lower energy rates, city officials can invest in the energy efficiency of city facilities. Increased energy efficiency lowers costs, and the savings flow directly to the bottom line for improved fiscal health. Situational intelligence supports analyzing and scoring buildings according to their energy efficiency and projecting savings from capital investments in efficiency.

With buildings and facilities operating more efficiently, cities are in a position to begin implementing microgrids and energy storage devices to keep city services running in times of crisis, take advantage of renewable energy sources such as solar and wind, and participate in demand response programs that would increase reliability, lower energy bills and reduce carbon footprint. Situational intelligence maps and projects energy availability from storage devices and from renewable sources.

Energy and water, our most recent topics in this series, are constant needs for city residents. In a future blog post, we’ll see how progressive use cases might help to evolve services of a more intermittent nature, such as crisis response.


Situational Intelligence and Smart Cities: Water


Previously, we looked at how progressive use cases can help cities evolve their smart infrastructure, and how that might work for city transportation. Let’s look at how this evolution might work for municipal water systems.

As a first step, situational intelligence supports mapping water flows through meters across the city. By comparing projected and actual water flows, situational intelligence can locate water mains most in need for repair or replacement. Critical and proactive repairs can help address the 10 percent or more of water lost in most distribution systems due to leaks. Reclaiming lost water raises production, which saves money by either increasing sales or holding down operating costs.

With the savings from repairing leaks, a city can add a customer service module to their situational intelligence platform to provide better service to customers at lower operating costs. At the same time, customer satisfaction with city water service can rise.

Once a city can map water flows across the distribution system and increase customer service and satisfaction, it is in position to implement a water conservation program to increase sustainability and ensure water supply for future economic development. Situational intelligence supports water conservation by flagging anomalies in water consumption patterns governed by conservation program rules.

In a future blog post, we’ll see how this progression can work for energy.


Situational Intelligence and Smart Cities: Transportation


In the opening of this series of posts on situational intelligence and smarter cities, I outlined how cities can progress through solving different types of use cases to build their smart infrastructure of the future while also addressing current needs. Let’s look at how this might work for transportation in an evolving city.

As a first step, city officials can use situational intelligence to score and rank the most congested portions of city roadways and test possible solutions. Cities can also score and rank the risk posed by aging and overworked bridges, overpasses, and tunnels. Quantifiable evidence of the probability of asset failure, and the consequences should failure happen, provides strong evidence for seeking new funds to infrastructure improvement.

After addressing the most congested and riskiest transportation corridors, city planners can use situational intelligence to study and improve public transportation speed and capacity through major corridors. This helps to further reduce congestion and degradation of city streets.

Using situational intelligence to bring together traffic information, city planning, environmental data, economic development, and population growth data, city planners can design future neighborhoods and transportation corridors that provide increased convenience and safety with decreased noise and air pollution and traffic congestion.

In future blog posts, I’ll lay out similar use case progressions that can help evolve smarter cities today.


Optimizing Crew Dispatch


The service call seems like a dying art. For repairs at your home, it can be hard to pin down a repair technician to a four-hour arrival window. When the technician arrives, will he or she have the right training, tools, and supplies to diagnosis and repair the problem? If not, you’re stuck with follow-up appointment sometime within another four-hour arrival window. Magnify this problem for mission-critical equipment costing thousands or millions of dollars, and service calls become a critical event in asset-intensive industries.

Field service embodies complexity that can benefit from situational intelligence. One large utility in Canada found that it could save millions of dollars a year simply by better scheduling of field work crews. The utility supplies power to a sprawling, rural service territory. There are more power poles than people, and drive times are long. When you’re paying repair crews by the hour to drive great distances, reducing the number of trips quickly produces savings.

The first way to optimize crew dispatch is accurately locating and diagnosing the problem. This is easy when a homeowner calls to say that their dishwasher doesn’t work. It’s much harder with geographically large and diverse networks such as utilities and transportation. The growing ubiquity of sensors leading to the Internet of Things can assist in location and diagnosis. Sensor data benefits from situational intelligence’s enrichment with IT and external data, such as financial information and weather data.

Once the problem has been located and properly diagnosed, optimization depends on sending the right personnel at the right time with the right tools and supplies. This requires correlating information from multiple data sources such as personnel records (to determine skill sets of repair technicians), workforce management, customer service, fleet management, inventory management and more. Many of these sources could be manually queried and correlated given enough time and staff; because repairs are often rush jobs, situational intelligence simplifies data correlation to arrive at the optimal repair strategy.

For proactive organizations, optimizing crew dispatch means doing double duty. As long as a crew is repairing a fault on a particular device, is there any other upcoming maintenance that they should perform? This would save dispatching another crew to the same spot in the near future. Likewise, are there any other repair or maintenance needs near the item to be fixed? Doing work while in the vicinity increases efficiency, and possibly reliability as well. The only way to quickly spot opportunities for doing double duty is having situational intelligence in place for your asset maintenance and operations. And of course, serendipitous work needs to be factored into planning personnel, tools, and supplies.

How could, or does, your organization use situational intelligence to restore the lost art of service calls?




Security: A Top Five Trend in Analytics for 2015


The Institute for Operations Research and the Management Sciences (INFORMS) recently named security one of the top five trends in analytics for 2015.

As stated in the INFORMS announcement:

With the number of security breaches on major corporations being reported almost weekly, such as at Target, The Home Depot and more recently at Sony Pictures, there will be a significant increase in investment across the board in safeguarding commerce and privacy on the Internet. The importance of applying analytics methods – from using decision analysis to guide investment choices, to statistical methods, to detect-and-anticipate breaches and optimization models, to improve infrastructure design for safety, reliability and performance – will accelerate and continue to grow in 2015.

As we wrote in a recent post, situational intelligence provides a crucial, comprehensive platform for use in preventing, preparing for, detecting and responding to physical and cyber security threats.



Situational Intelligence and Security


To be continually vigilant, asset-intensive organizations need to prepare for and prevent the possibility of physical and cyber attacks, and then quickly and accurately detect and respond to any attacks that do occur. Preparation for, prevention and detection of, and response to attacks each introduces degrees of complexity to the security challenge. The typical utility has many siloed systems independently evaluating security measures, making it difficult to connect isolated but possibly related events and making the security challenge even more complex.  As a result, potentially dangerous situations may be ignored while investigation into false alarms consumes time and money

Preparing for and preventing attacks

When preparing for attacks a utility should assess both the likelihood of attack and the consequences of an attack on each asset. The probability of attack on organizational assets often comes down to motive and location.

  • Are physical systems and assets safe from authorized field workers, contractors, and ex-employees?
  • Are cyber systems and assets safe from third party and external threats such as denial-of-service (DoS) attackers?
  • Is a physical asset easily accessible to those wishing to cause damage or disruption?
  • Is a cyber asset – such as a computer, control system or communication network component – indicating unexpected connections, failed logins or uncharacteristically extended response times?

Organizations also need to assess the consequences should an asset be attacked. An asset might be very vulnerable to attack, but the consequence of losing that asset might be negligible, reducing the priority of safeguarding that particular asset over other, more important assets.

By correlating data from disparate assets across the organization to calculate the probability and consequence of attack, situational intelligence solutions provide an accurate ranking of security risks that can be proactively addressed.

Detecting attacks

The number of assets at large organizations can run from the tens of thousands to the tens of millions. Such scope means a huge area to patrol and lots of data to protect. To separate actual attacks from malfunctions or flukes, it is useful to correlate multiple data points that occur close to each other in both space and time.

In a large scale event, dozens of alarms may trigger at once. For instance, if an entire building is somehow damaged, all alarm-equipped assets in that building will send out signals. Instead of receiving dozens of individual notifications, operators would benefit from a system that correlates those individual items in real-time into a single, larger, more meaningful alert.

Using situational intelligence solutions, data and alarms from multiple, disparate sources can be correlated and presented to users in a single view, drawing attention to anomalous conditions and facilitating fast, informed decision-making.

Responding to attacks

Your situational intelligence system has detected an attack—now what? First, you need to understand exactly what has happened. Because situational intelligence correlates data across the dimensions of space, time and network node, operators can quickly close in on the root cause of an event. They can also see at a glance the network impact upstream and downstream of the event.

Next, you need to know who should be notified, which repair crews should be dispatched where, which first responders to contact, and what reports need to be filed. The period immediately following an attack is critical for controlling damage, preventing injury, collecting evidence and apprehending suspects.

Once an attack has been resolved, it’s good to review process and procedures, to improve security and prevention and to better prepare for the next possible attack. Situational intelligence systems can capture spatial-temporal-nodal information for later analysis. This helps operators, administrators and investigators study, assess and revise responses to attacks.

For more information about situational intelligence and security, see this white paper.


Converting Big Data to Little Data for Big Payoff


The secret to benefiting from Big Data lies not in accessing all the data, but in identifying, analyzing and acting on the right subset of data.

The primary goal of situational intelligence is to simplify access to high volumes of heterogeneous data and transform it to actionable information. With situational intelligence, users have the flexibility to hide or display the data they want to see on-the-fly, view performance over time, get a birds-eye view of a situation and drill-down to the details of specific assets to troubleshoot root causes, interface to related documents and applications to follow defined procedures, examine diagrams and asset documentation, or take corrective actions.

For instance, it is difficult to identify infrequent errors or combinations of factors within millions of data records by using traditional display formats such as tables and charts. However, the combination of geospatial visualizations, temporal displays and anomaly detection models can alert users immediately to the fact that a problem occurred and pinpoint precisely where and when it happened (and might happen again).

How would this work in the real world? Consider the 2+ million vehicle fleet of the United States Postal Service and the associated need for tires. Simply tracking all those vehicles simultaneously on a GIS application, waiting to identify vehicles with tire problems, is pointless and inefficient. Similarly, searching through a tabular report on all 2+ million vehicles looking for evidence of old or risky tires is so overwhelming as to be useless.

What if a USPS district purchasing manager wanted to avoid towing bills and downtime from tire failures by building a forward-looking budget for purchasing new tires for the coming 12 months?

Correlating and analyzing data on the types of vehicles, dates of tire purchases, miles traveled per month, typical weather and road conditions or other location information can give more precise information about current tire wear. This combination of data can identify and rank the vehicles that will potentially need new tires.

Analyzing this information with contracts and price lists from tire suppliers and budget projections for the coming year, the district purchasing manager can easily make data-driven decisions about which few vehicles out of thousands in her fleet require new tires. Plotting those vehicles and their routes on a map shows which delivery areas will be affected by tire replacement, and how many substitute vehicles might be needed to cover for those out of service for tire replacement. That keeps mail arriving on time and might even postpone the next price increase for stamps.


Analytics Forecasting Wind and Wildfire Damage in San Diego


Thirty terabytes of data per day, every day–that’s what San Diego Gas & Electric (SDG&E) collects and analyzes to understand weather across their service territory.

As described in this Intelligent Utility article, SDG&E correlates temperature, humidity, wind speed, and solar radiation data with customer and grid operational data to prevent wildfires, promote public safety, and proactively deploy crews to areas at risk for wind or fire damage.

This correlation of IT, operations, and external data is a clear example of situational intelligence in action. Acting on data-driven insights to proactively deploy crews allows SDG&E to keep customers and infrastructure safe, prevent outages, and reduce the duration of outages that do occur.

Understanding weather is only one area where SDG&E is deploying analytics to make sense of big data in real time. “We expect that analytics will play a role in most or all of our business operations,” says Hanan Eisenman, SDG&E communications manager, quoted in the article. Other areas include IT, security and customer service.