Big data is often talked about in the abstract. We can now collect huge numbers and apply clever algorithms, so we think the answers must always be better and can be taken straight from the new technology. But the devil is in the detail.
With data collection and algorithms running on auto pilot, they can end up averaging a huge quantity of measurements down to a few numbers. Errors and unwanted information can be buried within a mass of records. Big data sets, make it easier to force a correlation that isn’t necessarily accurate.
Modern survey data collection techniques, such as video recording and Number Plate Recognition software, has given us the illusion of a 100% perfect sample. This is not true and adjustment factors and factoring of results are still a significant part of the final result.
The theme of this paper is to take an origin destination survey data set and apply a range of fuzzy logic matching techniques that might be applied to produce a trip table with a greater number of matches and then test the results for their credibility. The pressure to drive more numbers into the matched tables creates distortions.
By correlating with travel time results and managing the impaired entries in a data file of vehicle registration plates, debunks the usage of a common technique of just substituting any two characters of a plate record to force a greater number of matched events.
There can be a genuine application of fuzzy logic to enhance a match rate, but this has to be done in the context of traffic engineering skills, knowledge and wisdom in producing credible data outcomes.
Proper data collection, management and processing is a critical part of our profession which is too often not taught or over looked.
Read More (download PDF)
Presented by John Reid, Austraffic Managing Director, at the 2018 AITPM National Conference
