SecurityScorecard’s scoring update involves a monthly recalibration, which is an update to the baselines that we use to calculate our scores in order to reflect changes to companies, the cybersecurity landscape, and the internet. Our scoring algorithm is a measure of the typical number of findings for an organization versus their digital footprint size. The score is developed based on how many standard deviations an organization is better or worse than the average number of findings for an organization of a particular digital footprint size. We recalibrate to make sure that the average number of findings of issues and factors is up to date.
We update the scoring calibration monthly using two months worth of data. We believe this is the best option for customers because the expected score impact of recalibrations is minimized and this allows for frequent improvements to the platform. The monthly cadence also guarantees our commitment to unbiased and fair scores.
SecurityScorecard provides users a top-level score and factor-level scores for their own company and those that they monitor in their portfolios. A company’s top-level score consists of all the factor scores combined and weighted to capture an accurate representation of the cyber threat landscape, while factor scores consist of different issue types. As we add or remove issues and information with recalibration, certain factor scores may change while others stay the same. Changes in a company’s factor-level score will lead to a change in their top-level score.
The figure to the left shows the calibration for the open resolver finding (open resolver is a finding where a DNS service is improperly configured in such a way that allows it to be abused for malicious purposes). For both axes we take the logarithm base 10 to accommodate the several orders of magnitude spread in the data. Therefore 3 on the x axis means 10^3 or 1,000 IPs. The hexagonal bins are color coded based on how many companies fall into that area. Hexagons that are red have more data points than hexagons that are blue. The red dots are the average values calculated for various ranges of IPs. The pink line represents the average value of findings for an organization of any digital footprint size. The orange lines are the average plus the standard deviation and the average minus the standard deviation respectively. When we calibrate we calculate the pink and orange lines.
Why are Scoring Updates important?
By keeping the calibration up to date, SecurityScorecard provides customers with scores that continuously reflect the true nature of the cybersecurity landscape. This prevents scores from drifting over time as the number of findings on the internet moves away from the historical baseline.
Additionally, scoring updates allow us to add new issue types, retire old ones, and re-weight the severity of issues which make scores more predictive of negative outcomes such as breaches and malware. We can’t add new issue types without understanding the baseline to score against, which we get from recalibration. Factor scores are a weighted combination of issues, which are also recalibrated. Each issue is contributing some expectation value to the factor score. If we remove an issue, we need to recalibrate the factor without it.
When there are no changes to issues and weights, we can expect the score changes to be less significant than when we add in new issues or re-weight the severity of issues.
Since scoring updates affect all Scorecards in the platform, users will also experience score changes within their portfolios. Additionally, the reports, analytics, and other features based on portfolios scores might change.
What To Look for In the Platform
In Platform Banner
For users that will experience a change, in platform banners will communicate the potential score impact.
Mouseover of the scoring update icon under your score on your Scorecard page provides information on your potential score impact.
The scoring update icon on the History Page and Issue Level Event Log will indicate score changes due to calibration
The score impact in the platform is a reflection of how the recalibration change impacts your score on the day it was computed. Since we run scores daily, the impact may change as we use new data.
If you have any questions or comments, please contact us at firstname.lastname@example.org.