Set up template questions and scoring in Titan AI

Whether you're finishing a template you just created or coming back to change one later, the same tools apply: go to My Network > Templates, open the template, select Edit, and use the options below to configure questions, make bulk changes, and set up scoring. If you haven't created a template yet, see Create a template.

Configure individual questions

Click into any question to access a few additional options that control how it behaves and how answers are handled:

• Optional: Mark a question as not required. Questions are required by default; making one optional lets respondents leave it blank and still submit the questionnaire.
• LLM evaluations: Use AI to automatically evaluate a respondent's answer, so you don't have to review every response manually. See Set up AI evaluations below.
• Question display conditions: Show a question only when it's relevant, for example, based on how a previous question was answered, so respondents see just the questions that apply to them.

Edit with Titan Agent

Titan Agent provides a chat interface that lets you edit faster than clicking through the interface. Titan Agent is especially useful for making bulk changes, such as rewriting response types across multiple questions at once.

Note: Titan Agent isn't available on Global (organization-shared) templates.

1. Describe the change you want in plain language. For example:
   • "Rename the section called Sheet1 to Network Security"
   • "Change question 4 to a multiple-choice question with options Yes, No, Partial"
   • "Add an LLM evaluation question to question 7"
2. Titan Agent proposes the edit for your review before applying it.

Set up AI evaluations

AI evaluations run automatically when a vendor submits their questionnaire. They surface issues and flag answers that don't meet your criteria, so your team can focus on exceptions instead of reviewing every response.

To set up AI evaluations, open the question you want to configure and expand the LLM evaluations section.

You can add multiple evaluation types to a single question. Three evaluation types are available: Policy check, Document extraction, and Evidence check.

Policy check

Use Policy check to define an acceptable answer for a specific question. If the vendor's answer meets your policy, it passes silently. If it doesn't, Titan AI flags it in the questionnaire summary.

• Enter your policy criteria in the Instructions field. For example, "TLS 1.2 or higher."
• The Instructions field is optional. If you leave it blank, the LLM assesses the answer based on the question context alone.

Document extraction

Use Document extraction to pull specific fields out of vendor-uploaded documentation, such as a SOC 2 report. Extracted values appear in the questionnaire summary, and Titan AI flags any it can't find.

• Select Add field. Name the field and add an optional instruction describing what to look for. For example: field name "Audit date," instruction "The date the most recent SOC 2 audit was completed."
• By default, the LLM searches only the file attached to that specific question.
• To search all documents uploaded to the questionnaire, select Include all vendor evidence in scope.

Evidence check

Use Evidence check to verify that a vendor's text answer is supported by their uploaded documentation. For example, confirm that a claim like "we encrypt data at rest" is backed by evidence.

• Add an instruction so the LLM knows what to look for. For example, "Does the vendor's documentation confirm that data at rest is encrypted using AES-256 or equivalent?"
• By default, the LLM reviews only the file attached to that question.
• To search all documents attached to the questionnaire, select Include all vendor evidence in scope.

Document extraction isn't a prerequisite for Evidence check; you can use Evidence check on its own. If the documentation doesn't support the answer, Titan AI flags it in the questionnaire summary.

Set up risk scoring

Risk scoring lets you assign a numeric risk weight to individual questions and calculate an overall risk score for each vendor assessment, so you can compare vendors and prioritize follow-up instead of reading every answer manually. Scoring is supported on select (single/multiple-choice), text, and file-upload questions.

How it works

Define a risk matrix with named risk levels, for example, Low, Medium, and High, each carrying a point value for a failing answer. Tag questions with a risk level, and failing answers accumulate points based on that level. A higher score indicates greater risk.

Note: More points mean more risk. Point values can include decimals if you need finer-grained weighting.

To set up risk scoring:

1. Open your template and select Risk Scoring at the top of the screen.
2. Turn on Pass/Fail. This lets you designate answers as passing or failing.
3. Under For this template, select an existing risk matrix from the dropdown, or choose New risk matrix.
4. Your matrix includes Low, Medium, and High risk levels by default. For each level, set the number of points assigned when a question at that level receives a failing answer.
5. To assign a risk level to a specific question, open the question and choose the appropriate level. Titan Agent can help you apply risk levels across multiple questions at once.

You can manage and reuse matrices across templates from Manage matrices.

Scoring reference (default matrix)