\chapter{Validation Techniques}
\label{ch:valid}

Once a model is set up, there is a need to access its quality.  In order to capture how well do the estimated probabilities of defaults match the true defaults, there is a need to derive a method beyond goodness of fit techniques.  The common technique of assessment is to analyze the accuracy ratio of the estimated probabilities.  The ratio is a calculation of the cumulative accuracy ratio (CAP) curve.  The CAP curve measures discriminatory power of the model's estimates.  The curve is a way to visually see if the defaults are occurring with those issuers that have lower ratings.  The area between the CAP curve and above the diagonal is the accuracy ratio.  The diagonal line represents the probability of guessing. There is a need to maximize this area.  The area above the curve is used because the latter gives the expected CAP curve of an uninformative rating system which does not discriminate at all between low and high default risks. \cite{stien02}  No CAP calculation of the cap curve was reported in this paper, because of the lack of information of the actual ratings of all the issuers in the sample space, such as \textbf{AAA, Aa3,...}.  However, for each model, the accuracy ratio was calculated.

\section{Bootstrapping Confidence Intervals for Accuracy Ratios}

The accuracy ratios are estimates of the models ability to discriminate between high and low credit risks against the actually defaulted issuers.  In this paper, to obtain a confident estimate, used bootstrapping simulations to derive a a confidence interval.  After reading related papers on default models, much of the validation was presented using CAP and accuracy ratios.  Furthermore, confidence intervals were presented, therefore, this paper will follow this procedure.  Analyzing the outputs, there is small difference in the accuracy ratio, however, overall model 2 and 3 do substantiate results from previous sections.



\subsection{Bootstrapping Confidence Intervals: Output}
\begin{figure}
\includegraphics[scale=0.4]{boot.png} 
\caption{Confidence Intervals for Accuracy Ratios of Model.}
\end{figure}



\subsection{Accuracy Ratios Confidence Intervals: Interpretation}
\begin{itemize}
\item \underline{Model 1:} The accuracy ratio is $85.29\%$, the bootstrapped $95\%$ confidence interal is $[68.84\%, 95.28\%]$
\item \underline{Model 2:} The accuracy ratio is $84.13\%$, the bootstrapped $95\%$ confidence interal is $[68.48\%, 96.43\%]$
\item \underline{Model 3:} The accuracy ratio is $85.22\%$, the bootstrapped $95\%$ confidence interal is $[68.42\%, 94.24\%]$
\end{itemize}



