
\newpage
\section{When is P-opening worthwhile?}


%\subsection{When is P-opening worthwhile?}
\label{sec:numerical-apx}

We conduct an additional numerical study with the goal of examining when P-opening is worthwhile. 
We apply a similar process for generating instances (Section~\ref{sec:numerical_instances}) with the exception choosing 100 as the number prototypical boxes, which are illustrated in Figure~\ref{fig:scatter_boxes}.


\begin{figure}[!h]
  \centering
  \includegraphics[width=7cm]{experiments_graphs/boxes_scatter.png}
  \caption{An example of how prototypical boxes are generated. Each point is a box sampled from the distribution. Red triangles are those which are selected, while blue circles are those which are not. }
  \label{fig:scatter_boxes}
\end{figure}



To get a sense for when P-opening is particularly useful under an optimal policy, we define the \textit{P-ratio}. This is the ratio between the expected number of P-openings and the total expected number of P-openings and F-openings (of closed-boxes), both computed for an optimal policy at the initial state of a problem instance.


\begin{figure}[htbp]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=0.8\textwidth]{experiments_graphs/when_P_is_important.png}
\end{subfigure}%
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=0.8\textwidth]{experiments_graphs/Weitzman_performance.png}
  \label{diag:Weitzman_performance}
\end{subfigure}
\caption{ When is P-opening important? \\
Left figure: P-ratio of $\pi^{\textrm{OPT}}$ versus proportion  of boxes with $\sigma_i^P > \sigma_i^F$. \\
Right figure:   Suboptimality (ratio of the value of $\pi^{\textrm{Weitzman}}$ to that of the optimal policy)) of $\pi^{[N], \emptyset}$ versus proportion of boxes with $\sigma_i^P > \sigma_i^F$. }
\label{fig:when_P_important}
\end{figure}


Figure \ref{fig:when_P_important} (left) plots the P-ratio of $\pi^{\textrm{OPT}}$ versus the proportion of boxes with $\sigma_i^P > \sigma_i^F$. We observe a positive correlation between these two quantities, indicating that when there are more boxes with $\sigma_i^P > \sigma_i^F$, the optimal policy tends to take advantage of P-opening more frequently.
We also calculate the performance of the policy $\pi^{\textrm{Weitzman}}$, which simply ignores all the P-opening options, i.e., the committing policy $\pi^{[N],\emptyset}$. Figure \ref{fig:when_P_important} (right) shows its suboptimality versus the proportion of boxes with $\sigma_i^P > \sigma_i^F$. We similarly observe that it tends to perform worse when there are more boxes with larger P-thresholds.
These observations further suggest that the interaction between the P-thresholds and F-thresholds impacts optimal decisions, and the relative magnitude of P-thresholds compared to F-thresholds determine the extant to which P-opening should be utilized. 
This may also explain why the simple index policy $\pi^{\textrm{index}}$ performs so well.






Next, we consider the heterogeneity of boxes. 
For each problem instance, we define \textit{dispersion} as the sum of standard deviations of the P-thresholds and F-thresholds that are associated with closed boxes. 
Intuitively, when the dispersion is higher, boxes are more heterogeneous. The upper panels of Figure~\ref{fig:P_F_dispersion} illustrate dispersion. Each panel shows a problem instance, where every point on the scatter plot represents a box (e.g., the left panel shows an instance with 9 boxes, while the right panel shows a problem instance with 10 boxes). The coordinates of each box correspond to its F-opening and P-opening thresholds. 
Intuitively, the instance on the left has boxes that are more homogeneous while the instance on the right has boxes that are more heterogeneous, which are reflected by the dispersion metric (the dispersion of the left instance is smaller than 5, while the dispersion of the right instance is greater than 20). 


\begin{figure}[htbp]
        \centering
        \begin{subfigure}[b]{0.475\textwidth}
            \centering
            \includegraphics[width=0.8\textwidth]{experiments_graphs/boxes_scatter_small_dispersion.png}
            % \caption[]{\small Example instance where dispersion is relatively small ($<5$). }
            % \label{fig:mean and std of net14}
        \end{subfigure}
        \hfill
        \begin{subfigure}[b]{0.475\textwidth}
            \centering
            \includegraphics[width=0.8\textwidth]{experiments_graphs/boxes_scatter_big_dispersion.png}
            % \caption[]{\small Example instance where dispersion is relatively large ($>20$). }
            % \label{fig:mean and std of net24}
        \end{subfigure}
        \vskip \baselineskip
        \begin{subfigure}[b]{0.475\textwidth}
            \centering
            \includegraphics[width=0.8\textwidth]{experiments_graphs/P_F_dispersion.png}
            % \caption[]{\small P-ratio of the optimal policy versus dispersion.}
            % \label{fig:mean and std of net34}
        \end{subfigure}
        \hfill
        \begin{subfigure}[b]{0.475\textwidth}
            \centering
            \includegraphics[width=0.8\textwidth]{experiments_graphs/more_boxes.png}
            \label{fig:more_boxes}
            % \caption[]{\small P-ratio of the optimal policy versus the number of boxes. }
            % \label{fig:mean and std of net44}
        \end{subfigure}
        \caption[]
        {\small Heterogeneity of boxes. 
        Figures from left to right, top to bottom correspond to:  example instance where dispersion is relatively small ($<5$),
        example instance where dispersion is relatively large ($>20$),
        P-ratio of the optimal policy versus dispersion, and
        P-ratio of the optimal policy versus the number of boxes. 
        }
        \label{fig:P_F_dispersion}
\end{figure}


The lower-left panel of Figure~\ref{fig:P_F_dispersion} plots the P-ratio versus dispersion. We observe that, when the dispersion is large (i.e., above 15), the P-ratio tends to be small, meaning that there are relatively fewer P-openings in the optimal policy. Intuitively, when boxes are very different in terms of the potential information gain which is proxied by the opening thresholds, boxes are more likely to dominate each other. In such settings, an optimal policy tends to directly F-open the boxes according to a predefined order. 


Finally, the lower-right panel of Figure \ref{fig:P_F_dispersion} shows the P-ratio versus the number of boxes of the problem instance. To generate this figure, we randomly sample several box instances, vary their P-opening costs, and investigate how an optimal policy behaves when there are $N$ i.i.d. such boxes. We observe that the optimal policy exploits P-opening more often when there are sufficiently many boxes. The panel complements panel (c), showing that when the dispersion is low (here instances are homogeneous), there is interaction affect with the number of boxes. 






