Model: To underscore that our method serves as an off-the-shelf method that is applicable to different policies, we ablate with a Multilayer Perceptron (MLP) based single-step policy and a GPT-like causal transformer policy. This MLP ingests a combination of the frozen visual embeddings from step-wise RGB observations and goal images followed by a 1D BatchNorm, as well as the 9D proprioceptive data encoded through a single layer complemented by a LayerNorm.
Our GPT policy removes the BatchNorm and replaces the MLP with the causal self-attention blocks consisting of 8 layers, 8 heads, and an embedding dimension of 768. We set an attention dropout rate of 0.1 and a context length of 10. We transition from the conventional LayerNorm to the Root Mean Square Layer Normalization (RMSNorm) and enhance the transformer with rotary position embedding (RoPE). Actions are predicted via a linear. At inference time, we cache the keys and values of the self-attention at every step, ensuring that there's no bottleneck as the context length scales up. Nevertheless, in the FrankaKitchen tasks, we observed that a longer context length tends to overfit and performance drop. Therefore, we consistently use a context length of 10 for all experiments. More details can be found in appendix.

Reinforcement Learning: All RL experiments are trained using the Proximal Policy Optimization (PPO) RL algorithm implemented within the AllenAct RL training framework. In our RL setting, the configurations for both training and inference remain consistent. This is analogous to the inference for IL as detailed in Appendix. A2. Specifically, the task is also specified by an unlabeled video trajectory $\tau$. Given the initial observation $o_0$ and UVD subgoal $g_0\in\tau_{goal}$, the agent continuously predicts and executes actions conditioned on subgoal $g_0$ using the online policy with frozen visual encoder $\phi$, until the condition $d_\phi(o_t;g_0) < \epsilon$ satisfied for some timestep $t$ and positive threshold $\epsilon$. As shown in Eq. 4, we provide progressive rewards defined as goal-embedding distance difference using UVD subgoals. Recognizing that the distance between consecutive subgoals can vary, we employ the normalized distance function: $\bar{d_\phi}(o_t;g_i) := d_\phi(o_t;g_i) / d_\phi(g_{i-1}; g_i)$. This ensures that $\bar{d_\phi}(o_{t-h};g_i) \approx 1$ for some timestep $t-h$ that the subgoal was transitioned from $g_{i-1}$ to $g_i$. Additionally, we provide modest discrete rewards for encouraging (chronically) subgoal transitions, and larger terminal rewards for the full completion of task sub-stages, which is equivalent as the embedding distance between the observation and the final subgoal becomes sufficiently small. To sum up, at timestep $t$, the agent is receiving a weighted reward \begin{equation}\label{eq:literal_rl_rewards} \begin{aligned} R_t &= \alpha\cdot\left(\bar{d_\phi} (o_{t-1};g_i) - d_\phi(o_t;g_i) \right) \\ &+ \beta \cdot \mathbf{1}_{\bar{d_\phi}(o_t;g_i) < \epsilon} \\ &+ \gamma \cdot \mathbf{1}_{\bar{d_\phi}(o_t;g_m) < \epsilon} \end{aligned} \end{equation} based on the RGB observations $o_t, o_{t-1}$, corresponding UVD subgoal $g_i\in\tau_{goal}$, and final subgoal $g_m\in\tau_{goal}$. While similar reward formulations appear in works, we are the first in delivering optimally monotonic implicit rewards unsupervisedly by UVD, derived directly from RGB features. In our experiments, we use $\alpha=5, \beta=3, \gamma=6, \epsilon=0.2$, and confine the first term within the range $[-\alpha, \alpha]$ in case edge cases in feature space. For the final-goal-conditioned RL baseline, it is equivalent as $g_i = g_m = o_{T}\in \tau = \{o_0, \cdots, o_T\}$ and $\beta=0$ in Equation above. Tab. II illustrates that the simple incorporation of UVD-rewards greatly enhances performance. We also showcase a comparison of evaluation rewards between GCRL and GCRL augmented with our UVD rewards. This is done using the R3M and VIP backbones, as seen in Fig. 2. This highlights the capability of our UVD to offer more streamlined progressive rewards. This capability is pivotal for the agent to adeptly manage the challenging, multi-stage tasks presented in FrankaKitchen. To the best of our knowledge, ours is the first work to achieve such a high success rate in the FrankaKitchen task without human reward engineering and additional training. Notably, our RL agent, trained with the optimally monotonic UVD-reward, can complete 4 sequential tasks in as few as 90 --- a stark contrast to the over 200 steps observed in human-teleoperated demonstrations. This further illustrates the UVD-reward's potential to encourage agents to accomplish multi-stage goals more efficiently. The videos of rollouts can be found on our website.

Inference: We now elucidate the specifics of applying UVD subgoals in a multi-task setting during inference. Remember that given a video demonstration represented as $\tau = (o_0, \cdots, o_T)$ and UVD-identified subgoals $\tau_{goal} = (g_0, \cdots, g_m)$, we can extract an augmented trajectory labeled with goals, represented as $\tau_{aug} = {(o_a, a_0, g_0), \cdots, {o_T, a_T, g_m}}$. This is useful for goal-conditioned policy training, as discussed in Sec. IV-B. For inference, we can similarly produce an augmented offline trajectory without the necessity of ground-truth actions, i.e., $\tau_{aug,infer} = \{(o_0, g_0), \cdots, (o_T, g_m)\}$. In the online rollout, after resetting the environment to $o_0$, the agent continuously predicts and enacts actions conditioned on subgoal $g_0$ using the trained policy. This continues until the embedding distance between the current observation and the subgoal surpasses a pre-set positive threshold $\epsilon$ at a specific timestep $i$, i.e. $d_\phi(o_i; g_0) < \epsilon$, where $\phi$ is the same frozen visual backbone used in decomposition and training. Following this, the subgoal will be seamlessly transitioned to the next, continuing until success or failure is achieved. In practice, the straightforward goal-relaying inference method might face accumulative errors during multiple subgoal transitions, especially due to noise from online rollouts. However, when an agent is guided explicitly by tasks depicted in a video, incorporating the duration dedicated to each subgoal can help reduce this vulnerability. To clarify, once we've aligned subgoals with observations from the video, we also draw a connection between the timesteps of observations and their corresponding subgoals. We denote the subgoal budget for subgoal $g_i = o_t$ as $\mathcal{B}_{g_i} := n + 1$ where $g_{i-1} = o_{t-n-1}$ based on Eq. 3. Building on this, we propose a secondary criterion for switching subgoals: verify if the relative steps completing the current stage are in the neighborhood of the subgoal budget. This measure ensures timely transitions: it avoids prematurely switching before completing a sub-stage or delaying the transition despite accomplishing the sub-stage in the environment. To sum up, given an ongoing observation $o_t$ and subgoal $g_i$ at timestep $t$, and considering the preceding subgoal $g_{i-1}$ at timesteps $t - h$, the subgoal will transition to $g_{i+1}$ if \begin{equation} d_\phi(o_t; g_i) < \epsilon \quad \text{and} \quad |h - \mathcal{B}_{g_i}| < \delta \end{equation} We use $\epsilon=0.2$ and $\delta=2$ steps for all of our experiments, except in baseline tests that are conditioned solely on final goals.