Pretrained vision-language-action models for robots often fail when a task is performed in a slightly different environment. With Vision-Language Steering, you can get better generalization performance out of a frozen robot policy, by steering the sampling process of your pretrained diffusion or flow-matching policy to handle out-of-distribution changes like obstacle position or changing visual appearance. Shuo Lin and Ishneet Sukhviner Singh join us to talk about how to better generalize robot policies with no new training data.
Watch Episode 108 of RoboPapers today, with Jiafei Duan and Ruijie He!
Abstract
Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train–test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time.
We propose Vision–Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation–language inputs without modifying policy parameters. By leveraging vision–language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements.
Learn More
Project Page: https://vision-language-steering.github.io/webpage/
ArXiV: https://arxiv.org/pdf/2602.03973
Transcript
Transcripts are lightly edited by ChatGPT.
RJ He (00:01)
Okay. Hi everyone. Welcome to another episode of Robot Papers. Today we are super excited to have with us Shuo Liu and Ishneet, who will be sharing their paper on Vision-Language Steering: Steering Pre-Trained Robot Policies via Vision-Language Models. Shuo, Ishneet, do you all want to introduce yourselves?
Shuo Liu (00:29)
Yes. Hi everyone. My name is Shuo Liu. For now I’m a research engineer at Cortex Robot AI and previously I graduated from University of Washington, where I was also a student researcher at Allen Institute for AI, advised by Professor Ranji Krishna and Professor Jiafei Duan, focus on robot learning from develop foundation models to deploying them into real world.
RJ He (01:01)
Isn’t it?
Ishneet (01:02)
Yeah. Hi everyone, my name is Ishneet. I work with Professor Jiafei on this paper, VLS, with Shuo. I’m currently at Menlo Research, where I also work on Asimov, and I’ll be going to Oxford for my undergrad later this year.
RJ He (01:17)
Super exciting. Amazing that you’re already having publications at this young age. It’s awesome. Okay, without further ado, Shuo, why don’t you take us through the paper?
Shuo Liu (01:26)
Okay, sure, yeah. Let me share the screen. Okay, so just make sure it’s it’s good to say right.
RJ He (01:38)
Yeah, it’s good.
Shuo Liu (01:39)
Yeah. Hi everyone. Today I want to share our work vision language theory. This work is finished when I was a master student at the University of Washington and advised by Professor Ranjik Krishna and Professor Jiafei Duan. And so today I want to introduce motivation of this work and the method we solve this problem and the experiment results and also some future works maybe could be done from this direction. So the problem is from the observation of some small front twining of the Like models like π0 or π0 Po, we found that current generative robot policies are can’t are kind of tightly coupled of the training specific configurations, so which cause some failing to decouple the reusable motor skills from the normal task constraints. Like for example, here we have some cubes on the table and if you want to stack them up to a tower with specific numbers you may call the π0.5 pre-trained value a policy to stack some cubes like if you want to stack two cubes and using the termination signal like the pushing the red button on the table it can do very well because you Because for this specific task you can fine twin the base policy a bit, like stack two cubes and push the button or stack four cubes to push the button. And but what if at test time you ask the base policy to stack three cubes? It’s never it’s never contained in the fine-tuning data set, so the base policy
Shuo Liu (03:33)
Can learn how to stack two cubes or stack four cubes and push the button, but it will never know how to st how to stack three cubes and push the button. This is because the training configuration we combine the language instruction and the image observation with the task two cubes and four cubes together and the polic the base policy is never contained with task specific instruction stack three cubes. So the motor skills like moving the gripper to the cube and stack it on another one and going push the red button is already included in the base policy. But at test time if you given another out of distribution scenario or out of distribution language instruction, the base policy can always fail. So we think this is from the standard imitation learning reset. So for now if we have a base policy like take π0.5 or Momo Act 2 for example, the base policy can be trained on the behavior cloning target, which means given an expert data set which contains observation, action, and the language for each index and the optimization target is to maximize the likelihood of the base policy to be to predict the next action chunk based on current observation and the absurd language instruction. So the objective is stand static and also is very distribution dependent, which means after fun training your base policy can only capture the special and semantic correlations on the training manifold if you change the observation or language at test time the base policy may confuse. This observation is clear at our daily work because if you take one pre-trained VLA policy you just don’t want to like use it in the
Shuo Liu (05:53)
In distribution tasks, you may change the scenario, you may change the language to the VLA. So for example, sometimes we may face observation out of distribution signal which means we may made some visual or special shift and some semantic shift which means the change of language distribution. So at that time the world may drift away from the training distribution and how we can solve it. We for now we have some two solutions. The first one is we recollect a small dataset out of distribution scenario and run another training cycle. But we need to repeat this circle for every new deployment and make the base policy useful for this specific scenario. But this is kind of misaligned with the target of we train very large VLA policy. That’s because the target is to make the base policy general generalizable but and we can using the pre-train and post-train recipe to make the skills like pick and place or open close this kind of low level skills already inside the VLA policy but we are paying the cost to relear the existing skills at test time s simply because we are not selectively adapt to the base policy at test time. So in this work we want to train we want to try another way that is steering the base policy at inference time, which means we’re taking the we’re taking the ability of generative models like photomatching or diffusion models which are
Shuo Liu (07:46)
Broadly used in current VLAs like the action part action expert part. We can take the advan advantag advancement of their ability to denoise iteratively. So for example the diffusion model is denoising to the target distribution from Gaussian noise and using many diff denoising steps and the flow matching are learning denoising paths to from the Gaussian points to the target distribution directly. So we can use the this the ar feature of denoising and we can add some guided guidance signals into the denoising parts to change the direction of denoising. But the challenge is can we find the right way to steer the flow matching and the diffusion policies to K and well kept their in-distribution performance. The way we can use is using the classifier guidance which means we can inject one gradient signal that push the denoising trajectory towards the out of distribution con condition, which is from the feature of diffusion policy and flow matching policy. We’re we’re not going too deep to of the mice today. But they do have this feature to be guid to be guided in the densing process. So here I want to introduce the real challenge if we can have a guidance signal to push the denoising process of these generative models, then how to make this such a guidance signal what should it to be? So in this work we introduced three requirements of this guidance signal. So firstly since the out of distribution observation and language they are structured so we need to encode the
Shuo Liu (09:59)
Out of distribution special and the semantic constraints into one into one like Into one feature and out one just class label because you cannot you cannot classification every new deployment based on the functioning scenario. And so we need so we are requiring two needs for the guidance signal. So first one, it should be correctly interrupt the geometry and logical structure of the out of distribution condition and secondly it should provide one dense informative gradient to the action trajectory diffusion process. So based on this tunes we designed one method to provide one very good guidance signal to answer these two questions and before we go into
RJ He (10:54)
Yeah. So maybe I can just jump in here. What kind of OOD conditions are you mostly concerned about? You know, in the motivat motivating example, you were talking about so you have examples with two and four blocks and then you know three is therefore out of con out of distribution. But what it you know, does this also would this also work with changing environmental conditions, you know, different lighting? Would it change with yeah, maybe you can share a little bit of which classes of
Shuo Liu (10:58)
Yes.
Shuo Liu (11:22)
Yeah.
RJ He (11:23)
O or D are you all primarily motivated by?
Shuo Liu (11:26)
Okay, . So in this in this slide I’m actually showing two videos of the rollouts of LIBERO-Pro one benchmark we are running for this paper. So for the left part is our internal liberal environment, which is kind of converged by current VLA policies because they can lower it very well to about ninety nine percent of task success rate. And we are split the out-of-distribution scenarios to two types. The first one is observation, which means the visual or special layout may change at the at test time, and the second one is semantic shift, which means you change the language instruction at test time. So for example, for observation shift we can see here on the left is original LibreO. Scenario and on the right is test time scenario. So the base policy are fun-twined on the left scenario and the test and the same layout. But LibrePro are using another layout during test time. So you can say the style and the plate, the their position has been changed. And for language instruction, which means you take the same observation which means the same RGB images with the same layout. But you use another language instruction, for example. If the training target is to push the tomato sauce into bucket, but at inference time you change the target to milk. So we typically split the out-of-distribution into these two types.
Jiafei Duan (13:17)
I think also like maybe just to add on, I feel that the so if you go back to the previous example about the two and four, I mean the that was the example that I told Shuo to do. Cause it’s not really like conventionally out of distribution, right? Because if you’re able to step four, means that you have essentially step three, but then it’s more of like action out of distribution where now you have to terminate earlier. And you realize most of these baseline methods have this latch on behavior. And this is a problem in imitation learning itself. So you sort of latch on onto this behavior the moment it kick starts a certain you know rollouts. So I think this is also very common in the self-driving car, you know, when they are doing imitation learning, when the moment you sort of execute a lane change, you have to commit to that lane change. So it’s it’s hard for you to actually generalize because I have never seen a time where it takes that feedback and makes some form of recovery. So this is like action-level out distribution. So I think. I think maybe like I think what show was trying to mention is that there’s many dimensions of our distribution and I think this word tries to cover every aspect of that. Yeah.
Ishneet (14:23)
Yeah, I think mainly we are interested in OOD cases where the underlying motor skill is still within the capability of the policy. So that means the policies has learned that sort of task. It’s not like we are getting a policy that has never learned folding clothes to suddenly fold clothes at inference time steering, but it’s more that policy has learned to stack cubes. But now if we have appearance OOD or we have viewpoint OOD or we have language OOD or we have position OOD, it should still be able to recover. And as Jiafei and Shuo mentioned, it should be able to change the action distribution and still recover and complete the task successfully.
RJ He (15:00)
Yep, cool. Yeah, I mean I think it probably will be a little bit of a stretch to say it covers all sorts of OOD because I think we you know, the space of possible OOD is very big. But and I don’t you know, given the kind of approach that you all are taking with sort of grounding it and stuff, I think it probably steers towards a certain types of classes. And so that yeah, wanted to sort of let you all enumerate that a bit. So that’s cool.
Shuo Liu (15:25)
Yeah, I think more generally, the OD is an ext extension from the expert data set we use. So maybe the observation can change a bit, or maybe the language can change a bit, or maybe the target action distribution can change, so based on the exp expert data set we use to function the well is, we may change the target or observation we want from the center of this. So it will change a bit but not too far away from what is already seen or trained on. Yeah. So yeah, so maybe I can continue to the related works part. Yeah, so here we list we list some related works by the time we write this paper and split them by some into some categories, like ITPIs and Dana Guide V GPIs and DSRL for one. So we split these related works into different categories depends on whether they need to have another training, maybe another model or the policy itself and whether they and where is the s steering signal come from and whether they are autonomous to steer the base policy, whether they have the well and reasoning ability and whether they are stage aware. So
RJ He (16:56)
Sorry, so all of these related works are alternatives for steering a base policy towards a broader distribution. Is that is that what you’re all comparing to?
Shuo Liu (17:07)
Yes, . Yes, yes. All these related works are focusing on how to how to steer the base policy into the out-of-distribution scenario we just talked about.
Shuo Liu (17:25)
And so compared to other works we have we have three advanta advantages. So firstly Bell is doing the regrounding not the retraining different from Dynaguide like Dynaguide need to train another latent dynamics model and VGPS need to train the Q function used from the offline rl and d DSRL is train the latent noise pol RL policy and for one need to train latent world model. So VLS is different because we make the steering signal from VLM quarry and There is no retrain needed. And also different from with VGPS and 412, because they are scoring the candid candidates and verifies the out output of the VLA policies. We use VLM to write a differentiable PyTorch reward function to the gradient to generate the gradient during the action denoising process. And also different from the DSRL and another other works to rein initial the noise or select the select the output of the VLA policies. Well S inject the gradient to action distribution trajectory denoising at every denoising step. So this can help to narrow the constraint constraints to make them reachable. So this is a comparison between the related works by the time we read this paper and after our paper released there are also some very nice stereo mentored and also introduced the stereo into world action model recently I think.
Shuo Liu (19:31)
So
RJ He (19:31)
Is there a sorry, just very high level, is there a which of these, you know, columns you think are the most significant difference between VLS and the and the signific and the previous work that
Shuo Liu (19:45)
Yeah, I think it’s three parts. So firstly we don’t need to train an any other thing. We don’t need to train another modular and we don’t need to train the RL for base policy or an another modular. And secondly, we get the steering signal from the ability of VLM reasoning. So We are different from others to in introduce VLM reasoning and using VLM to generate the steering signal. It’s not from re-drunk or reinitialize from the gauge noise. And secondly and thirdly we have the VLM in the loop to screen the process of task, whether it needs to be steered and was a progress of task. So we have these three different advantagements, yeah.
RJ He (20:42)
And I’m assuming therefore you’re it you are agnostic to what underlying VLM you’re actually using, right? Like you could swap them out with in future you could swap it out with better VLMs.
Shuo Liu (20:53)
Yes, we also have like ablation of the like the VLM models we use. So we use actually use different wells like for the process of grounding our recently talk is using Gemini and it’s using GPT and for the progress screen we are using Gemini.
RJ He (21:21)
Cool. All right. I’ll let you get to actually talk about.
Jiafei Duan (21:22)
VLM like GPT six, you know, could this task be just soft?
Shuo Liu (21:33)
Yes, by that time we use GPT-4o.
Jiafei Duan (21:38)
So it’s pretty far, like between GPT-4o and GBD6, it’s like not even the same generation. So do you think that will the problem of steering be solved just with better VLMs?
Shuo Liu (21:52)
Yeah. So here I want to introduce the full pipeline of VLS which meaning contain contains three parts. The first one is grounding, second generate and third to steer. So which is meaning to answer the question of how can we get a very good guidance signal for steering to meet these two needs. So given the we can start from the left part. Given the observation OT and language instruction, the pre-trained policy will take the robot state and initialized to generate a very raw prediction which can be used in the scene but it will have a very low success rate. So it need should be refined this raw action. So given the same observation we’re firstly doing the grounding. So where we’re using the large region model is Segment Anything Model and dyno pre-trained feature so we’re firstly using gemini to get the task specific objects and using segment anything to make task to make objects To make mask, segment mask for each object and using the dynamic to get the DINO feature for each object based on the mask and RTV pixels. So from the DINO feature, we class the DINO feature of each object into several key points based on their spatial coordinate and their dynamic. And using this kind of
Shuo Liu (23:40)
Using these key points we are grinding the task specific requirement into some special key points and this set this set of special key points can be used for the second stage which is well and generate the stage available function based on the based on based on the observation and language and also the k points we have for this thing. So the VLM we here we use the GPT4O will generate a series of guidance functions depends on the task progress needed. And the each guidance function will take k point set which is generated by the first stage. So it will be a special k point and also the partially denoised action sequence which is taken from the pre-trained policy denising steps. So the VLM is prompt is prompted to write this function into a PyTorch style so it could be strictly promised that the reward signal from this guidance function can be differentiable to the action sequence input. So this function will generate one guidance signal and we can get one gradient from this guidance functions and we can using the sterable ability of flow matching policies and diffusion policies to inject this gradient signal into their denoising loop. So here we take four steps to inject the signal into the denoising loop based on the raw prediction from pre-chained policy. So firstly we’re using the radius-based function which is
Shuo Liu (25:37)
Which is measures the diverse ability of the pre-trained prediction to give a very large area to explore and we intr introduce gradient from the reward functions using the gradient based refinement and finally we’re using resampling based on the samples itself to have a final refine prediction and then we execute the refine function and using different VLMs to screen the task progress to decide whether this stage need be steered and was with the steering s strings and also how to how to move to another the next stage. So here is a follow up of VOS. First we ground the thing and the observation and instruction into a set of key points and secondly we Query the VLM to generate a set of guidance functions based on the task requirement and second and thirdly we steer the raw prediction to refined prediction. So we can use we can take the first and the second step to meet the requirement one, because we can correctly interrupt the special and semantic requirement to set of k points to structure the out of distribution condition and we can use third part steering to meet the requirement that we need a dense and informative gradient to the partially denoise the action trajectory. And I will introduce in detail how we do the third steering part.
RJ He (27:33)
Cool. Yeah, this is a very interesting technique. Maybe a couple of questions first. Maybe could you share the motivation for on the grounding side of it? Why choose why use Sam Mas and the Dino features? Like what why do you think those features are the important things to be grounding on?
Shuo Liu (27:54)
Yes. So this step is to make the task relevant observation and instruction into structed like model that VLM can use. Because if you just give VLM an RGB image and instruction it may confuse the and output maybe output some text or maybe just re just reuse the image you gave it to. So we are using the this grounding method to make sure the observation and instruction and to ground it into a set of key points, special key points that each object may have, maybe assigned a few key points that can be used to use for description of the VLN.
Ishneet (28:47)
Yeah, I think we use sound for more.
RJ He (28:48)
Yeah, I guess for instance, you know, you’re all one of the ODD pieces are like the visual observations, right? So for instance, the background. That may or may not be really captured in the SAM mass and the dyno features. You know, i are there other groundings that you think may be maybe helpful to be more sort of more complete in the space of ODD conditions that you might be interested in?
Ishneet (29:12)
Hello.
Shuo Liu (29:17)
Yeah, I think DINO feature used here is because it well it is well trained for again the object itself, the parts I mean the different parts. So for different parts of maybe a part the pot handle and the And the part op opening part may be assigned different DINO feature and based on their special re relation and different dynam features, we can get the K-points for different useful parts for each object. And we can also get the we can also get different features from the same segment and the same mask because it’s it’s just take from the RGB image and applied for each object to split from the image and also now display the background.
Ishneet (30:22)
Hello?
RJ He (30:27)
Cool. And then maybe on the reward function, what can you give it at a high level like what is the prompt you all are asking? You know, that you put into GPD for all?
Shuo Liu (30:41)
Yes. So I maybe I can show some check yeah so here is here is the paper and we are showing some details of VLM reward generation so basically some instructions for the VLM to get the robot policy steering context and also we give some in-context examples like from the from the pick and place tasks and we are asking the VLM to generate the guidance functions in a in a way that use PyTorch to make sure that it’s differentiable to the input k points and we have some constraints that need to need to make the functions itself can solve can solve the guidance signal and some shape requirement and finally we ask the VLM to return a set of rule functions.
Shuo Liu (32:02)
And also for VLM sta for VLM stages reaching we are also using another very simple instruction to screen the task progress of the policy. Yeah.
RJ He (32:02)
Okay, interesting. Yeah.
Shuo Liu (32:17)
Yeah, so here is a prompt we use for the veil arms.
RJ He (32:30)
Yeah, it doesn’t seem like it has a lot of focus on the actual reward function. It seems like a there’s a lot of Dhanas that’s involved in gener in writing out that prom. But it’s interesting.
Shuo Liu (32:41)
Yes, because from our observation so for the very first version that VLM may generate like the real functions not in PyTorch still which means it cannot it cannot be differentiable to the action proposal and sometimes it will import another libraries from the repo but we cannot apply this because we are doing the runtime still re so i it should be generated and directly using in the pre-twin policy denoising part so we twin it literally iterate and have this final version yeah it’s also attached to the archive version at the at the final part yeah
RJ He (33:36)
Okay, great, thanks. Yeah. And you said you wanted to dive deeper into how the reward function gets injected to the denoising process, right? Yeah.
Shuo Liu (33:44)
Yes, I will introduce the play for this stair part. Maybe we can also introduce the first grounding and the reward generation part of it. So from the from the observation and the language introduction we first quarry, Gemini and second anything, then dyno for and finally the k points using the GPT to generate the real functions that are differentiable and stage a wear for the task. So here we using the reward as code not as a score so we can get the dense gradient from the these reward functions. And for stage three we want to deeply introduce processed from the initial partially denoised action proposal we use radius basis function kernel to make sure the divers of the action proposal at very early stage of denoising are very high. So it will using a pairwise representative force inside of the action proposal batch to keep each proposal keep away from each other and this can really help to make sure the denoising process not collapse into the local optimum. And so we can have a we can have a good diverse batch to start the gradient based guidance. So at the second stage we are introducing the gradient signal from the real functions we just introduced. So here we add the real gradient to the to the noise to the denoising process at every time step. So here the very dense and the differentiable signal are injected to the
Shuo Liu (35:44)
Denoising process and finally we’re using Feynman–Kac resampling process and to this is this is not gradient based this is resample based. So for a batch of samples it will reweight the samples based on the same reward signal from the functions and it’s kinda kind of like JNAC algorithm that copies the good examples and lower the weight of bad examples. And finally we have the refined action. So this is the process of stage three the steering from the initial raw pro prediction from the pre-trained value model to the final refined action that we can use in the scene. Yeah.
RJ He (36:38)
Cool. So just to make sure I understand the RBF kernel is there to try and increase the number of modes. So you get that diversity and then FK resampling is more that but once you start recognizing what are good modes and what are bad modes, that you wait further towards the good modes. Do they then kind of compete against each other a little bit because they’re
Shuo Liu (36:59)
Yes, yeah.
RJ He (37:04)
FT resembling may then lead again to a concentration towards fewer modes, you know, s hopefully the good modes, but it will lead to a smaller number of modes then.
Shuo Liu (37:15)
Yes, I think for n for here the refinement and resampling are using together because refinement can provide like kind of local steering. So it we are based on the gradient from the real signal for each for each sample. And the resample for here is kind of like a global mentor that can compare different samples into the same batch and select the good ones and lower the rate of bad ones. But it’s not just great weight into inside the algorithm itself it will help to improve the weight of sample the sample those action predictions that have good function reward signal score at this step but at the next step it will be another story because we are choosing from a very diverse sample port from the same batch because we are using RBF here to make sure we have a we have a very good port to select from and we doing the local requirement of each of them first and compare each other f into the batch to Select the good ones for next step denoising.
Jiafei Duan (38:37)
Yeah, I wanted to ask regarding the FK resampling versus to like just regularly resample you so you mentioned about the I guess like sort of like evolutional algorithm kind of approach. So like what’s the motivation behind that? Like why uses this approach specifically?
Shuo Liu (38:49)
Yes.
Shuo Liu (38:57)
I think this kind of because from the I chose this because we for action denoising we can have a batch of samples to compare and how can we how can we find the good examples from this batch. So one very classic algorithm is to using the rev revolution algorithm to compare to compare each sample’s ability of the same standard. So here we’re using the reward score here and to make sure the good ones can be sampled by higher chance for future and lowers lower the possibility of samples by ones. This is from the ability of denoising a batch of samples of the BLA models.
Jiafei Duan (39:58)
Got it. And my assumption is that this approach is used for image generation, like before you know, you apply it for action generation. Is that like so you do see that this actually helps quite a bit through the ablation with that case and
Shuo Liu (40:13)
Yes, . So Yes so yeah by the time we are exploring different mentors also inspired for from some image editing papers that using FK resampling for the image generation part image generation direction and we found that it can really help the to improve the performance of denoising samples in very quick and promising way. Yeah.
RJ He (40:53)
Cool. Yeah, why don’t maybe wanna make sure you have time to cover the results. Maybe you want to jump move on to that.
Shuo Liu (40:59)
Okay, sure. Okay, yeah. So for the experiment part, we mainly want to answer the requ research questions that first one can VLS adapt VLA models to joint observation and language shifts, and secondly does VLS provide a stronger adaptation than existing inference time steering answers? And also We want to do an ability study of what’s the contribution for each component and also whether ice can adapt the policies in the real-world deployment. So firstly we want to introduce the experiments that we did on the LIBERO-Pro and the LIBERO-Reasoning benchmark. So LIBERO-Pro is a benchmark that has a perturbation based on the original LibreO benchmark that can change the task layout and also task specific language instruction across four different seals. And LIBERO-Reasoning LIBERO-Reasoning is another benchmark that can change the visual scene change visual scene layout and also the camera viewpoints. So here is two tables relevant to the LIBERO-Pro results and the liberal LIBERO-Reasoning LIBERO-Reasoning benchmark. So we ha here we can find that compared to the base policies we use open VLA π0.5 and different environment of π0 and different
Shuo Liu (42:41)
Different steering method like do what you see and the π0 with vGPis here. Compared all of them VLS can get the highest score across two different liberal perturbation benchmarks. So here we want to introduce why VLA’s fail here and why VLS can success. That’s because given the very different observations and language shift the post trained policies maybe get the special reasoning and the training context together so it can doing the lazy motion like giving the kind of similar observation under different language instruction, it can it will directly mapping the observation to the action. Never consider what language used here. And VLS can do in the regrounding based on the Task inference time relevant observation and instruction pair to the 3D keypoint side and using the differentiable reward function to the action chunk. So it could be very sensitive to the inference time task configuration and improve the ability of base policies, whether it’s π0.5, π0, any flow matching based VLA expert or diffusion policy can work here. So basically the VLAs can actually learn how to use how to generate the motor skills relevant to the post training part relevant to the fine training task configuration but can very easily lose the semantics special and the special general generalized generalization and VLS can help to restore it. This is a first
Shuo Liu (44:34)
Research questions that we want to answer so VLIS can do adapt VLA models to joint observation and the language shift.
RJ He (44:46)
Would you say that a big motivation of this paper is kind of motivated by the Liberal Po setup where those are you know it’s those kind of task and position perturbations that you know LIBERO-Pro was designed for is what motivated you to sort of think about those types of O D c scenarios to solve.
Shuo Liu (45:12)
Yes, . So this is also from the observation that liberal is going to kind of converge by current value because recently many work can reach the success rate higher than ninety eight. So we found that One problem is that the BLA’s are functioned on the same observation and the action pair and for the inference time it will it will never change. So the BLA policies can very easily get one lazy mapping from the observation to the action. If they are not it will not consider what’s the language change or what’s the observation change. Yeah.
Ishneet (45:58)
Yeah, I think LIBERO-Pro was a good motivation for us because it’s a good data set where you’re able to actually do specific type of OD and test the performance of your setup. But I think the broad motivation was still that you know you have very strong pre-trained robot policies that have the motor skill and that have the capabilities in the policy, but they’re just not being utilized in the action output because if you have any observation or language context shifts, your action generation is just too it’s just tied too closely to your training time. Correlation. So I think LIBERO-Pro manifested in a way that it was a good benchmark was to actually quantify what the different OODs can materialize in and actually how to test the whole steering pipeline. But yeah, I think the main motivation still remained that the pre-trend policies are just quite brittle to OOD in general.
RJ He (46:48)
Yeah, no, that’s cool. And I think this technique seems like it’s quite general to expand beyond just the specific scenarios.
Jiafei Duan (46:57)
I have a question for show. Do you think that ‘cause the thing that you’re sort of steering I think mostly come from the action expert side of the VLAs, right? Like so do you think this will actually work for WAM, the world action model?
Shuo Liu (47:13)
Yes, I think recently another work is called sterile WAM is just released and I actually have not read it yet but I think for WEM if we can because it’s actually jointly denoising the feature videos and also the actions together. So we may apply this approach to the WAM models, yeah.
Jiafei Duan (47:43)
Got it. So it’s it’s it’s kind of applicable to also WM because of the action expert component of WM that’s pretty similar to VLA’s minus away the VLM side, which like you could in theory apply this to generic, both the world model side and also the VL VLA side. Yeah.
Ishneet (48:04)
Yeah, usually if WAMs have like a differentiable generative action expert, it would be quite interesting. I think there were some papers recently on WAMs where there was like a flow matching action head that decoded a continuous action chunk. There’s also one that was a lightweight action expert on it. So yeah, I think if you add steering on top of WAM, you probably have like more like more strong world representation. And then it’s also able to like iterate on the failures of its actions as well. So steering could be an interesting thing on the W A
RJ He (48:36)
But will say that this is this method is more limited to ones that are using an action hit like that? You know, what about say IDM type of approaches? Is this applicable or it’s probably not?
Shuo Liu (48:57)
Sorry, can you repeat the models like different models you just mentioned?
RJ He (49:03)
Sorry, so it sounds like this requ does require a model that has a action head that is kind of flow matching based. You know, what about but not all wo especially world action models are using that approach, right? Like there’s IDM and others. So do you think that would be applicable too or this is more limited to ones with flow action flow matching?
Ishneet (49:24)
Do you think that is available to or is it not a good job? I think steering can still be part of an IDM. As far as I understand an IDM correctly, it is differentiable end to end. So usually for us we do steering at the action generation level for VLA or maybe for world action model. We can still do the similar process for IDM, but because it’s just like a deterministic state to action decoder, you have like a video and you want to get actions from it, you can easily you can usually steer its input. So you can steer the diffusion process in the video. In the input of the IDM or maybe you steer an upstream planner rather than the IDM itself. Yeah. But I think it’s still possible with the IDM as well.
RJ He (50:10)
Cool. This is that’s that’s very exciting.
Shuo Liu (50:14)
Yeah, maybe I can introduce going to research question two, does well as provide a stronger adaptation than existing inference time theory. So here we compare to different theorem answers that are based on the same diffusion policy checkpoint on the CALVIN environment. So we compared with Spencer’s ITPS, Dana Guide, VLA pilot, and base policy diffusion and base diffusion policy itself. So we get one seven point four times improvement based on the base diffusion policy of the mobile objects and nine point six time of the base policy and we also gets a higher s score rather than other stereo mancers. So we think that’s because ITPIs need one hand picked the classifiers that need human in the loop and could very easily fail on the mobile objects and that’s because in the simulation running we are setting the K points at the first frame of ITPIs and not using the dynamically update and for Dana Guide since they are using one latent word model that using the Dino features distance to measure the to measure the reward score and inject into the action denoising process. But DINO feature is too cross for the fine special constraints like for CALVIN you may need to open the switch, open the drawers and pick up the
Shuo Liu (52:04)
Pick up the three cubes on the table. Only using the dano feature could make mistake because the maybe the cubes feature is too cross to describe in the whole image feature and the VLA pilot is using is using the sample quality of the i it’s depend it’s depend on the sample quality of base policy. So it’s it’s kind of irrelevant of VLS because just accept the gradient-based denoising. So they are doing the sampling and compare the different performance of each sample based on the same reward function. But they are not using the gradient signal and they are not doing the like doing the RBI process to make the samples very diverse. So if the base policy can generate a very narrow bench narrow batch of samples it will choose the best one with the highest score using the real function but will never like push it to another direction. So from the comparison of different steering methods we can find that VLS can compare cu combine the dense VLM written gradient signal and also the resampling process together to make the best for this CALVIN environment benchmark.
Shuo Liu (53:47)
So and also here maybe HN can introduce the Apple H part. Yeah. Yeah.
Jiafei Duan (53:52)
Wait, sure, I have a question about the previous one. So I mean like I understand a stronger adaptation, you know, compared to c current inference time steering approach. But is the latency better than the current approach? So like because you are steering and you are always doing this kind of like some form API cost to the VLMs, you know, then they creates latency right in the inference time. So how would that im latency compare to some of this other approach? Do you guys try that?
Shuo Liu (54:22)
We didn’t compare the latency across different mentors but we do have an abilation of the WS system itself. But from my experiments that dynaguide and well that could be dynamaguide could be much faster because Like for one specific scenario it will train one latent world model first and it’s very it’s very late white model and can be queried very fast. So at each so at each de denoising step it will compare the current observations DINO feature and the goal and the sorry the goal conditions DINO feature together and compute the feature distance to there’s action so it can be queried very fast and at pix is meaning human in the loop and I think BLA pilot is the closest to our messenger that’s because we are both using the remote functions reward functions generated by BLM and they are just not using the gradient based steering and so the latents the latents introduced the delay introduced can be the same level of VLA palette and ours, but both are slower than a guide. And also we have a ability of VLS system itself, the latency introduced.
Jiafei Duan (55:58)
Got it. Thanks.
RJ He (56:01)
Cool, yeah. I wanna be in I wanna be conscious of time, so maybe you can quickly touch on the ri final two research questions. Yeah.
Shuo Liu (56:06)
Okay, yeah. Okay, yeah, yeah. So maybe Ishneet can help to in introduce ablation part.
Ishneet (56:16)
Yeah, so we did ablation on each component in a system. So firstly you have like the you have the SAM and the dyno as well, which builds the geometric interface. Then you also have the VLM reward synthesis, then we did more ablation on the gradient guidance, the RBF repulsion, which maintains the diversity and also does the exploration for the steering. And then we also did ablations on Feynman category sampling. We also did ablations on adaptive guidance strength as well. And so based on all these ablations, what we realized was that the gradients guided steering is the most important. And actually when you remove that on the Calvin benchmark, then it’s not able to complete the task. And so what we realized is the programmatical rewards that the VLM generates does the heavy lifting in a system. And then Feynman–Kac resampling or the RBF is more of helping it explore and choosing the best one out of the different steering solutions. And so that helps to stabilize the steering process. And they are if you remove them or you keep them, it doesn’t make a big difference because the main thing is still the gradient guidance steering. And yeah, I think what we realized was that the gradient guidance steering is the main component that matters. And the other two are sort of inconsequential to the whole system. But if you add them to the whole system then they work. But if you r remove one of them, it doesn’t make that much of a difference. Yeah.
RJ He (57:44)
Sorry, just clarifying, what does it mean when you say you remove the gradients guided some part of it? Then what is the alternative when you remove that?
Ishneet (57:53)
So by gradient guidance steering I mean that we directly are modifying the denoising so that the trajectories move towards satisfying the reward. This is one of the main components in our paper that actually turns the reward that the VLM generates into an actual continuous signal. And we realize that if we remove this out of our pipeline and keep the others and do try to do steering, the performance drops a lot. So on Calvin we try to test this, it drops from like ninety percent to eighteen percent. So we realize that the gradient guidance part in a pipeline is actually what bends the generative trajectory towards the desired behavior.
RJ He (58:32)
Yeah, I mean that makes sense and in many ways I thought that is the only way that you could be influencing the action trajectories, right? I mean, if you don’t have that gradient’s guidance, you don’t have a your out your at on your enhancements don’t really have any impact.
Ishneet (58:47)
Yes, that is true.
Shuo Liu (58:51)
Yes, I think it’s also the key points that way compared to the VLA palette here because it actually an environment in environment of VLS just don’t have the gradient guidance part, so the performance showing in CALVIN we can overperform it because we are using the action specific gradient to inject to the denizing loop. Yeah. So here is which yes so it we also did real world evaluations and designs task that can measure the sterility of different
RJ He (59:21)
Okay, cool. Thanks. Yeah, maybe your final research question?
Shuo Liu (59:38)
Models whether it’s π0.5 itself or π0.5 plus VLS. So for the task for the in distribution performance, VLS can help the π0.5 checkpoint and just fun tune on several demos of the task specific data and VLS sorry VLS can help the base policy π0.5 performs better than the policy itself and for out of distribution performance. VLS can hold the success rate at the same level of in-distribution, but π0.5 can drop a bit because it’s a new observation and a new language combination at the inference time. So VLS is actually helps the base policy can be used in the out of distribution scenario.
Shuo Liu (01:00:38)
Yeah, so here we want to go to the conclusions that with the contribution of VLS and w with the future direction. So VLS is introducing the combination ability of VLA policies. So the pre-chrint policies can combine the motor skills they learned from the fan training process together at inference time by VLS. And we found that VLMs are very good at writing the remote functions and introduce the guided guidance signal into the denoising trajectory d denoising loop and we found that steering is actually better than selection paired with mentors like VGPS and VLM in the loop. And we also find that we also introduced that VLS is one training free and policy agnostic method that can be deployed on any flow matching or diffusion based policies, whether they are diffusion policy or VLA policies with an flow matching expert, action expert. Yeah. That’s the in the intr our introduce of our work VLS. Thank you, yeah.
RJ He (01:01:57)
Great. Well thanks guys. Yeah, this is a super cool technique. I think has a lot of potential. Especially bringing in the VLMs to help with writing the reward function to do the steering. So congrats on the paper. Also it sounds like you guys just got accepted to Coral, so congrats for that as well. Looking forward to seeing it at that conference. Maybe just to wrap things up, as we always ask, any papers that
Shuo Liu (01:02:16)
Yeah.
RJ He (01:02:25)
You all have come across recently, you know, whether related to steering or otherwise, that you all want to do a shout out on?
Shuo Liu (01:02:32)
Okay maybe I can stop sharing first.
Shuo Liu (01:02:46)
Sorry, can you like repeat the question?
RJ He (01:02:49)
Yeah, no i mean, you know, as we ask with every episode, you know, any interesting papers that you all come across recently that you want to do a shout out on.
Shuo Liu (01:03:00)
Yeah. Recently I think a very new trend for robot policies is called in context learning that is from the generalist one point five and skilled AI. Also previously we have the immediate like text time training. So I think it’s like new trend of how can we adapt the pre-trained policies at inference time to like fit different task specific requirements. Like if you just showing one demo of the task specific and the robot policy can just learn it. So I think it’s kind of a emergence from the very large scale of pre-trained data they have and also and also I think the tax test time training paper also introduced one very interesting mansor that introduced trainable model sorry the net networks in inside the pre-trained policy itself. So I think it’s also kind of aligned with the motivation we want to do the s this steering because we are both want to make sure the pre-trained value policies can be general generalized to the new task specific requirements.
RJ He (01:04:34)
Yeah, cool. Yeah, indeed. I think this space of this is a quite an interesting space together and many of these different techniques of adapting to it. Yeah, Ishneet, anything for you?
Ishneet (01:04:34)
Yeah, for definitely.
Ishneet (01:04:45)
Yeah, I think a recent paper that was very intrigued by was called Piggy and that paper was very interesting because what it does is it is it is doing like a orchestration layer and it’s orchestrating different VLAs as well as the model’s natural capabilities to do like just control the end effective pose and get it to do the task and we see this a lot with GPT-6 being released and you see it’s able to do tasks in Libro and almost solve Libro just natively. And I think that’s very interesting and how these LMs, the way they’re trained in the future as well, if they’re trained with more and more web data, they just generalize to videos and you realize all you need is really just a very good action head. And then you know it might you may not need world action models or something and maybe just scaling L LMs is the way to go, maybe.
RJ He (01:05:35)
Yeah, cool. Yeah, indeed, as Jafi was mentioning earlier, sort of y b methods are UI you guys that are relying on GP you know, some of these LIMs, you get to benefit for free as these LIMs get better as well. So maybe that just on its own solves the problem. Awesome. Thanks guys. Thanks for coming on our pod. And looking forward to yeah, more great works from you guys. Thanks again.
Ishneet (01:06:02)
Thank you so much.
Jiafei Duan (01:06:03)
Thank you. Bye.
Shuo Liu (01:06:03)
Okay. Secure is yeah.
RJ He (01:06:04)
All right, bye.
