Speed up multimodal RL coaching with SkyRL on Amazon SageMaker HyperPod

Reinforcement studying (RL) post-training is changing into an ordinary step in constructing succesful language mannequin brokers. Fashions study to cause and act throughout sequences of steps by producing trajectories, receiving rewards, and updating their coverage primarily based on outcomes. Working this at scale, throughout a number of nodes with a whole lot of GPU-hours of rollouts per coaching run, requires persistent cluster infrastructure. That infrastructure must maintain lengthy jobs, get better from {hardware} failures with out shedding progress, and supply visibility into coaching dynamics as they unfold.

Amazon SageMaker HyperPod offers this infrastructure for large-scale machine studying (ML) workloads on Amazon Elastic Kubernetes Service (Amazon EKS). By means of its cluster resiliency options, it repeatedly screens node well being and mechanically replaces defective nodes, so a {hardware} failure doesn’t take the cluster down with it. Paired with checkpointing, a coaching job can decide up from its final saved step as an alternative of restarting from scratch. This issues for lengthy multi-node RL runs, the place a single {hardware} failure would in any other case value hours of rollout progress. Mixed with the Ray capabilities on HyperPod, you may create Ray clusters from SageMaker Studio, submit jobs remotely utilizing safe connections, and monitor coaching by way of pre-built Amazon Managed Grafana dashboards that the HyperPod Observability EKS add-on provisions for you.

On this publish, we present easy methods to use these capabilities to run SkyRL, an open-source RL framework, to coach a Qwen3-VL-8B vision-language mannequin to navigate visible mazes utilizing Group Relative Coverage Optimization (GRPO) on SageMaker HyperPod. Ranging from the VisGym SFT checkpoint, a supervised fine-tuning (SFT) start line, GRPO post-training on HyperPod improves the maze clear up price from 43.75% to greater than 95% on a hard and fast 64-maze analysis set.

Stipulations

To comply with this walkthrough, you want:

• A SageMaker HyperPod cluster with Amazon EKS orchestration that has at the very least 3 ml.g7e.12xlarge cases and one ml.r5d.16xlarge occasion.
• The next Kubernetes operators put in in your cluster: KubeRay operator, HyperPod Observability EKS add-on, and HyperPod Ray Endpoint Operator (for distant job submission). See the Ray on HyperPod getting began information.
• The Amazon FSx for Lustre CSI driver put in on the cluster. You additionally want an Amazon FSx for Lustre filesystem, a PersistentVolume backed by that filesystem, and a PersistentVolumeClaim (ReadWriteMany) that the pods can mount. The coaching job makes use of this at /shared for checkpoint storage, Low-Rank Adaptation (LoRA) adapter synchronization, and analysis output.
• A SageMaker Studio area with permissions to hook up with your HyperPod cluster. See establishing SageMaker Studio for Ray.
• The toolkit-for-ray-on-sagemaker-ai Python bundle put in.

Background

This part critiques the reinforcement studying ideas behind the coaching and the cluster topology the walkthrough makes use of.

Multi-turn RL and GRPO

Normal single-turn RL assigns a reward to a single mannequin output. Multi-turn RL as an alternative trains an agent over a complete sequence of steps, the place it observes a state, acts, will get suggestions, and strikes on to the subsequent state. The coverage learns from the reward accrued over all the episode reasonably than from anybody step.

Think about the instance downside of navigating a 2D maze. One episode is a single run at a maze, and every flip is one transfer: the mannequin seems on the present image of the maze, chooses a course or decides to cease, and the surroundings sends again the up to date view. Rewards are sparse, so the mannequin earns 1.0 solely when it truly reaches the aim inside the transfer restrict and nothing in any other case. There isn’t any move-by-move reply key to coach towards, since whether or not a transfer was good depends upon the strikes round it.

That is the place SkyRL’s Group Relative Coverage Optimization (GRPO) is available in. For every beginning place, the agent runs the maze a number of occasions below the present coverage, and GRPO grades these runs towards each other, reinforcing those that beat the group’s common and pushing down those that path it. That within-group comparability is the entire coaching sign, which lets GRPO work with out a separate critic or worth mannequin.

Coaching topology

The answer mentioned right here runs SkyRL on a HyperPod Ray cluster with three GPU employee nodes and a CPU head node. SkyRL colocates inference and coaching on the identical GPUs: vLLM engines generate rollouts (full maze episodes) whereas a coverage mannequin sharded with Absolutely Sharded Information Parallel (FSDP) handles gradient updates. After every optimizer step, up to date LoRA adapter weights sync from the coaching ranks to the inference engines by way of Amazon FSx for Lustre shared storage.

These are the occasion sorts we used. Different GPU cases and cluster sizes work as properly, offered the employees have sufficient GPU reminiscence for the mannequin.

• Employees: 3x ml.g7e.12xlarge (2x NVIDIA RTX PRO 6000 Blackwell GPUs every, 6 GPUs complete).
• Head: ml.r5d.16xlarge (512 GB RAM, manages Ray GCS, dashboard, and LoRA adapter consolidation).
• Coverage mannequin: Qwen3-VL-8B with LoRA (rank 32), sharded throughout the 6 GPUs utilizing PyTorch FSDP.
• Rollout engines: 6 colocated vLLM cases, one per GPU.
• Shared storage: Amazon FSx for Lustre at /shared, used for LoRA sync and analysis output.

HyperPod offers the cluster infrastructure: the Ray cluster is created from SageMaker Studio, job submission makes use of the sagemaker_ray:// protocol, and coaching metrics movement mechanically into pre-built Amazon Managed Grafana dashboards by way of the HyperPod Observability add-on.

Answer overview

The next steps stroll by way of getting ready the coaching surroundings, launching the cluster, operating the job, monitoring progress, and internet hosting the skilled mannequin.

Step 1: Put together the container picture

To get began rapidly, use the next Dockerfile to construct a container picture with SkyRL, VisGym, and their dependencies pre-installed. That is the picture you’ll specify when launching your Ray cluster on HyperPod within the subsequent step. It builds on the official NovaSky-AI SkyRL base and pins each SkyRL and VisGym to particular commit SHAs so the construct is reproducible:

FROM novaskyai/skyrl-train-ray-2.57.0-py3.12-cu13.0

ENV HF_HUB_ENABLE_HF_TRANSFER=1 
    QWEN_VL_MODEL=Qwen/Qwen3-VL-8B-Instruct 
    UV_PROJECT_ENVIRONMENT=/residence/ray/anaconda3

# SkyRL's full FSDP stack into the system python Ray makes use of (uv .venv is invisible to `ray begin`).
# Pinned to a commit SHA so the construct is reproducible.
ARG SKYRL_REF=4298730b55bb01fe1b711662df53dca42a3b7615
RUN git clone https://github.com/NovaSky-AI/SkyRL.git /residence/ray/skyrl 
    && cd /residence/ray/skyrl && git checkout ${SKYRL_REF} 
    && cd /residence/ray/skyrl/skyrl-train 
    && uv sync --active --extra fsdp 
    && /residence/ray/anaconda3/bin/python -c 
       "import ray, torch, vllm, transformers, flash_attn; 
        from vllm_router.launch_router import launch_router; 
        print('skyrl stack OK', torch.__version__, vllm.__version__)"

# uv sync prunes Ray's dashboard extras; restore ray[default] so the total dashboard begins.
RUN uv pip set up --python /residence/ray/anaconda3/bin/python "ray[default]==2.57.0" 
    && /residence/ray/anaconda3/bin/python -c 
       "from ray.dashboard.optional_deps import aiohttp"

# VisGym maze surroundings and dataset generator.
ARG VISGYM_REF=184fbd5e5dc81e32c8b944d9e40ac54dad62e3f2
RUN git clone https://github.com/anyscale/VisGym.git /residence/ray/visgym 
    && cd /residence/ray/visgym && git checkout ${VISGYM_REF} 
    && uv pip set up --python /residence/ray/anaconda3/bin/python -e . "pygame==2.6.1"

ENV FI_PROVIDER=efa FI_EFA_USE_DEVICE_RDMA=1 NCCL_PROTO=easy NCCL_DEBUG=INFO
WORKDIR /workspace
CMD ["/bin/bash"]

Construct the picture and push it to an Amazon Elastic Container Registry (Amazon ECR) repository in your account. Notice the total picture URI, as you’ll use it when creating the Ray cluster within the subsequent step:

ACCOUNT=$(aws sts get-caller-identity --query Account --output textual content)
REGION=us-west-2
IMAGE_URI="${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/ray/skyrl-visgym:newest"

aws ecr get-login-password --region ${REGION} | 
  docker login --username AWS --password-stdin ${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com

docker construct -t skyrl-visgym .
docker tag skyrl-visgym ${IMAGE_URI}
docker push ${IMAGE_URI}

echo "Picture URI: ${IMAGE_URI}"

Step 2: Launch the Ray cluster from SageMaker Studio

Navigate to SageMaker Studio, select HyperPod, choose your cluster, then go to the Duties tab. From the duty sort listing, select RayCluster, then select Create Ray Cluster.

Within the creation kind, give the cluster the title skyrl-visgym, set the top occasion sort to ml.r5d.16xlarge, and add three employees utilizing ml.g7e.12xlarge. Set the container picture to the IMAGE_URI you pushed in Step 1.

The occasion sorts listed below are what we used for this walkthrough. Different occasion sorts will work, however hold one constraint in thoughts for the top node: it wants massive reminiscence. The top consolidates LoRA adapter shards from the GPU employees at every checkpoint save, which briefly hundreds the total adapter weight set into CPU reminiscence. We used ml.r5d.16xlarge for its massive reminiscence capability (512 GB RAM) to accommodate this.

Mounting Amazon FSx for Lustre

To connect your Amazon FSx filesystem, select the YAML button within the top-right nook of the creation kind to modify to the uncooked manifest editor, then add the amount and mount to each the top and employee pod specs. The related part for every pod seems like this:

containers:
  - title: ray-head  # or ray-worker
    volumeMounts:
      - title: shared
        mountPath: /shared
volumes:
  - title: shared
    persistentVolumeClaim:
      claimName: <your-fsx-pvc-name>

Substitute <your-fsx-pvc-name> with the title of the PersistentVolumeClaim backed by your Amazon FSx filesystem. With this in place, /shared is on the market on each node within the cluster and the coaching job can learn and write checkpoints, LoRA weights, and analysis output from the pods.

Activate Distant endpoints so you may submit jobs and open dashboards with out a native kubectl port-forward. The cluster generates IAM-authenticated URLs for each.

Determine 2: The Create Ray Cluster kind in SageMaker Studio with distant endpoints turned on for job submission and dashboard entry

As soon as the cluster reaches Working standing, the Actions menu within the Duties tab affords Open Ray Dashboard, Open Grafana, and cluster administration choices.

Determine 3: The skyrl-visgym Ray cluster at Working standing within the SageMaker Studio Duties tab, with the Actions menu open

Step 3: Put together the coaching script

Save the next as train_job.sh in your working listing. The script downloads the SFT checkpoint and generates datasets on first run (each go to Amazon FSx, so that they persist throughout runs), then launches the GRPO coaching job.

The SFT checkpoint provides GRPO a robust start line: Qwen3-VL-8B pre-trained on VisGym demonstrations already is aware of easy methods to parse a maze picture and emit structured transfer actions, so GRPO solely must refine which sequences attain the aim.

With coach.placement.colocate_all=true, the vLLM rollout engines and FSDP coverage employees share the identical GPUs. Throughout rollout, GPUs run inference in parallel. Through the coverage replace step, they run FSDP coaching collectively. The lora_sync_path factors to Amazon FSx so up to date adapter weights are instantly seen to the inference engines on the nodes after every optimizer step. With out colocation, you want separate GPU swimming pools for coaching and inference, they usually sit idle ready for one another between phases, a ping-pong sample that wastes compute. Colocation avoids that idle time by having coaching and inference take activates the identical {hardware}. That’s the reason gpu_memory_utilization=0.45 is ready conservatively: every GPU wants headroom for each the FSDP shard and the vLLM KV cache on the identical time.

A number of different parameters are value noting. n_samples_per_prompt=8 controls what number of rollout trajectories GRPO generates per maze immediate to compute the group benefit. max_turns=15 caps every episode at 15 strikes. hf_save_interval=20 consolidates LoRA adapter shards onto the top node and saves a Hugging Face-compatible checkpoint each 20 steps. eval_interval=10 runs the held-out 64-maze analysis each 10 steps so you may observe clear up price as coaching progresses.

The job additionally writes full coaching checkpoints so it could get better from an interruption. Setting ckpt_interval=20 saves the entire coaching state to ckpt_path each 20 steps. That state consists of the mannequin weights, optimizer state, studying price schedule, and dataloader place. resume_mode=newest tells SkyRL to select up from the newest checkpoint below that path when the job begins. Level ckpt_path at sturdy shared storage that isn’t tied to a single node, comparable to an Amazon Easy Storage Service (Amazon S3) prefix or your Amazon FSx mount. Hold the trail secure throughout runs so a restarted job can discover its checkpoint. That is what pairs with HyperPod cluster resiliency. When a node fails, HyperPod detects and replaces it mechanically, and if you resubmit the job it continues from the final saved step as an alternative of beginning over. The complete checkpoints written right here seize coaching state for resumption, whereas the hf_save_interval exports seize inference-ready LoRA adapters, so the 2 run alongside one another for various functions.

#!/usr/bin/env bash
# GRPO fine-tune from the VisGym SFT checkpoint on maze_2d/simple.
set -euxo pipefail
cd /residence/ray/skyrl
export HF_HUB_ENABLE_HF_TRANSFER=0
export SKYRL_RAY_PG_TIMEOUT_IN_S=600
RUN_ID=$(date +%Ypercentmpercentd-%HpercentMpercentS)
EXPORT_PATH="/shared/runs/${RUN_ID}-train"

ENV_ID=maze_2d/simple
TRAIN_DIR=/tmp/visgym_train_256
EVAL_DIR=/tmp/visgym_eval_seeded
SFT_CKPT=/shared/fashions/visgym_sft_mixed_qwen3vl

if [ ! -f "$SFT_CKPT/config.json" ]; then
  echo "=== Downloading SFT checkpoint ==="
  mkdir -p "$SFT_CKPT"
  python -c "
from huggingface_hub import snapshot_download
snapshot_download(repo_id='VisGym/visgym_model', allow_patterns="mixed_qwen3vl/*",
    local_dir="$SFT_CKPT", local_dir_use_symlinks=False)
import shutil, os; src="https://aws.amazon.com/blogs/machine-learning/accelerate-multimodal-rl-training-with-skyrl-on-amazon-sagemaker-hyperpod/$SFT_CKPT/mixed_qwen3vl"
[shutil.move(os.path.join(src,f),'$SFT_CKPT') for f in os.listdir(src)]; os.rmdir(src)
"
fi

python examples/prepare/visgym/dataset.py --env_id "$ENV_ID" --num_rows 256 --output_dir "$TRAIN_DIR"
python examples/prepare/visgym/dataset.py --env_id "$ENV_ID" --num_rows 64  --seed --output_dir "$EVAL_DIR"

python examples/prepare/visgym/entrypoint.py 
  --env_variant sft 
  information.train_data="['$TRAIN_DIR/train.parquet']" 
  information.val_data="['$EVAL_DIR/train.parquet']" 
  coach.algorithm.advantage_estimator="grpo" 
  coach.coverage.mannequin.path="$SFT_CKPT" 
  coach.coverage.mannequin.lora.rank=32 
  coach.coverage.mannequin.lora.alpha=32 
  coach.coverage.mannequin.lora.lora_sync_path="/shared/lora" 
  coach.placement.colocate_all=true 
  coach.technique=fsdp 
  coach.placement.policy_num_nodes=3 
  coach.placement.policy_num_gpus_per_node=2 
  coach.placement.ref_num_nodes=3 
  coach.placement.ref_num_gpus_per_node=2 
  coach.ref.fsdp_config.cpu_offload=false 
  generator.inference_engine.num_engines=6 
  generator.inference_engine.tensor_parallel_size=1 
  generator.inference_engine.gpu_memory_utilization=0.45 
  generator.inference_engine.engine_init_kwargs.max_model_len=16000 
  surroundings.env_class=visgym 
  coach.epochs=20 
  coach.train_batch_size=24 
  coach.policy_mini_batch_size=12 
  coach.micro_forward_batch_size_per_gpu=1 
  coach.micro_train_batch_size_per_gpu=1 
  coach.update_epochs_per_batch=1 
  coach.max_prompt_length=2048 
  generator.sampling_params.max_generate_length=1024 
  generator.sampling_params.temperature=0.7 
  generator.max_turns=15 
  generator.max_input_length=8192 
  generator.n_samples_per_prompt=8 
  generator.vision_language_generator=true 
  generator.batched=false 
  coach.remove_microbatch_padding=false 
  coach.algorithm.use_kl_loss=false 
  coach.coverage.optimizer_config.lr=3.0e-6 
  coach.eval_interval=10 
  coach.eval_before_train=true 
  coach.ckpt_interval=20 
  coach.hf_save_interval=20 
  coach.logger="console" 
  coach.project_name="vlm_maze_2d_easy" 
  coach.run_name="sft_grpo_${RUN_ID}" 
  coach.resume_mode=newest 
  coach.log_path="/tmp/skyrl-logs" 
  coach.dump_eval_results=true 
  coach.export_path="$EXPORT_PATH" 
  coach.ckpt_path="s3://<your-bucket>/skyrl-visgym/ckpts/sft-grpo"

Step 4: Submit the coaching job remotely

When toolkit-for-ray-on-sagemaker-ai is put in, Ray’s customary Jobs CLI authenticates by way of the cluster’s secured endpoint utilizing the sagemaker_ray:// handle scheme the bundle registers. The library authenticates to the Ray endpoint utilizing your AWS credentials, so that you don’t have to do it your self. This fashion, you may submit and observe jobs from a laptop computer, a CI/CD pipeline, or an surroundings with AWS credentials, with no kubectl port-forward and no direct community path to the cluster.

First, authenticate towards the EKS cluster:

aws eks update-kubeconfig --name <eks-cluster-name> --region us-west-2

Then submit the job, passing the present listing because the working dir so train_job.sh is uploaded to the cluster head:

# Tackle format: sagemaker_ray://<ray-cluster-name>/<namespace>
ray job submit 
  --address sagemaker_ray://skyrl-visgym/default 
  --submission-id sft-train 
  --working-dir . 
  -- bash train_job.sh

As soon as submitted, observe progress utilizing the identical handle:

# Record jobs and examine standing
ray job listing --address sagemaker_ray://skyrl-visgym/default

# Stream logs
ray job logs sft-train 
  --address sagemaker_ray://skyrl-visgym/default --follow

You can too observe the job in SageMaker Studio below the Duties tab, or open the Ray Dashboard straight from the cluster Actions menu for a full job view with per-actor useful resource utilization.

Step 5: Monitor coaching progress

HyperPod offers two monitoring surfaces: the Ray Dashboard for job-level visibility, and Amazon Managed Grafana for infrastructure and coaching metrics. Each are accessible straight from the Duties tab in SageMaker Studio.

Ray Dashboard

From the Duties tab in SageMaker Studio, select Open Ray Dashboard. This generates a short-lived authenticated URL for you mechanically.

You can too generate the URL from the HyperPod CLI:

hyp create ray-dashboard-connection 
  --cluster-name <ray-cluster-name> 
  --namespace <kubernetes-namespace>

The command returns a presigned URL. Open it in a browser to view the Ray dashboard. The session is legitimate for as much as six hours. For extra particulars, see Producing a dashboard connection URL within the HyperPod documentation.

The next screenshot exhibits the Jobs view with the operating job, its present step, and per-worker useful resource utilization.

Determine 4: The Ray Dashboard Jobs view exhibiting the operating coaching job and per-worker GPU utilization

HyperPod Observability dashboards

From the Duties tab, select Open Grafana. The HyperPod Observability EKS add-on provisions 4 pre-built Ray dashboards in Amazon Managed Grafana: Ray Core, Ray Information, Ray Practice, and Ray Serve. All 4 seem below a Ray folder and assist filtering by cluster title. Here’s a part of the core dashboard exhibiting CPU, GPU, and reminiscence utilization whereas the coaching is in progress.

Determine 5: Amazon Managed Grafana Ray Core dashboard panels for CPU, GPU, and reminiscence utilization throughout coaching

Monitoring analysis accuracy

SkyRL runs an analysis move each eval_interval=10 steps towards the mounted 64-maze held-out set and logs eval/all/pass_at_1 to the console. You possibly can grep for it within the job logs:

ray job logs sft-train 
  --address sagemaker_ray://skyrl-visgym/default | grep "eval/all/pass_at_1"

In our experiment, the mannequin reached 75% clear up price round step 100 and peaked at 96.875% (62/64 mazes) at step 160, in comparison with a baseline of 43.75% (28/64 mazes) earlier than GRPO post-training. Your outcomes will fluctuate primarily based on hyperparameters and the maze configuration. As soon as the clear up price reaches your goal, the LoRA adapter at that step is prepared for inference. Checkpoints are saved to /shared/runs/<run_id>/global_step_<N>/coverage/adapter_model.safetensors each 20 steps.

Step 6: Host the skilled mannequin for inference

With coaching full, the artifact you deploy is a LoRA adapter reasonably than a full mannequin, and Ray Serve hundreds that adapter on demand at request time. The one requirement is the place the adapter lives. Ray Serve’s dynamic LoRA loader reads adapters from cloud storage comparable to Amazon S3, so start by staging the adapter in an S3 prefix. Copy the adapter from the checkpoint step you chose throughout coaching into that prefix:

aws s3 cp --recursive 
  /shared/runs/<run_id>/global_step_N/coverage/ 
  s3://<your-bucket>/lora-adapters/maze-grpo/

Every adapter occupies its personal subdirectory beneath this prefix, and that subdirectory title (maze-grpo on this instance) is the title you’ll use to request the adapter as soon as the endpoint is reside.

Serve with Ray Serve on a Ray cluster

To host the adapter, run Ray Serve on a Ray cluster constructed from the AWS Deep Learning Container for Ray Serve LLM, which bundles Ray Serve, the ray[llm] stack, and vLLM. With it, you may arise an OpenAI-compatible endpoint utilizing the built-in ray.serve.llm:build_openai_app builder and no customized picture. We advocate operating inference on a RayService, which KubeRay reconciles into its personal Ray cluster (see the KubeRay docs).

A RayService describes its Serve utility by way of a serveConfigV2 spec, and inside that spec the ray.serve.llm:build_openai_app builder accepts one llm_configs entry per base mannequin, as proven within the following configuration:

serveConfigV2: |
  purposes:
    - title: maze-vlm
      route_prefix: /
      import_path: ray.serve.llm:build_openai_app
      args:
        llm_configs:
          - model_loading_config:
              model_id: visgym-qwen3vl
              # base SFT mannequin on FSx
              model_source: /shared/fashions/visgym_sft_mixed_qwen3vl
            lora_config:
              # adapter prefix in S3
              dynamic_lora_loading_path: s3://<your-bucket>/lora-adapters
              max_num_adapters_per_replica: 1
            engine_kwargs:
              enable_lora: true
              # match the coaching LoRA rank
              max_lora_rank: 32
              max_loras: 1
              max_model_len: 16000
            deployment_config:
              autoscaling_config: { min_replicas: 1, max_replicas: 1 }

On this configuration, model_source factors on the base SFT mannequin on the Amazon FSx mount, whereas dynamic_lora_loading_path factors on the S3 prefix you populated within the earlier step. Set max_lora_rank to the identical LoRA rank you used throughout coaching. Lastly, ensure the Ray Serve replicas can receive AWS credentials to learn the adapters out of your S3 bucket. On Amazon EKS, the advisable mechanism for that is Amazon EKS Pod Identification.

Dynamic LoRA loading per request

With this configuration in place, a single deployment serves each the bottom mannequin and each adapter saved below the S3 prefix, and also you select which one to run by way of the mannequin area on every request:

• Set mannequin to visgym-qwen3vl to run the bottom SFT mannequin by itself.
• Set mannequin to visgym-qwen3vl:maze-grpo to use the GRPO adapter on prime of that base mannequin. The ID follows the <base-model-id>:<adapter-name> conference, the place the adapter title is the subdirectory you created below dynamic_lora_loading_path.

The primary time an adapter is requested, Ray Serve downloads it from Amazon S3 onto the duplicate and caches it, so each later request reuses the loaded weights with out downloading once more. As a result of the endpoint speaks the OpenAI API, you may name it from an OpenAI-compatible shopper. For the entire set of configuration choices, see the Ray Serve LLM documentation and the multi-LoRA deployment guide.

With the adapter deployed, the mannequin can run a full maze episode by itself, taking within the present view at every flip and returning its subsequent transfer till it reaches the aim. The loop that drives this (constructing the immediate, encoding the maze picture, parsing the motion, stepping the surroundings) is identical one SkyRL and VisGym already use for rollouts. Level that surroundings on the served endpoint reasonably than rebuilding it round a uncooked shopper.

Here’s a pattern end result, the skilled mannequin navigating two mazes finish to finish:

Determine 6: The fine-tuned mannequin fixing two mazes finish to finish, taking one transfer per flip till it reaches the aim

Clear up

Whenever you end the walkthrough, clear up the assets you created to cease incurring prices. Open the Duties tab in SageMaker Studio, choose the skyrl-visgym Ray cluster, and select Delete from the Actions menu. For those who created a separate RayService to host the skilled adapter, delete that as properly with kubectl delete rayservice <title>. To launch the GPU capability itself, scale down the HyperPod occasion teams you added for this walkthrough, or delete the HyperPod cluster from the Amazon SageMaker AI console. For those who not want the SageMaker Studio area, you may delete it from the SageMaker AI console too.

Conclusion

On this publish, we walked by way of an end-to-end multimodal RL coaching workflow on Amazon SageMaker HyperPod: constructing a customized container picture, launching a Ray cluster from SageMaker Studio, ranging from the VisGym SFT checkpoint to provide GRPO a robust initialization, submitting the job utilizing the sagemaker_ray:// protocol, monitoring analysis accuracy by way of the Ray Dashboard and Grafana dashboards provisioned by the HyperPod Observability add-on, and internet hosting the skilled adapter for inference as soon as it reached the goal clear up price. Your entire workflow runs on the resilient, self-healing compute infrastructure that Amazon SageMaker HyperPod offers.

The identical sample applies to SkyRL workloads or different Ray-based RL frameworks. The HyperPod infrastructure, the Studio console expertise, and the Amazon FSx shared storage work with frameworks that use customary Ray APIs.

To get began, see the Amazon SageMaker HyperPod documentation and the Ray on HyperPod getting began information.

Concerning the authors