Managing Open Data Hub

In the OpenShift Container Platform web console, click User Management → Groups.

Click the name of the group containing the users that you want to view.

In the OpenShift Container Platform web console, click User Management → Groups.

Click the name of the group you want to add users to.

Click Actions → Add Users.

In the Users field, enter the relevant user name to add to the group.

Click Save.

From the Open Data Hub dashboard, click Settings → User management.

Select your Open Data Hub administrator groups: Under Data science administrator groups, click the text box and select an OpenShift group. Repeat this process to define multiple administrator groups.

Select your Open Data Hub user groups: Under Data science user groups, click the text box and select an OpenShift group. Repeat this process to define multiple user groups.

Click Save changes.

Obtain the location of the default image that you want to use as the base for your custom image.

Create a standard Containerfile or Dockerfile.

For the FROM instruction, specify the base image location that you copied in Step 1, for example:

Optional: Install OS images:

Optional: Add Python packages:

Build the image file. For example, you can use podman build locally where the image file is located and then push the image to a registry that is accessible to Open Data Hub:

$ podman build -t my-registry/my-custom-image:0.0.1 .
$ podman push my-registry/my-custom-image:0.0.1

From the Open Data Hub dashboard, click Settings → Environment setup → Workbench images.

Optional: If you want to associate an accelerator and you have not already created an accelerator profile or a hardware profile, click Create profile on the row containing the image and complete the relevant fields. If the image does not contain an accelerator identifier, you must manually configure one before creating an associated accelerator profile or a hardware profile.

Click Import new image. Alternatively, if no previously imported images were found, click Import image.

In the Image location field, enter the URL of the repository containing the image. For example: quay.io/my-repo/my-image:tag, quay.io/my-repo/my-image@sha256:xxxxxxxxxxxxx, or docker.io/my-repo/my-image:tag.

In the Name field, enter an appropriate name for the image.

Optional: In the Description field, enter a description for the image.

Optional: From the Accelerator identifier list, select an identifier to set its accelerator as recommended with the image. If the image contains only one accelerator identifier, the identifier name displays by default.

Optional: Add software to the image. After the import has completed, the software is added to the image’s meta-data and displayed on the workbench creation page.

Optional: Add packages to the workbench images. After the import has completed, the packages are added to the image’s meta-data and displayed on the workbench creation page.

Click Import.

Log in to the OpenShift Container Platform console as an Open Data Hub administrator.

In the Administrator perspective, click Home → API Explorer.

In the search bar, enter OdhApplication to filter by kind.

Click the OdhApplication custom resource (CR) to open the resource details page.

From the Project list, select the Open Data Hub application namespace; the default is opendatahub.

Click the Instances tab.

Click Create OdhApplication.

On the Create OdhApplication page, copy the following code and paste it into the YAML editor.

apiVersion: dashboard.opendatahub.io/v1
kind: OdhApplication
metadata:
  name: examplename
  namespace: opendatahub
  labels:
    app: odh-dashboard
    app.kubernetes.io/part-of: odh-dashboard
spec:
  enable:
    validationConfigMap: examplename-enable
  img: >-
    <svg width="24" height="25" viewBox="0 0 24 25" fill="none" xmlns="http://www.w3.org/2000/svg">
    <path d="path data" fill="#ee0000"/>
    </svg>
  getStartedLink: 'https://example.org/docs/quickstart.html'
  route: exampleroutename
  routeNamespace: examplenamespace
  displayName: Example Name
  kfdefApplications: []
  support: third party support
  csvName: ''
  provider: example
  docsLink: 'https://example.org/docs/index.html'
  quickStart: ''
  getStartedMarkDown: >-
    # Example

    Enter text for the information panel.

  description: >-
    Enter summary text for the tile.
  category: Self-managed | Partner managed | Red Hat managed

Modify the parameters in the code for your application.

Click Create. The application details page opens.

Log in to Open Data Hub.

In the left menu, click Applications → Explore.

Locate the new tile for your application and click it.

In the information pane for the application, click Enable.

Log in to the OpenShift Container Platform console as an Open Data Hub administrator.

Open the dashboard configuration file:

In the spec.dashboardConfig section, set the value of enablement to false to disable the ability for dashboard users to add applications to the dashboard.

Click Save to apply your changes and then click Reload to make sure that your changes are synced to the cluster.

In the OpenShift Container Platform web console, switch to the Administrator perspective.

Switch to the odh project.

Click Operators → Installed Operators.

Click on the Operator that you want to uninstall. You can enter a keyword into the Filter by name field to help you find the Operator faster.

Delete any Operator resources or instances by using the tabs in the Operator interface.

On the Operator Details page, click the Actions drop-down menu and select Uninstall Operator.

Select Uninstall to uninstall the Operator, Operator deployments, and pods. After this is complete, the Operator stops running and no longer receives updates.

Log in to the OpenShift Container Platform console as an Open Data Hub administrator.

Open the dashboard configuration file:

In the spec.dashboardConfig section, set either or both of the following options:

Click Save to apply your changes and then click Reload to make sure that your changes are synced to the cluster.

Log in to the OpenShift Container Platform console as an Open Data Hub administrator.

Open the dashboard configuration file:

In the spec:notebookController section, set the value of enabled to false to remove the Start basic workbench tile from the list of applications on the Applications → Enabled page.

Click Save to apply your changes and then click Reload to make sure that your changes are synced to the cluster.

Log in to the OpenShift Container Platform console as a cluster administrator.

In the Administrator perspective, click Workloads → Deployments.

From the Project drop-down list, select opendatahub.

In the Name column, click the name of the deployment for the component that you want to customize resources for.

On the Deployment details page that is displayed, click the YAML tab.

Find the .spec.template.spec.containers.resources section.

Update the value of the resource that you want to customize. For example, to update the memory limit to 500Mi, make the following change:

containers:
        - resources:
            limits:
                cpu: '2'
                memory: 500Mi
            requests:
                cpu: '1'
                memory: 1Gi

Click Save.

Click Reload.

Log in to the OpenShift Container Platform console as a cluster administrator.

In the Administrator perspective, click Workloads → Deployments.

From the Project drop-down list, select opendatahub.

In the Name column, click the name of the deployment for the component to which you want to add the annotation.

On the Deployment details page that opens, click the YAML tab.

Find the metadata.annotations: section.

Add the opendatahub.io/managed: true annotation.

metadata:
  annotations:
     opendatahub.io/managed: true

Click Save.

Click Reload.

Log in to the OpenShift Container Platform console as a cluster administrator.

In the Administrator perspective, click Workloads → Deployments.

From the Project drop-down list, select opendatahub.

In the Name column, click the name of the deployment for the component for which you want to remove the annotation.

Click the Options menu .

Click Delete Deployment.

To enable GPU support on an OpenShift cluster, follow the instructions here: NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

Delete the migration-gpu-status ConfigMap.

Restart the dashboard replicaset.

Install the latest version of the Intel Gaudi Base Operator from OperatorHub.

Create and configure a custom workbench image for Intel Gaudi AI accelerators. A prebuilt workbench image for Gaudi accelerators is not included in Open Data Hub.

Manually define and configure an accelerator profile or a hardware profile for each Intel Gaudi AI device in your environment.

In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

$ oc login <openshift_cluster_url> -u <admin_username> -p <password>

Install the Red Hat build of Kueue Operator on your OpenShift Container Platform cluster as described in the Red Hat build of Kueue documentation.

Activate the Kueue integration. You can use the predefined names for the default cluster queue and default local queue, or specify custom names.

Verify that the Red Hat build of Kueue pods are running:

$ oc get pods -n openshift-kueue-operator

kueue-controller-manager-d9fc745df-ph77w    1/1     Running
openshift-kueue-operator-69cfbf45cf-lwtpm   1/1     Running

Verify that the default ClusterQueue was created:

$ oc get clusterqueues

Optional: When you migrate from the embedded Kueue to Red Hat build of Kueue, the Open Data Hub Operator automatically moves your existing Kueue configuration from the kueue-manager-config ConfigMap to the Kueue custom resource (CR).

$ oc annotate configmap kueue-manager-config -n <applications-namespace> opendatahub.io/managed=false

Log in to the OpenShift Container Platform web console as a cluster administrator.

Uninstall the embedded Kueue component to avoid potential configuration conflicts.

Install the Red Hat build of Kueue Operator on your OpenShift Container Platform cluster:

Activate the Red Hat build of Kueue Operator in Open Data Hub:

Enable Kueue management for existing projects by applying the kueue.openshift.io/managed=true label to each project namespace:

$ oc label namespace <project-namespace> kueue.openshift.io/managed=true --overwrite

In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

$ oc login <openshift_cluster_url> -u <admin_username> -p <password>

Verify that a resource flavor exists or create a custom one, as follows:

Verify that a default cluster queue exists or create a custom one, as follows:

Verify that a local queue that points to your cluster queue exists for your project namespace, or create a custom one, as follows:

In the OpenShift Container Platform console, click Workloads → ConfigMaps.

From the Project list, select odh.

Search for the codeflare-operator-config config map, and click the config map name to open the ConfigMap details page.

Click the YAML tab to show the config map specifications.

In the data:config.yaml:kuberay section, you can edit the following entries:

To save your changes, click Save.

To apply your changes, delete the pod:

In the OpenShift Container Platform console, click Workloads → Deployments.

Search for the codeflare-operator-manager deployment, and then click the deployment name to open the deployment details page.

Click the Pods tab. When the status of the codeflare-operator-manager-<pod-id> pod is Running, the pod is ready to use. To see more information about the pod, click the pod name to open the pod details page, and then click the Logs tab.

Log in to the OpenShift Console as a cluster administrator.

Enable NVIDIA GPU support in Open Data Hub.

To simplify the management of NVIDIA networking resources, install and configure the NVIDIA Network Operator, as follows:

[Optional] To use Single Root I/O Virtualization (SR-IOV) deployment modes, complete the following steps:

Use the Machine Configuration Operator to increase the limit of pinned memory for non-root users in the container engine (CRI-O) configuration, as follows:

Verify that the Operators are installed correctly, as follows:

Verify that RDMA is being used, as follows:

In the OpenShift CLI (oc), verify the OpenShift authentication type by running the following command:

oc get authentication.config/cluster -o jsonpath='{.spec.type}'

Verify that your OIDC provider is configured as expected by running the following command:

oc get authentication.config/cluster -o jsonpath='{.spec.type}'

Verify that the kube-apiserver has rolled out changes as expected.

oc get co kube-apiserver

NAME              VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
kube-apiserver    4.14.9    True        False         False      1d

Define the the following environment variables. You must replace the placeholder values with the actual details from your OIDC Identity Provider (IDP):

# Replace with your actual values
KEYCLOAK_DOMAIN="keycloak.example.com"
KEYCLOAK_REALM="your-realm"
KEYCLOAK_CLIENT_ID="your-client-id"
KEYCLOAK_CLIENT_SECRET="your-client-secret"

Create the client secret in the openshift-ingress namespace:

oc create secret generic keycloak-client-secret \
    --from-literal=clientSecret=$KEYCLOAK_CLIENT_SECRET \
    -n openshift-ingress

Update the GatewayConfig custom resource to enable OIDC authentication by patching it with the Secret reference and OIDC details:

oc patch gatewayconfig default-gateway --type='merge' -p='{
    "spec": {
      "oidc": {
        "issuerURL": "https://'$KEYCLOAK_DOMAIN'/realms/'$KEYCLOAK_REALM'",
        "clientID": "'$KEYCLOAK_CLIENT_ID'",
        "clientSecretRef": {
          "name": "keycloak-client-secret",
          "key": "clientSecret"
        }
      }
    }
  }'

Verify that the client secret has been created and that the GatewayConfig shows the correct OIDC configuration:

oc get secret keycloak-client-secret -n openshift-ingress
oc get gatewayconfig default-gateway -o jsonpath='{.spec.oidc}'

# Expected output (for secret)
NAME                     TYPE     DATA   AGE
keycloak-client-secret   Opaque   1      2m
# Expected output (for GatewayConfig)
{"clientID":"your-client-id","clientSecretRef":{"key":"clientSecret","name":"keycloak-client-secret"},"issuerURL":"https://keycloak.example.com/realms/your-realm"}

After configuring and authenticating the Gateway for your identity provider, you need to ensure that you can access your OpenShift console.

Check the GatewayConfig status to verify that the OIDC configuration was successfully provisioned:

oc get gatewayconfig default-gateway -o yaml

Verify the kube-auth-proxy deployment is running successfully in the openshift-ingress namespace:

oc get deployment kube-auth-proxy -n openshift-ingress

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
kube-auth-proxy   1/1     1            1           5m

Check the status and accessibility of the data-science-gateway:

oc get gateway data-science-gateway -n openshift-ingress

NAME                   CLASS                        ADDRESS                                                                        PROGRAMMED   AGE
data-science-gateway   data-science-gateway-class   aa87f5da7f0c748d5aa63b4916604108-107643684.us-east-1.elb.amazonaws.com         True         5m

Test the OpenID Connect (OIDC) discovery endpoint by running the following command:

curl -s https://your-keycloak-domain/realms/your-realm/.well-known/openid-configuration

Create a ClusterRole that grants users read and list access to OpenShift projects in the console:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: odh-projects-read
rules:
- apiGroups: ["project.openshift.io"]
  resources: ["projects"]
  verbs: ["get","list"]

Bind the odh-projects-read ClusterRole to your Identity Provider (IDP) group (e.g., odh-users):

oc adm policy add-cluster-role-to-group odh-projects-read odh-users

Grant the ability to create and manage new projects by assigning the built-in self-provisioner ClusterRole to your group:

oc adm policy add-cluster-role-to-group self-provisioner odh-users

Log in to the OpenShift Container Platform web console as a cluster administrator.

In the OpenShift Container Platform console, click Operators → Installed Operators.

Search for the Open Data Hub Operator, and then click the Operator name to open the Operator details page.

Click the DSCInitialization tab.

Click the default instance name (for example, default-dsci) to open the instance details page.

Click the YAML tab to show the instance specifications.

In the spec.monitoring section, set the value of the managementState field to Managed, and configure metrics, alerting, and tracing settings as shown in the following example:

# ...
spec:
  monitoring:
    managementState: Managed                 # Required: Enables and manages the observability stack
    namespace: opendatahub    # Required: Namespace where monitoring components are deployed
    alerting: {}                              # Alertmanager configuration, uses default settings if empty
    metrics:                                  # Prometheus configuration for metrics collection
      replicas: 1                             # Optional: Number of Prometheus instances
      resources:                              # CPU and memory requests and limits for Prometheus pods
        cpulimit: 500m                        # Optional: Maximum CPU allocation in millicores
        cpurequest: 100m                      # Optional: Minimum CPU allocation in millicores
        memorylimit: 512Mi                    # Optional: Maximum memory allocation in mebibytes
        memoryrequest: 256Mi                  # Optional: Minimum memory allocation in mebibytes
      storage:                                # Storage configuration for metrics data
        size: 5Gi                             # Required: Storage size for Prometheus data
        retention: 90d                        # Required: Retention period for metrics data in days
      exporters: {}                           # External metrics exporters
    traces:                                   # Tempo backend for distributed tracing
      sampleRatio: '0.1'                      # Optional: Portion of traces to sample, expressed as a decimal
      storage:                                # Storage configuration for trace data
        backend: pv                           # Required: Storage backend for Tempo traces (pv, s3, or gcs)
        retention: 2160h                      # Optional: Retention period for trace data in hours
      exporters: {}                           # External traces exporters
# ...

Click Save to apply your changes.

In the OpenShift Container Platform web console, click Workloads → Pods.

From the project list, select opendatahub.

Confirm that there are running pods for your configuration. The following pods indicate that the observability stack is active:

alertmanager-data-science-monitoringstack-#      2/2   Running   0   1m
data-science-collector-collector-#               1/1   Running   0   1m
prometheus-data-science-monitoringstack-#        2/2   Running   0   1m
tempo-data-science-tempomonolithic-#             1/1   Running   0   1m
thanos-querier-data-science-thanos-querier-#     2/2   Running   0   1m

Log in to the OpenShift Container Platform web console as a cluster administrator or project owner.

Click Workloads → Deployments.

In the Project list at the top of the page, select the project where your workload is deployed.

Identify the deployment that you want to collect metrics from and click its name.

On the Deployment details page, click the YAML tab.

In the YAML editor, add the required label under the spec.template.metadata.labels section, as shown in the following example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <example_name>
  namespace: <example_namespace>
spec:
  template:
    metadata:
      labels:
        monitoring.opendatahub.io/scrape: 'true'
# ...

Click Save.

Access Prometheus by using a route:

Alternatively, access Prometheus locally by using port forwarding:

In the Prometheus web console, search for a metric exposed by your workload.

Log in to the OpenShift Container Platform web console as a cluster administrator.

Click Operators → Installed Operators.

Select the Open Data Hub Operator from the list.

Click the DSCInitialization tab.

Click the default DSCI instance, for example, default-dsci, to open its details page.

Click the YAML tab.

In the spec.monitoring.metrics section, add an exporters list that defines the external receiver configuration, as shown in the following example:

spec:
  monitoring:
    metrics:
      exporters:
        - name: <external_exporter_name>
          type: <type>
          endpoint: https://example-otlp-receiver.example.com:4317

Click Save.

Verify that the OpenTelemetry Collector pods restart and apply the new configuration:

$ oc get pods -n opendatahub

In your external observability platform, verify that new metrics from Open Data Hub appear in the metrics list or dashboard.

Find the OpenTelemetry Collector endpoint.

Connect your visualization tool to the Tempo query service.

Confirm that your instrumented application is generating and exporting trace data.

Verify that the OpenTelemetry Collector pod is running in the monitoring namespace:

$ oc get pods -n opendatahub | grep collector

Access your visualization tool and confirm that new traces appear in the trace list or search view.

In a terminal window, log in to the OpenShift CLI (oc) as a cluster administrator:

$ oc login https://api.198.51.100.10:6443

Verify that the Alertmanager pods are running in the monitoring namespace:

$ oc get pods -n opendatahub | grep alertmanager

alertmanager-data-science-monitoringstack-0   2/2   Running   0   2h
alertmanager-data-science-monitoringstack-1   2/2   Running   0   2h

Confirm that a ClusterIP service exposes the Alertmanager web interface on port 9093:

$ oc get svc -n opendatahub | grep alertmanager

data-science-monitoringstack-alertmanager     ClusterIP   198.51.100.5   <none>   9093/TCP

Start a local port forward to the Alertmanager service:

$ oc port-forward svc/data-science-monitoringstack-alertmanager 9093:9093 -n opendatahub

In a web browser, open the following URL to access the Alertmanager web interface:

http://localhost:9093

Log in to the OpenShift Container Platform as a cluster administrator.

Click Operators → Installed Operators and then click the Open Data Hub Operator.

Click the DSC Initialization tab.

Click the default-dsci object.

Click the YAML tab.

In the spec section, update the .spec.devFlags.logmode flag with the log level that you want to set.

apiVersion: dscinitialization.opendatahub.io/v1
kind: DSCInitialization
metadata:
  name: default-dsci
spec:
  devFlags:
    logmode: development

Click Save.

In a terminal window, if you are not already logged in to your OpenShift Container Platform cluster as a cluster administrator, log in to the OpenShift Container Platform CLI as shown in the following example:

$ oc login <openshift_cluster_url> -u <admin_username> -p <password>

To access the full content of the changed custom resources, set the OpenShift Container Platform audit log policy to WriteRequestBodies or a more comprehensive profile. For more information, see Configuring the audit log policy.

Fetch the audit log files that are available for the relevant control plane nodes. For example:

oc adm node-logs --role=master --path=kube-apiserver/ \
  | awk '{ print $1 }' | sort -u \
  | while read node ; do
      oc adm node-logs $node --path=kube-apiserver/audit.log < /dev/null
    done \
  | grep opendatahub > /tmp/kube-apiserver-audit-opendatahub.log

Search the files for the DSC and DSCI custom resources. For example:

jq 'select((.objectRef.apiGroup == "dscinitialization.opendatahub.io"
                or .objectRef.apiGroup == "datasciencecluster.opendatahub.io")
              and .user.username != "system:serviceaccount:openshift-operators:redhat-ods-operator-controller-manager"
              and .verb != "get" and .verb != "watch" and .verb != "list")' < /tmp/kube-apiserver-audit-opendatahub.log