{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# ADES\n", "\n", "This notebook demostrates usage of the EODH ADES API using `pyeodh`.\n", "\n", "EODH provides Application Deployment & Execution Service - ADES, to which you can deploy workflows and execute parametrised processing jobs. Pyeodh provides an interface to simplify interaction with ADES from python scripts.\n", "\n", "First we need to instantiate pyeodh client and get the ADES entrypoint.\n", "\n", "Note: This API requires authentication credentials to be provided by the user (in this case read from environment variables). This is a subject to change as the hub is implementing proper IAM solution." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "from pprint import pp\n", "\n", "from requests import HTTPError\n", "\n", "import pyeodh\n", "\n", "username = os.getenv(\"ADES_USER\")\n", "token = os.getenv(\"ADES_TOKEN\")\n", "\n", "client = pyeodh.Client(username=username, token=token)\n", "ades = client.get_ades()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Processes (or workflows) are predefined applications which can be parametrised and executed by users. To get a list of currently available processes in our user workspace call `Ades.get_processes()` method:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "display\n", "echo\n", "convert-stac\n", "water-bodies\n", "convert-url\n" ] } ], "source": [ "for p in ades.get_processes():\n", " print(p.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can fetch a specific workflow if you know it's ID using `Ades.get_process()` method. The `Process` object also contains metadata giving us more information about the process and how to execute it, for example the schema of inputs we can use to parametrise the process or output schema." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'fn': {'title': 'the operation to perform',\n", " 'description': 'the operation to perform',\n", " 'schema': {'type': 'string', 'default': 'resize', 'nullable': True}},\n", " 'size': {'title': 'the percentage for a resize operation',\n", " 'description': 'the percentage for a resize operation',\n", " 'schema': {'type': 'string', 'default': '50%', 'nullable': True}},\n", " 'url': {'title': 'the image to convert',\n", " 'description': 'the image to convert',\n", " 'schema': {'type': 'string',\n", " 'default': 'https://eoepca.org/media_portal/images/logo6_med.original.png',\n", " 'nullable': True}}}\n", "{'converted_image': {'title': 'converted_image',\n", " 'description': 'None',\n", " 'extended-schema': {'oneOf': [{'allOf': [{'$ref': 'http://zoo-project.org/dl/link.json'},\n", " {'type': 'object',\n", " 'properties': {'type': {'enum': ['application/json']}}}]},\n", " {'type': 'object',\n", " 'required': ['value'],\n", " 'properties': {'value': {'oneOf': [{'type': 'object'}]}}}]},\n", " 'schema': {'oneOf': [{'type': 'object'}]}}}\n" ] } ], "source": [ "convert_url_proc = ades.get_process(\"convert-url\")\n", "\n", "pp(convert_url_proc.inputs_schema)\n", "pp(convert_url_proc.outputs_schema)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get list of previously executed jobs" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "45cfba32-4ad1-11ef-a8c6-66f4c7b2b89b convert-url failed\n", "9dcfec66-4ad1-11ef-ab61-66f4c7b2b89b convert-url failed\n", "d10202bc-53ec-11ef-8d4c-06180dac6ed1 water-bodies successful\n" ] } ], "source": [ "for j in ades.get_jobs():\n", " print(j.id, j.process_id, j.status)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Only one process with the same ID can exist. To demonstrate deploying a process further down in this notebook, we first need to undeploy `convert-url`. Note that attempting to delete a non-existent process will result in 4xx http status code." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "try:\n", " ades.get_process(\"convert-url\").delete()\n", "except HTTPError:\n", " print(\"Process not found, no need to undeploy.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's deploy the `convert-url` process again. There are 2 ways we can provide the CWL file - either referencing the file by URL or by passing the CWL file content directly. Note that `Ades.deploy_process()` will fail if we try to create a process with ID that already exists. If we want to update an existing process, we should use `Process.update()` method instead. Both methods can handle URL or CWL YAML. In this example we deploy a process referencing by URL and then update it by passing the new CWL YAML content directly. Also note that when updating a worklow you need to provide the entire workflow, the API does not support partial updates (e.g. to change the description we need to provide the entire workflow again)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "convert-url Convert URL\n" ] } ], "source": [ "convert_url_proc = ades.deploy_process(\n", " cwl_url=\"https://raw.githubusercontent.com/EOEPCA/deployment-guide/main/deploy/samples/requests/processing/convert-url-app.cwl\"\n", ")\n", "print(convert_url_proc.id, convert_url_proc.description)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "convert-url Convert URL YAML\n" ] } ], "source": [ "cwl_yaml = \"\"\"cwlVersion: v1.0\n", "$namespaces:\n", " s: https://schema.org/\n", "s:softwareVersion: 0.1.2\n", "schemas:\n", " - http://schema.org/version/9.0/schemaorg-current-http.rdf\n", "$graph:\n", " # Workflow entrypoint\n", " - class: Workflow\n", " id: convert-url\n", " label: convert url app\n", " doc: Convert URL YAML\n", " requirements:\n", " ResourceRequirement:\n", " coresMax: 1\n", " ramMax: 1024\n", " inputs:\n", " fn:\n", " label: the operation to perform\n", " doc: the operation to perform\n", " type: string\n", " url:\n", " label: the image to convert\n", " doc: the image to convert\n", " type: string\n", " size:\n", " label: the percentage for a resize operation\n", " doc: the percentage for a resize operation\n", " type: string\n", " outputs:\n", " - id: converted_image\n", " type: Directory\n", " outputSource:\n", " - convert/results\n", " steps:\n", " convert:\n", " run: \"#convert\"\n", " in:\n", " fn: fn\n", " url: url\n", " size: size\n", " out:\n", " - results\n", " # convert.sh - takes input args `--url`\n", " - class: CommandLineTool\n", " id: convert\n", " requirements:\n", " ResourceRequirement:\n", " coresMax: 1\n", " ramMax: 512\n", " hints:\n", " DockerRequirement:\n", " dockerPull: eoepca/convert:latest\n", " baseCommand: convert.sh\n", " inputs:\n", " fn:\n", " type: string\n", " inputBinding:\n", " position: 1\n", " url:\n", " type: string\n", " inputBinding:\n", " position: 2\n", " prefix: --url\n", " size:\n", " type: string\n", " inputBinding:\n", " position: 3\n", " outputs:\n", " results:\n", " type: Directory\n", " outputBinding:\n", " glob: .\n", "\"\"\"\n", "\n", "convert_url_proc.update(cwl_yaml=cwl_yaml)\n", "print(convert_url_proc.id, convert_url_proc.description)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's execute our deployed process. We need to provide inputs as a dictionary in this format, see `Process.inputs_schema` property for inputs this particular workflow is expecting." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "f280956e-58ad-11ef-8ed7-8635b5d444bf running ZOO-Kernel accepted to run your service!\n" ] } ], "source": [ "convert_url_job = convert_url_proc.execute(\n", " {\n", " \"fn\": \"resize\",\n", " \"url\": \"https://eoepca.org/media_portal/images/logo6_med.original.png\",\n", " \"size\": \"50%\",\n", " }\n", ")\n", "\n", "print(convert_url_job.id, convert_url_job.status, convert_url_job.message)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The job should now be running. Call `Job.refresh()` method to get the most up-to-date status and interrogate `Job.status` and `Job.message` properties. Note that these properties only hold the latest response from the API, and don't keep any historical records." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "f280956e-58ad-11ef-8ed7-8635b5d444bf running processing environment created, preparing execution\n" ] } ], "source": [ "convert_url_job.refresh()\n", "print(convert_url_job.id, convert_url_job.status, convert_url_job.message)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can continually poll the job using a simple loop and print status and message udpates like so:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "Status: running\n", "Message: processing environment created, preparing execution\n", "Message: execution submitted\n", "Message: delivering outputs, logs and usage report\n", "Message: Post-execution hook\n", "\n", "\n", "Status: successful\n", "Message: ZOO-Kernel successfully run your service!\n" ] } ], "source": [ "import time\n", "\n", "from pyeodh.ades import AdesJobStatus\n", "\n", "old_status = \"\"\n", "old_message = \"\"\n", "while convert_url_job.status == AdesJobStatus.RUNNING.value:\n", " time.sleep(2)\n", " convert_url_job.refresh()\n", " if convert_url_job.status != old_status:\n", " print(\"\\n\")\n", " print(f\"Status: {convert_url_job.status}\")\n", " if convert_url_job.message != old_message:\n", " print(f\"Message: {convert_url_job.message}\")\n", "\n", " old_status = convert_url_job.status\n", " old_message = convert_url_job.message" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After the job has finished successfully, we can view the results, where the data files are referenced by assets" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "logo6_med.original-resize-1723653500.022521511 {'logo6_med.original-resize': }\n" ] } ], "source": [ "results = convert_url_job.get_result_items()\n", "for res in results:\n", " print(res.id, res.assets)" ] } ], "metadata": { "kernelspec": { "display_name": "pyeodh-7PEGLt8C-py3.12", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 2 }