diff --git a/content/_redirects b/content/_redirects index 6e8120df13..8751cdd8ac 100644 --- a/content/_redirects +++ b/content/_redirects @@ -31,9 +31,10 @@ /docs/pipelines/install-sdk/ /docs/pipelines/sdk/install-sdk/ /docs/pipelines/lightweight-python-components/ /docs/pipelines/sdk/lightweight-python-components/ /docs/pipelines/build-pipeline/ /docs/pipelines/tutorials/build-pipeline/ -/docs/pipelines/pipelines-tutorial/ /docs/gke/pipelines/pipelines-tutorial/ -/docs/pipelines/tutorials/pipelines-tutorial/ /docs/gke/pipelines/pipelines-tutorial/ -/docs/gke/pipelines-tutorial/ /docs/gke/pipelines/pipelines-tutorial/ +/docs/pipelines/pipelines-tutorial/ /docs/pipelines/tutorials/cloud-tutorials/ +/docs/pipelines/tutorials/pipelines-tutorial/ /docs/pipelines/tutorials/cloud-tutorials/ +/docs/gke/pipelines-tutorial/ /docs/pipelines/tutorials/cloud-tutorials/ +/docs/gke/pipelines/pipelines-tutorial/ /docs/pipelines/tutorials/cloud-tutorials/ /docs/gke/authentication-pipelines/ /docs/gke/pipelines/authentication-pipelines/ /docs/pipelines/metrics/ /docs/pipelines/sdk/pipelines-metrics/ diff --git a/content/docs/gke/pipelines/pipelines-tutorial.md b/content/docs/gke/pipelines/pipelines-tutorial.md deleted file mode 100644 index b7dfdb4d06..0000000000 --- a/content/docs/gke/pipelines/pipelines-tutorial.md +++ /dev/null @@ -1,621 +0,0 @@ -+++ -title = "Pipelines End-to-end on GCP" -description = "An end-to-end tutorial for Kubeflow Pipelines on GCP" -weight = 100 -+++ - -This guide walks you through a Kubeflow Pipelines sample that runs an MNIST -machine learning (ML) model on Google Cloud Platform (GCP). - -## Introductions - -[Kubeflow Pipelines](/docs/pipelines/) is a platform for building and -deploying portable, scalable ML workflows based on -Docker containers. When you install Kubeflow, you get Kubeflow Pipelines too. - -By working through this tutorial, you learn how to deploy Kubeflow on -Kubernetes Engine (GKE) and run a pipeline supplied as a Python script. -The pipeline trains an MNIST model for image classification and serves the model -for online inference (also known as online prediction). - -### Overview of GCP and GKE - -Google Cloud Platform (GCP) is a suite of cloud computing services running -on Google infrastructure. The services include compute power, data storage, -data analytics, and machine learning. - -The [Cloud Shell][cloud-shell] is a browser interface that provides command-line access to cloud resources that you can use to interact with GCP, including the `gcloud` command and others. - -[Kubernetes Engine][kubernetes-engine] (GKE) is a managed service on GCP where -you can deploy containerized applications. You describe the resources that your -application needs, and GKE provisions and manages the underlying -cloud resources automatically. - -Here's a list of the primary GCP services that you use when following this -guide: - - * [Deployment Manager][deployment-manager] - * [GKE][kubernetes-engine] - * [Compute Engine][compute-engine] - * [Container Registry][container-registry] - * [Cloud Storage][cloud-storage] - -### The model and the data - -This tutorial trains a [TensorFlow][tensorflow] model on the -[MNIST dataset][mnist-data], which is a *hello world* scenario for machine -learning. - -The MNIST dataset contains a large number of images of hand-written digits in -the range 0 to 9, as well as the labels identifying the digit in each image. - -After training, the model can classify incoming images into 10 categories -(0 to 9) based on what it's learned about handwritten images. In other words, -you send an image to the model, and the model does its best to identify the -digit shown in the image. -Prediction UI - -In the above screenshot, the image shows a hand-written **7**. This image was -the input to the model. The table below the image shows a bar graph for each -classification label from 0 to 9, as output by the model. Each bar -represents the probability that the image matches the respective label. -Judging by this screenshot, the model seems pretty confident that this image -is a 7. - -## Set up your environment - -Let's get started! - -### Set up your GCP account and SDK - -Follow these steps to set up your GCP environment: - -1. Select or create a project on the [GCP Console][gcp-console]. -1. Make sure that billing is enabled for your project. See the guide to - [modifying a project's billing settings][billing-guide]. -1. Use [Cloud console to grant your team access][granting-changing-revoking-access] to Kubeflow by assigning them the following roles: - - - **Project Owner**: Ensures that your team can access all of the resources used in this guide. - - **IAP-secured Web App User**: This guide uses Cloud Identity-Aware Proxy (IAP) to secure access to your Kubeflow cluster. Your team must be members of the IAP-secured Web App User role to authenticate with the Kubeflow web application. - -Notes: - -* As you work through this tutorial, your project uses billable components of - GCP. To minimise costs, follow the instructions to - [clean up your GCP resources](#cleanup) when you've finished with them. -* This guide uses [Cloud Shell][cloud-shell] to manage your GCP environment, to save you the steps of installing [Cloud SDK][cloud-sdk] and [kubectl][kubectl]. - -### Start your Cloud Shell - -Follow the link to activate a -[Cloud Shell environment](https://console.cloud.google.com/cloudshell) in your -browser. - -### Set up some handy environment variables - -Set up the following environment variables for use throughout the tutorial: - -1. Set your GCP project ID. In the command below, replace `` - with your [project ID][gcp-project-id]: - - ``` - export PROJECT= - gcloud config set project ${PROJECT} - ``` - -1. Set the zone for your GCP configuration. Choose a zone that offers the - resources you need. See the guide to [GCP regions and zones][regions-zones]. - * Ensure you have enough Compute Engine regional capacity. - By default, the GKE cluster setup described in this guide - requires 16 CPUs. - * If you want a GPU, ensure your zone offers GPUs. - - For example, the following commands set the zone to `us-central1-c`: - - ``` - export ZONE=us-central1-c - gcloud config set compute/zone ${ZONE} - ``` - -1. If you want a custom name for your Kubeflow deployment, set the - `DEPLOYMENT_NAME` environment variable. The deployment name must be - **4-20 characters** in length. If you don't set this - environment variable, your deployment gets the default name of `kubeflow`: - - ``` - export DEPLOYMENT_NAME=kubeflow - ``` - -### Deploy Kubeflow - -Deploy Kubeflow on GCP: - -1. Follow the instructions in the - guide to [deploying Kubeflow on GCP](/docs/gke/deploy/), - taking note of the following: - - * If you want the most simple deployment experience, use the Kubeflow deployment web app - as described in the guide to - [deployment using the UI](/docs/gke/deploy/deploy-ui/). - The deployment web app currently supports - **Kubeflow {{% kf-deployment-ui-version %}}**. - * For more control - over the deployment, use the guide to - [deployment using the CLI](/docs/gke/deploy/deploy-cli/). - The CLI supports **Kubeflow {{% kf-latest-version %}}** and later versions. - * Make sure that you enable **Cloud Identity-Aware Proxy (IAP)** - as prompted during the deployment process. - * When setting up the **authorized redirect URI** for the **OAuth client - credentials**, use the same value for the `` as you used - when setting up the `DEPLOYMENT_NAME` environment variable earlier in this - tutorial. - * The following screenshot shows the Kubeflow deployment UI with hints about - the value for each input field: - - Prediction UI - -1. (Optional) If you want to examine your cluster while waiting for the - Kubeflow dashboard to - be available, you can use `kubectl` to connect to your cluster: - - * Connect your Cloud Shell session to the cluster: - - ``` - gcloud container clusters get-credentials \ - ${DEPLOYMENT_NAME} --zone ${ZONE} --project ${PROJECT} - ``` - - * Switch to the `kubeflow` namespace to see the resources on the Kubeflow - cluster: - - ``` - kubectl config set-context $(kubectl config current-context) --namespace=kubeflow - ``` - - * Check the resources deployed in the `kubeflow` namespace: - - ``` - kubectl get all - ``` - -1. Access the Kubeflow UI, which becomes available at the following URI after - several minutes: - - ``` - https://.endpoints..cloud.goog/ - ``` - - The following screenshot shows the Kubeflow UI: - Prediction UI - -1. Click **Pipelines** to access the pipelines UI. The pipelines UI - looks like this: - Pipelines UI - -Notes: - -* While the deployment is running, you can watch your resources appear on the - GCP console: - * [Deployment on Deployment Manager][gcp-console-deployment-manager] - * [Cluster on GKE][gcp-console-kubernetes-engine] - * [Computing resources on Compute Engine][gcp-console-compute-engine] - -* It can take 10-15 minutes for the URI to become available. Kubeflow needs - to provision a signed SSL certificate and register a DNS name. - - If you own/manage the domain or a subdomain with [Cloud DNS][dns] - then you can configure this process to be much faster. See - [kubeflow/kubeflow#731](https://github.com/kubeflow/kubeflow/issues/731). - -### Create a Cloud Storage bucket - -The next step is to create a Cloud Storage bucket to hold your trained model. - -[Cloud Storage][cloud-storage] is a scalable, fully-managed object/blob store. -You can use it for a range of scenarios including serving website content, -storing data for archival and disaster recovery, or distributing large data -objects to users via direct download. This tutorial uses Cloud Storage to -hold the trained machine learning model and associated data. - -Use the [`gsutil mb`][gsutil-mb] command to create a storage bucket. Your -*bucket name* must be unique across all of Cloud Storage. -The following commands create a bucket in the region that corresponds to the -zone which you specified earlier in the tutorial: - -``` -export BUCKET_NAME=${PROJECT}-${DEPLOYMENT_NAME}-bucket -export REGION=$(gcloud compute zones describe $ZONE --format="value(region.basename())") -gsutil mb -c regional -l ${REGION} gs://${BUCKET_NAME} -``` - -## Prepare your pipeline - -To simplify this tutorial, you can use a set of prepared files that include -the pipeline definition and supporting files. The project files are in the -[Kubeflow examples repository](https://github.com/kubeflow/examples/tree/master/pipelines/mnist-pipelines) -on GitHub. - -### Download the project files - -Clone the project files and go to the directory containing the MNIST pipeline -example: - -``` -cd ${HOME} -git clone https://github.com/kubeflow/examples.git -cd examples/pipelines/mnist-pipelines -``` - -As an alternative to cloning, you can download the -[Kubeflow examples repository zip file](https://github.com/kubeflow/examples/archive/master.zip). - -### Set up Python - -You need **Python 3.5 or above**. This tutorial uses Python 3.7. -If you don't have a Python 3 environment set up, install -[Miniconda](https://conda.io/miniconda.html) as described below: - -* In a Debian/Ubuntu/[Cloud shell](https://console.cloud.google.com/cloudshell) - environment, run the following commands: - - ```bash - apt-get update - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh - bash Miniconda3-latest-Linux-x86_64.sh - ``` - -* In a Windows environment, download the - [installer](https://repo.continuum.io/miniconda/Miniconda3-latest-Windows-x86_64.exe) - and make sure you select the "*Add Miniconda to my PATH environment variable*" - option during the installation. - -* In a Mac environment, download the - [installer](https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh) - and run the following command: - - ```bash - bash Miniconda3-latest-MacOSX-x86_64.sh - ``` - -Create a clean Python 3 environment (this tutorial uses Python 3.7): - -```bash -conda create --name mlpipeline python=3.7 -conda activate mlpipeline -``` - -If the `conda` command is not found, be sure to add Miniconda to your path: - -```bash -export PATH=MINICONDA_PATH/bin:$PATH -``` - -### Install the Kubeflow Pipelines SDK - -Install the Kubeflow Pipelines SDK, along with other Python dependencies defined -in the `requirements.txt` file: - -```bash -pip install -r requirements.txt --upgrade -``` - -### Compile the sample pipeline - -The pipeline is defined in the Python file `mnist_pipeline.py` which you -downloaded from GitHub. When you execute that Python file, it compiles the -pipeline to an intermediate representation which you can then upload to the -Kubeflow Pipelines service. - -Run the following command to compile the pipeline: - -```bash -python3 mnist_pipeline.py -``` - -Alongside your `mnist_pipeline.py` file, you should now have a file called -`mnist_pipeline.py.tar.gz` which contains the compiled pipeline. - -## Run the pipeline - -Go back to the the Kubeflow Pipelines UI, which you accessed in an earlier step -of this tutorial. Now you're ready to upload and run your pipeline using that -UI. - -1. Click **Upload pipeline** on the Kubeflow Pipelines UI: - Upload a pipeline via the UI - -1. Upload your `mnist_pipeline.py.tar.gz` file and give the pipeline a name: - Enter the pipeline upload details - -1. Your pipeline now appears in the list of pipelines on the UI. - Click your pipeline name: - Uploaded pipeline in list of pipelines - -1. The UI shows your pipeline's graph and various options. - Click **Create run**: - Pipeline graph and options - -1. Supply the following **run parameters**: - - * **Run name:** A descriptive name for this run of the pipeline. You can - submit multiple runs of the same pipeline. - * **bucket-path:** The Cloud Storage bucket that you created earlier to hold the - results of the pipeline run. - - The sample supplies the values for the other parameters: - - * train-steps: The number of training steps to run. - * learning-rate: The learning rate for model training. - * batch-size: The batch size for model training. - - Then click **Start**: - Starting a pipeline run - -1. The pipeline run now appears in the list of runs: - List of pipeline runs - -1. Click the run to see its details. In the following screenshot, the first - two components (`train` and `serve`) have finished successfully and the third - component (`web-ui`) is still running: - A running pipeline - -1. Click on any component to see its logs. - Logs for a pipeline component - -1. When the pipeline run is complete, look at the logs for the `web-ui` - component to find the IP address created for the MNIST web interface. - Copy the IP address and paste it into your web - browser's address bar. The web UI should appear. - - Below the connect screen, you should see a prediction UI for your MNIST - model. - Prediction UI - - Each time you refresh the page, it loads a random image from the MNIST test - dataset and performs a prediction. In the above screenshot, the image shows a - hand-written **7**. The table below the image shows a bar graph for each - classification label from 0 to 9. Each bar represents - the probability that the image matches the respective label. - -Notes: - -* You can find your trained model data in the bucket path you entered in step 5 of this procedure. - -## Understanding the pipeline definition code - -The pipeline is defined in the Python file `mnist_pipeline.py` which you -downloaded from GitHub. The following sections give an overview of the content -of that file. - -### Decorator - -The `@dsl.pipeline` decorator provides metadata about the pipeline: - -``` -@dsl.pipeline( - name='MNIST', - description='A pipeline to train and serve the MNIST example.' -) -``` - -### Function header - -The `mnist_pipeline` function defines the pipeline. The function includes a -number of arguments which are exposed in the Kubeflow Pipelines UI when you -create a new run of the pipeline. -Although you pass these arguments as strings, the arguments are of type -[`kfp.dsl.PipelineParam`](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/dsl/_pipeline_param.py). - -``` -def mnist_pipeline(model_export_dir='gs://your-bucket/export', - train_steps='200', - learning_rate='0.01', - batch_size='100'): -``` - -### The training component (`train`) - -The following block defines the `train` component, which handles the training -of the ML model: - -``` -train = dsl.ContainerOp( - name='train', - image='gcr.io/kubeflow-examples/mnist/model:v20190304-v0.2-176-g15d997b', - arguments=[ - "/opt/model.py", - "--tf-export-dir", model_export_dir, - "--tf-train-steps", train_steps, - "--tf-batch-size", batch_size, - "--tf-learning-rate", learning_rate - ] -).apply(gcp.use_gcp_secret('user-gcp-sa')) -``` -A component consists of a -[`kfp.dsl.ContainerOp`](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/dsl/_container_op.py) -object with a name and a container path. The container image for -the MNIST training component is defined in the MNIST example's -[`Dockerfile.model`](https://github.com/kubeflow/examples/blob/master/mnist/Dockerfile.model). - -The training component runs with access to your `user-gcp-sa` secret, which -ensures the component has read/write access to your Cloud Storage bucket for -storing the output from the model training. - -### The model serving component (`serve`) - -The following block defines the `serve` component, which serves the trained -model for prediction: - -``` -serve = dsl.ContainerOp( - name='serve', - image='gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:\ - 7775692adf28d6f79098e76e839986c9ee55dd61', - arguments=[ - '--model-export-path', model_export_dir, - '--server-name', "mnist-service" - ] -).apply(gcp.use_gcp_secret('user-gcp-sa')) -serve.after(train) -``` -The `serve` component differs from the `train` component with respect to -how long the service lasts. While `train` runs a single container and then -exits, `serve` runs a container that launches long-lived resources in the -cluster. - -The `ContainerOP` takes two arguments: - -* A path pointing to the location of your trained model. -* A server name. - -The component creates a Kubeflow -[`tf-serving`](https://github.com/kubeflow/kubeflow/tree/master/kubeflow/tf-serving) -service within the cluster. This service lives on after the pipeline has -finished running. - -You can see the Dockerfile used to build this container in the -[Kubeflow Pipelines repository](https://github.com/kubeflow/pipelines/blob/master/components/kubeflow/deployer/Dockerfile). -Like the `train` component, `serve` requires access to the `user-gcp-sa` secret -for access to the `kubectl` command within the container. - -The `serve.after(train)` line specifies that this component must run -sequentially after the `train` component is complete. - -### The web UI component (`web-ui`) - -The following block defines the `web-ui` component, which displays a simple -web page. The web application sends an image (picture) to the trained model and -displays the prediction results: - -``` -web_ui = dsl.ContainerOp( - name='web-ui', - image='gcr.io/kubeflow-examples/mnist/deploy-service:latest', - arguments=[ - '--image', 'gcr.io/kubeflow-examples/mnist/web-ui:\ - v20190304-v0.2-176-g15d997b-pipelines', - '--name', 'web-ui', - '--container-port', '5000', - '--service-port', '80', - '--service-type', "LoadBalancer" - ] -).apply(gcp.use_gcp_secret('user-gcp-sa')) -web_ui.after(serve) -``` - -Like `serve`, the `web-ui` component launches a service that continues to exist -after the pipeline is complete. Instead of launching a Kubeflow resource, the -`web-ui` component launches a standard Kubernetes deployment/service pair. You -can see the Dockerfile that builds the deployment image in the -[`./deploy-service/Dockerfile`](https://github.com/kubeflow/examples/blob/master/pipelines/mnist-pipelines/deploy-service/Dockerfile) -that you downloaded with the sample files. This image runs the -`gcr.io/kubeflow-examples/mnist/web-ui:v20190304-v0.2-176-g15d997b-pipelines` -container, which was built from the MNIST example's -[web-ui Dockerfile](https://github.com/kubeflow/examples/blob/master/mnist/web-ui/Dockerfile). - -This component provisions a LoadBalancer service that gives external access to a -`web-ui` deployment launched in the cluster. - -### The main function - -The `main` function compiles the pipeline, converting the Python program to -the intermediate YAML representation required by the Kubeflow Pipelines service -and zipping the result into a `tar.gz` file: - -``` -if __name__ == '__main__': - import kfp.compiler as compiler - compiler.Compiler().compile(mnist_pipeline, __file__ + '.tar.gz') -``` - - -## Clean up your GCP environment - -Run the following command to delete your deployment and related resources: - -``` -gcloud deployment-manager --project=${PROJECT} deployments delete ${DEPLOYMENT_NAME} -``` - -Delete your Cloud Storage bucket when you've finished with it: - -``` -gsutil rm -r gs://${BUCKET_NAME} -``` - -As an alternative to the command line, you can delete the various resources -using the [GCP Console][gcp-console]. - -## Next steps - -Build your own machine-learning pipelines with the [Kubeflow Pipelines -SDK](/docs/pipelines/sdk/sdk-overview/). - -[mnist-data]: http://yann.lecun.com/exdb/mnist/index.html - -[tensorflow]: https://www.tensorflow.org/ - -[kubernetes]: https://kubernetes.io/ -[kubectl]: https://kubernetes.io/docs/reference/kubectl/kubectl/ - -[kubernetes-engine]: https://cloud.google.com/kubernetes-engine/ -[container-registry]: https://cloud.google.com/container-registry/ -[cloud-storage]: https://cloud.google.com/storage/ -[deployment-manager]: https://cloud.google.com/deployment-manager/ -[compute-engine]: https://cloud.google.com/compute/ -[billing-guide]: https://cloud.google.com/billing/docs/how-to/modify-project -[regions-zones]: https://cloud.google.com/compute/docs/regions-zones/ -[iap]: https://cloud.google.com/iap/ -[dns]: https://cloud.google.com/dns/docs/ - -[cloud-sdk]: https://cloud.google.com/sdk/docs/ -[gcloud]: https://cloud.google.com/sdk/gcloud/ -[gcp-project-id]: https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects -[gcp-locations]: https://cloud.google.com/about/locations/ -[gcp-console]: https://console.cloud.google.com/cloud-resource-manager -[gcp-console-kubernetes-engine]: https://console.cloud.google.com/kubernetes -[gcp-console-workloads]: https://console.cloud.google.com/kubernetes/workload -[gcp-console-storage]: https://console.cloud.google.com/storage -[gcp-console-consent]: https://console.cloud.google.com/apis/credentials/consent -[gcp-console-credentials]: https://console.cloud.google.com/apis/credentials -[gcp-console-deployment-manager]: https://console.cloud.google.com/dm/ -[gcp-console-compute-engine]: https://console.cloud.google.com/compute/ -[gcp-console-services]: https://console.cloud.google.com/kubernetes/discovery -[cr-tf-models]: https://console.cloud.google.com/gcr/images/tensorflow/GLOBAL/models - -[cloud-shell]: https://cloud.google.com/shell/ -[gcloud-container-clusters-create]: https://cloud.google.com/sdk/gcloud/reference/container/clusters/create -[gcp-machine-types]: https://cloud.google.com/compute/docs/machine-types -[gcp-service-account]: https://cloud.google.com/iam/docs/understanding-service-accounts -[gcp-container-registry]: https://console.cloud.google.com/gcr -[gsutil-mb]: https://cloud.google.com/storage/docs/gsutil/commands/mb -[gsutil-acl-ch]: https://cloud.google.com/storage/docs/gsutil/commands/acl#ch - -[granting-changing-revoking-access]: https://cloud.google.com/iam/docs/granting-changing-revoking-access#using_the -[release-page]:https://github.com/kubeflow/kubeflow/releases diff --git a/content/docs/pipelines/pipelines-quickstart.md b/content/docs/pipelines/pipelines-quickstart.md index f9fc6196c3..fd00548e63 100644 --- a/content/docs/pipelines/pipelines-quickstart.md +++ b/content/docs/pipelines/pipelines-quickstart.md @@ -8,8 +8,7 @@ weight = 10 feedbacklink="https://github.com/kubeflow/pipelines/issues" %}} Use this guide if you want to get a simple pipeline running quickly in -Kubeflow Pipelines. If you need a more in-depth guide, see the -[end-to-end tutorial](/docs/gke/pipelines-tutorial/). +Kubeflow Pipelines. * This quickstart guide shows you how to use one of the samples that come with the Kubeflow Pipelines installation and are visible on the Kubeflow Pipelines @@ -181,8 +180,6 @@ finished with them: * Learn more about the [important concepts](/docs/pipelines/concepts/) in Kubeflow Pipelines. -* Follow the [end-to-end tutorial](/docs/gke/pipelines-tutorial/) - using an MNIST machine-learning model. * This page showed you how to run some of the examples supplied in the Kubeflow Pipelines UI. Next, you may want to run a pipeline from a notebook, or compile and run a sample from the code. See the guide to experimenting with diff --git a/content/docs/pipelines/tutorials/cloud-tutorials.md b/content/docs/pipelines/tutorials/cloud-tutorials.md index 47e451d67f..d90d446928 100644 --- a/content/docs/pipelines/tutorials/cloud-tutorials.md +++ b/content/docs/pipelines/tutorials/cloud-tutorials.md @@ -9,5 +9,4 @@ weight = 1 See the guide to the Kubeflow docs.

{{% /alert %}} -* [Pipelines End-to-end on GCP](/docs/gke/pipelines-tutorial/): An end-to-end tutorial for Kubeflow Pipelines on Google Cloud Platform (GCP). * [Pipelines End-to-end on Azure](/docs/azure/azureendtoend/): An end-to-end tutorial for Kubeflow Pipelines on Microsoft Azure. \ No newline at end of file