Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readmes #63

Merged
merged 4 commits into from
Sep 14, 2021
Merged

Readmes #63

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ce82896
Update README.md
thibo73800 Sep 14, 2021
5ab2502
Update README.md
thibo73800 Sep 14, 2021
cf5c8e3
Update README.md
thibo73800 Sep 14, 2021
1ed49ae
Update README.md
thibo73800 Sep 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 74 additions & 76 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,50 @@
**Aloception** is a set of packages for computer vision built on top of popular deep learning libraries:
[pytorch](<https://pytorch.org/>) and [pytorch lightnig](https://www.pytorchlightning.ai/).


**Aloscene** extend the use of
[tensors](https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html) with **Augmented Tensors** designed to facilitate the use of computer vision data
(such as frames, 2d boxes, 3d boxes, optical flow, disparity, camera parameters...).
[tensors](https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html) with **Augmented Tensors** designed to facilitate the use of computer vision data
(such as frames, 2d boxes, 3d boxes, optical flow, disparity, camera parameters...).


```python
frame = aloscene.Frame("/path/to/image.jpg")
frame = frame.to("cpu")
frame.get_view().render()
```

**Alodataset** implement ready-to-use datasets for computer vision with the help of **aloscene** and **augmented tensors** to make it easier to transform and display your vision data.

```python
coco_dataset = alodataset.CocoDetectionDataset(sample=True)
for frame in coco_dataset.stream_loader():
frame.get_view().render()
```

**Alonet** integrates several promising computer vision architectures. You can use it for research purposes or to finetune and deploy your model using TensorRT. Alonet is mainly built on top of [ lightnig](https://www.pytorchlightning.ai/) with the help of
**aloscene** and **alodataset**.

```python
# Load model
model = alonet.detr.DetrR50(num_classes=91, weights="detr-r50").eval()

# Open and normalized frame
frame = aloscene.Frame("/path/to/image.jpg").norm_resnet()

# Run inference
pred_boxes = model.inference(model([frame]))

**NOTE**
# Add and display the predicted boxes
frame.append_boxes2d(pred_boxes[0], "pred_boxes")
frame.get_view().render()
```

One can use **aloscene** independently than the two other packages to handle computer vision data, or to improve its
training pipelines with **augmented tensors**.

### Note
One can use **aloscene** independently than the two other packages to handle computer vision data, or to improve its
training pipelines with **augmented tensors**.

# Install
## Install

Aloception's packages are built on top of multiple libraries. Most of them are listed in the **requirements.txt**
```
Expand All @@ -39,10 +62,30 @@ pip install -r requirements.txt
Once the others packages are installed, you still need to install pytorch based on your hardware and environment
configuration. Please, ref to the `pytorch website <https://pytorch.org/>`_ for this install.

## Getting started

# Alonet
<ul>
<li><a href="https://visual-behavior.github.io/aloception/getting_started/getting_started.html">Getting started</a></li>
<li><a href="https://visual-behavior.github.io/aloception/getting_started/aloscene.html">Aloscene: Computer vision with ease</a></li>
<li><a href="https://visual-behavior.github.io/aloception/getting_started/alodataset.html">Alodataset: Loading your vision datasets</a></li>
<li><a href="https://visual-behavior.github.io/aloception/getting_started/alonet.html">Alonet: Loading & training your models</a></li>
<li><a href="https://visual-behavior.github.io/aloception/getting_started/augmented_tensor.html">About augmented tensors</a></li>
</ul>


## Tutorials

<ul>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/data_setup.html">How to setup your data?</a></li>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/training_detr.html">Training Detr</a></li>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/finetuning_detr.html">Finetuning DETR</a></li>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/training_deformable_detr.html">Training Deformable DETR</a></li>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/finetuning_deformable_detr.html">Finetuning Deformanble DETR</a></li>
<li><a href="https://visual-behavior.github.io/aloception/tutorials/tensort_inference.html">Exporting DETR / Deformable-DETR to TensorRT</a></li>
</ul>

# Alonet

## Models

| Model name | Link | alonet location | Learn more
Expand All @@ -54,89 +97,44 @@ configuration. Please, ref to the `pytorch website <https://pytorch.org/>`_ for

## Detr

### Running inference with detr_r50
Here is a simple example to get started with **Detr** and aloception. To learn more about Detr, you can checkout the <a href="#tutorials">Tutorials<a/> or the <a href="./alonet/detr">detr README</a>.

```
python alonet/detr/detr_r50.py /path/to/image.jpg
```
```python
# Load model
model = alonet.detr.DetrR50(num_classes=91, weights="detr-r50").eval()

### Training Detr from scratch
```
python alonet/detr/train_on_coco.py
```
# Open and normalized frame
frame = aloscene.Frame("/path/to/image.jpg").norm_resnet()

### Running evaluation of detr-r50

```
python alonet/detr/eval_on_coco.py --weights detr-r50 --batch_size 1
```
# Run inference
pred_boxes = model.inference(model([frame]))

# Add and display the predicted boxes
frame.append_boxes2d(pred_boxes[0], "pred_boxes")
frame.get_view().render()
```
| all | .50 | .55 | .60 | .65 | .70 | .75 | .80 | .85 | .90 | .95 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
box | 39.93 | 58.80 | 56.57 | 54.00 | 50.99 | 47.06 | 42.17 | 36.12 | 28.74 | 18.62 | 6.25 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
```


## Deformable Detr

### Running inference with deformable detr
Here is a simple example to get started with **Deformable Detr** and aloception. To learn more about Deformable, you can checkout the <a href="#tutorials">Tutorials<a/> or the <a href="./alonet/deformable_detr">deformable detr README</a>.

```
python alonet/deformable_detr/deformable_detr_r50.py /path/to/image.jpg
python alonet/deformable_detr/deformable_detr_r50_refinement.py /path/to/image.jpg
```
```python
# Loading Deformable model
model = alonet.deformable_detr.DeformableDetrR50(num_classes=91, weights="deformable-detr-r50").eval()

### Training Deformable Detr R50 from scratch
```
python alonet/deformable_detr/train_on_coco.py --model_name MODEL_NAME
```
With MODEL_NAME either deformable-detr-r50-refinement or deformable-detr-r50 for with/without box refinement.
# Open, normalize frame and send frame on the device
frame = aloscene.Frame("/home/thibault/Desktop/yoga.jpg").norm_resnet().to(torch.device("cuda"))

### Running evaluation of Deformable Detr R50
```
python alonet/deformable_detr/eval_on_coco.py --model_name MODEL_NAME --weights MODEL_NAME --batch_size 1 [--ap_limit NUMBER_OF_SAMPLES]
```
With MODEL_NAME either deformable-detr-r50-refinement or deformable-detr-r50 for with/without box refinement.
# Run inference
pred_boxes = model.inference(model([frame]))

Evaluation on 1000 images COCO with box refinement
```
| all | .50 | .55 | .60 | .65 | .70 | .75 | .80 | .85 | .90 | .95 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
box | 44.93 | 62.24 | 60.26 | 58.00 | 55.76 | 52.51 | 48.07 | 42.99 | 36.13 | 24.28 | 9.02 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
# Add and display the predicted boxes
frame.append_boxes2d(pred_boxes[0], "pred_boxes")
frame.get_view().render()
```

## RAFT

### Running inference with RAFT

Inference can be performed using official pretrained weights:
```
python alonet/raft/raft.py --weights=raft-things image1.png img2.png
```
or with a custom weight file:
```
python alonet/raft/raft.py --weights=path/to/weights.pth image1.png img2.png
```
### Running evaluation of RAFT
RAFT model is evaluated with official weights "raft-things".

The **EPE** score is computed on Sintel training set. This reproduces the results from RAFT paper : table 1 (training data:"C+T", "method:our(2-views), eval:"sintel training clean).

```
python alonet/raft/eval_on_sintel.py
```

This results in EPE=1.46, which is similar to 1.43 in the paper (obtained as a median score for models trained with 3 differents seeds).

### Training RAFT from scratch
To reproduce the first stage of RAFT training on FlyingChairs dataset:
```
python alonet/raft/train_on_chairs.py
```

It is possible to reproduce the full RAFT training or to train on custom dataset by creating the necessary dataset classes and following the same approach as in `train_on_chairs.py`

# Alodataset

Expand Down