D2Go - Use Detectron2 on cellular devices

D2Go is a production-equipped software system from FacebookResearch, which supports end-to-stop model training and deployment for cellular structures.

D2Go

D2Go is a manufacturing-ready software machine from FacebookResearch, which supports end-to-give up version schooling and deployment for mobile structures. D2Go presents both built-in command-line tools and an API. This README will stroll you thru how you can use both the CLI and API to:

Training a custom version

Exporting a model to Torchscript

Quantization-conscious Training

Installation

Install PyTorch Nightly (use CUDA 10.2 as an instance, see info at PyTorch Website):

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly

Install Detectron2 (other installation options at Detectron2):

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

Install mobile_cv:

python -m pip install 'git+https://github.com/facebookresearch/mobile-vision.git'

Install d2go:

git clone https://github.com/facebookresearch/d2go
cd d2go & python -m pip install .

For greater information, test out the set up segment of the D2Go README.

Command-line

Inference with a pre-skilled version

To run inference with a pre-skilled version, navigate into the demo directory, choose a version from model_zoo, e.G. Faster_rcnn_fbnetv3a_C4.Yaml, and then run the demo.Py script.

usage: demo.py [-h] [--config-file FILE] [--webcam] [--video-input VIDEO_INPUT] [--input INPUT [INPUT ...]] [--output OUTPUT] [--confidence-threshold CONFIDENCE_THRESHOLD] [--opts ...]

Detectron2 demo for builtin configs

optional arguments:
  -h, --help            show this help message and exit
  --config-file FILE    path to config file
  --webcam              Take inputs from webcam.
  --video-input VIDEO_INPUT
                        Path to video file.
  --input INPUT [INPUT ...]
                        A list of space separated input images; or a single glob pattern such as 'directory/*.jpg'
  --output OUTPUT       A file or directory to save output visualizations. If not given, will show output in an OpenCV window.
  --confidence-threshold CONFIDENCE_THRESHOLD
                        Minimum score for instance predictions to be shown
  --opts ...            Modify config options using the command-line 'KEY VALUE' pairs

Example usage:

wget http://images.cocodataset.org/val2017/000000439715.jpg -q -O input.jpg
python demo.py --config-file faster_rcnn_fbnetv3a_C4.yaml --input input.jpg --output output.jpg

Training a custom model

The CLI additionally helps schooling and evaluating custom fashions, but as some distance as I realize, it most effective helps the built-in records-sets, along with:

• COCO
• LVIS Instance Segmentation
• Cityscapes

• ...

Before schooling, you want to installation the built-in facts-sets. For greater information, please follow the instructions on the 'Use Builtin Datasets' page.

To teach a version run:

d2go.train_net --config-file ./configs/faster_rcnn_fbnetv3a_C4.yaml

To evaluate a model checkpoint run:

d2go.train_net --config-file ./configs/faster_rcnn_fbnetv3a_C4.yaml --eval-only \
MODEL.WEIGHTS <path to weights>

Exporting to Torchscript & Int8 Model

You can export a skilled version to Torchscript the usage of the d2go.Exporter.

Example utilization:

d2go.exporter --config-file configs/faster_rcnn_fbnetv3a_C4.yaml \
--predictor-types torchscript --output-dir ./ \
MODEL.WEIGHTS https://mobile-cv.s3-us-west-2.amazonaws.com/d2go/models/246823121/model_0479999.pth

The exporter additionally helps put up-education quantization:

d2go.exporter --config-file configs/faster_rcnn_fbnetv3a_C4.yaml \
--output-dir ./ --predictor-type torchscript_int8 \
MODEL.WEIGHTS https://mobile-cv.s3-us-west-2.amazonaws.com/d2go/models/246823121/model_0479999.pth

Quantization-conscious Training

To run quantization-conscious training, you want to feature the QUANTIZATION section in your model configuration.

d2go.train_net --config-file configs/qat_faster_rcnn_fbnetv3a_C4.yaml \
MODEL.WEIGHTS https://mobile-cv.s3-us-west-2.amazonaws.com/d2go/models/246823121/model_0479999.pth

API

D2Go also gives an API, which allows you to use, teach and export your models with a syntax very much like simple Detectron2. As with Detectron2, I recommend using Google Colab for schooling the version as Google Colab makes it incredible easy to get setup and start coding.

Inference with a pre-trained version

To use a pre-trained version for inference, load it from the model_zoo the use of the get approach and create a DemoPredictor.

import cv2
from matplotlib import pyplot as plt

from d2go.model_zoo import model_zoo
from d2go.utils.demo_predictor import DemoPredictor

# get model
model = model_zoo.get('faster_rcnn_fbnetv3a_C4.yaml', trained=True)

# get image
!wget http://images.cocodataset.org/val2017/000000439715.jpg -q -O input.jpg
im = cv2.imread("./input.jpg")

# Create predictor
predictor = DemoPredictor(model)

# Make prediction
outputs = predictor(im)

The predictions may be displayed the use of the Visualizer.

from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog

v = Visualizer(im[:, :, ::-1], MetadataCatalog.get("coco_2017_train"))
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
plt.imshow(out.get_image()[:, :, ::-1])

You can find all the to be had models on the "D2Go Model Zoo and Baselines" web page.

Training a custom model

Training a custom model with D2Go is nearly identical to the usage of simple Detectron2. First, we need to register the records-set, following the detectron2 custom dataset educational.

For the ballon segmentation statistics-set this appears as follows:

# download, decompress the data
!wget https://github.com/matterport/Mask_RCNN/releases/download/v2.1/balloon_dataset.zip
!unzip balloon_dataset.zip > /dev/null

Python

# if your dataset is in COCO format, this cell can be replaced by the following three lines:
# from detectron2.data.datasets import register_coco_instances
# register_coco_instances("my_dataset_train", {}, "json_annotation_train.json", "path/to/image/dir")
# register_coco_instances("my_dataset_val", {}, "json_annotation_val.json", "path/to/image/dir")
import os
import json
import numpy as np
from detectron2.structures import BoxMode

def get_balloon_dicts(img_dir):
    json_file = os.path.join(img_dir, "via_region_data.json")
    with open(json_file) as f:
        imgs_anns = json.load(f)

    dataset_dicts = []
    for idx, v in enumerate(imgs_anns.values()):
        record = {}
        
        filename = os.path.join(img_dir, v["filename"])
        height, width = cv2.imread(filename).shape[:2]
        
        record["file_name"] = filename
        record["image_id"] = idx
        record["height"] = height
        record["width"] = width
      
        annos = v["regions"]
        objs = []
        for _, anno in annos.items():
            assert not anno["region_attributes"]
            anno = anno["shape_attributes"]
            px = anno["all_points_x"]
            py = anno["all_points_y"]
            poly = [(x + 0.5, y + 0.5) for x, y in zip(px, py)]
            poly = [p for x in poly for p in x]

            obj = {
                "bbox": [np.min(px), np.min(py), np.max(px), np.max(py)],
                "bbox_mode": BoxMode.XYXY_ABS,
                "segmentation": [poly],
                "category_id": 0,
            }
            objs.append(obj)
        record["annotations"] = objs
        dataset_dicts.append(record)
    return dataset_dicts

for d in ["train", "val"]:
    DatasetCatalog.register("balloon_" + d, lambda d=d: get_balloon_dicts("balloon/" + d))
    MetadataCatalog.get("balloon_" + d).set(thing_classes=["balloon"], evaluator_type="coco")
balloon_metadata = MetadataCatalog.get("balloon_train")

To confirm the statistics loading is correct, permit's visualize the annotations of randomly selected samples inside the training set:

import random

dataset_dicts = DatasetCatalog.get('balloon_train')
for d in random.sample(dataset_dicts, 3):
    img = cv2.imread(d["file_name"])
    v = Visualizer(img[:, :, ::-1], metadata=balloon_metadata, scale=0.5)
    v = v.draw_dataset_dict(d)
    plt.figure(figsize = (14, 10))
    plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))
    plt.show()

After verifying that the information loaded efficiently, we can high-quality-music a COCO-pretrained FBNetV3A Mask R-CNN model at the balloon dataset. For this, we want to adjust the default faster_rcnn_fbnetv3a_C4.Yaml config to work with the balloon records-set.

from d2go.runner import GeneralizedRCNNRunner

def prepare_for_launch():
    runner = GeneralizedRCNNRunner()
    cfg = runner.get_default_cfg()
    cfg.merge_from_file(model_zoo.get_config_file("faster_rcnn_fbnetv3a_C4.yaml"))
    cfg.MODEL_EMA.ENABLED = False
    cfg.DATASETS.TRAIN = ("balloon_train",)
    cfg.DATASETS.TEST = ("balloon_val",)
    cfg.DATALOADER.NUM_WORKERS = 2
    cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("faster_rcnn_fbnetv3a_C4.yaml")  # Let training initialize from model zoo
    cfg.SOLVER.IMS_PER_BATCH = 2
    cfg.SOLVER.BASE_LR = 0.00025  # pick a good LR
    cfg.SOLVER.MAX_ITER = 600    # 600 iterations seems good enough for this toy dataset; you will need to train longer for a practical dataset
    cfg.SOLVER.STEPS = []        # do not decay learning rate
    cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset (default: 512)
    cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (ballon). (see https://detectron2.readthedocs.io/tutorials/datasets.html#update-the-config-for-new-datasets)
    # NOTE: this config means the number of classes, but a few popular unofficial tutorials incorrect uses num_classes+1 here.
    os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
    return cfg, runner

cfg, runner = prepare_for_launch()

Now that we have created a config for the ballon facts-set, we will create a version and begin education.

Exporting to Torchscript & Int8 Model

To export the educated version, we want to name the convert_and_export_predictor approach, passing it the config, model, and a dataloader.

import copy
from detectron2.data import build_detection_test_loader
from d2go.export.api import convert_and_export_predictor
from d2go.export.d2_meta_arch import patch_d2_meta_arch

import logging

# disable all the warnings
previous_level = logging.root.manager.disable
logging.disable(logging.INFO)

patch_d2_meta_arch()

pytorch_model = runner.build_model(cfg, eval_only=True)
pytorch_model.cpu()

datasets = cfg.DATASETS.TRAIN[0]
data_loader = runner.build_detection_test_loader(cfg, datasets)

predictor_path = convert_and_export_predictor(
  copy.deepcopy(cfg),
  copy.deepcopy(pytorch_model),
  "torchscript_int8@tracing",
  './',
  data_loader
)

# recover the logging level
logging.disable(previous_level)

After changing the model, we can test it out by growing a predictor.

from mobile_cv.predictor.api import create_predictor
from d2go.utils.demo_predictor import DemoPredictor

model = create_predictor(predictor_path)
predictor = DemoPredictor(model)

dataset_dicts = DatasetCatalog.get('balloon_val')
for d in random.sample(dataset_dicts, 3):    
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1], metadata=balloon_metadata, scale=0.8)
    v = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    plt.figure(figsize = (14, 10))
    plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))
    plt.show()

Deploying model on Android

D2Go may be used on Android the usage of the native torchvision Ops library. The D2Go crew gives a D2Go Android demo app that suggests how to prepare and use D2Go fashions on Android. The code is based totally on a preceding PyTorch Android Object Detection demo app that makes use of a pre-educated YOLOv5 model, with changed pre-processing and submit-processing code required through the D2Go version.

Below I'll stroll you thru the needed steps to get the Android demo work. This could be very much like the README from the demo itself, but it incorporates a few adjustments I had to make to get the demo to work on my machine.

1. Install PyTorch 1.8.0 and torchvision 0.9.0, for example:

conda create -n d2go python=3.8.5
conda activate d2go
pip install torch torchvision

2. Install Detectron2, mobile_cv, and D2Go

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -m pip install 'git+https://github.com/facebookresearch/mobile-vision.git'
git clone https://github.com/facebookresearch/d2go
cd d2go & python -m pip install .

3. Create the D2Go model

git clone https://github.com/pytorch/android-demo-app
cd android-demo-app/D2Go
python create_d2go.py

This will create the quantized D2Go version and shop it at android-demo-app/D2Go/ObjectDetection/app/src/most important/belongings/d2go.Pt.

The length of the quantized D2Go model is handiest 2.6MB.

4. Build and run the D2GO

In Android Studio, open android-demo-app/D2Go (now not android-demo-app/D2Go/ObjectDetection). If an mistakes "Gradle’s dependency can be corrupt" takes place, visit Android Studio - File - Project Structure... , change the Android Gradle Plugin Version to four.0.1, and the Gradle Version to 4.10.1.

After selecting the suitable Gradle model, the project must construct without any errors. Once the construct is achieved, you could install the model for your Android tool or create an Android emulator.

Result:

Deploying a custom version

To use a custom version with the Android demo, the version first desires to br converted, as shown in the create_d2go.Py report.

from typing import List, Dict
import torch

class Wrapper(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
        coco_idx_list = [1]

        self.coco_idx = torch.tensor(coco_idx_list)

    def forward(self, inputs: List[torch.Tensor]):
        x = inputs[0].unsqueeze(0) * 255
        scale = 320.0 / min(x.shape[-2], x.shape[-1])
        x = torch.nn.functional.interpolate(x, scale_factor=scale, mode="bilinear", align_corners=True, recompute_scale_factor=True)
        out = self.model(x[0])
        res : Dict[str, torch.Tensor] = {}
        res["boxes"] = out[0] / scale
        res["labels"] = torch.index_select(self.coco_idx, 0, out[1])
        res["scores"] = out[2]
        return inputs, [res]

orig_model = torch.jit.load(os.path.join(predictor_path, "model.jit"))
wrapped_model = Wrapper(orig_model)
scripted_model = torch.jit.script(wrapped_model)
scripted_model.save("d2go.pt")

Now that the model was converted and saved as d2go.Pt, you want to replace the d2go.Pt report in the android-demo-app/D2Go/ObjectDetection/app/src/important/assets/ folder along with your custom version and adjust the lessons.Txt report to include your custom lessons.

Related Articles :

1. Learn Python
2. Python Inside The Cloud
3. Python In Excel