Squashed commit of the following:

commit 6ec7f6d Author: KAAANG <79990647+SAKURA-CAT@users.noreply.github.com> Date: Thu Feb 22 16:13:17 2024 +0800 record max step and min step (#341) commit ce48cd8 Author: KAAANG <cunykang@gmail.com> Date: Thu Feb 22 15:51:42 2024 +0800 delete Video commit a11a7ed Author: Bote Huang <106975133+KashiwaByte@users.noreply.github.com> Date: Thu Feb 22 15:36:32 2024 +0800 Feat/video chart (#338) * feat:create class Video * feat:add __init__,add video chart * feat:add function for saving video * feat:add function for getting list of video * feat:add function get_data * feat:add function __save and __prepare_video * feat:add function load_from_str and load_from_BytesIO * bugfix: fix __save function * del test file commit 6b5b9c3 Author: KAAANG <79990647+SAKURA-CAT@users.noreply.github.com> Date: Wed Feb 21 15:25:33 2024 +0800 Feat/new line chart (#334) * add type——linear * label formatter * update tick commit 788d8c1 Author: Ze-Yi LIN <58305964+xiaolin199912@users.noreply.github.com> Date: Tue Feb 20 21:07:39 2024 +0800 Feat/image-add-parameters-boxes-masks (#329) * add BoundingBoxes class * update something * Update transfer_logfile_0.1.4.py * add ImageMask class * bugfix commit 65a6378 Author: Ze-Yi LIN <58305964+xiaolin199912@users.noreply.github.com> Date: Mon Feb 19 18:42:11 2024 +0800 Readme v0.2.0 (#326) commit 7b964e9 Author: Zirui Cai <74649535+Feudalman@users.noreply.github.com> Date: Mon Feb 19 18:29:05 2024 +0800 Fixbug/tag sort (#327)
SwanHubX · Feb 22, 2024 · 3aa1108 · 3aa1108
1 parent d1d6f8f
commit 3aa1108
Show file tree

Hide file tree

Showing 27 changed files with 665 additions and 119 deletions.
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -17,7 +17,8 @@
     "run": "core",
     "system": "core",
     "data": "guard",
-    "unit": "test"
+    "unit": "test",
+    "migrate": "Husky"
   },
   "material-icon-theme.files.associations": {
     ".env.mock": "Tune"

diff --git a/README.md b/README.md
@@ -8,10 +8,10 @@ Track and visualize all the pieces of your machine learning pipeline
 </p>
 
 <p align="center">
-  <a href="https://github.com/SwanHubX/SwanLab/stargazers"><img src="https://img.shields.io/github/stars/SwanHubX/SwanLab?style=social" alt= /></a>
+  <a href="https://github.com/SwanHubX/SwanLab/stargazers"><img src="https://img.shields.io/github/stars/SwanHubX/SwanLab?style=social"/></a>
   <a href="https://github.com/SwanHubX/SwanLab/blob/main/LICENSE"><img src="https://img.shields.io/github/license/SwanHubX/SwanLab.svg?color=brightgreen" alt="license"></a>
   <a href="https://github.com/SwanHubX/SwanLab/commits/main"><img src="https://img.shields.io/github/last-commit/SwanHubX/SwanLab" alt="license"></a>
-  <a href="https://pypi.python.org/pypi/swanlab"><img src="https://img.shields.io/pypi/v/swanlab?color=orange" alt= /></a>
+  <a href="https://pypi.python.org/pypi/swanlab"><img src="https://img.shields.io/pypi/v/swanlab?color=orange"/></a>
   <a href="https://pepy.tech/project/swanlab"><img alt="pypi Download" src="https://static.pepy.tech/badge/swanlab/month"></a>
   <a href="https://github.com/SwanHubX/SwanLab/discussions"><img alt="Github Discussion" src="https://img.shields.io/badge/discussions-GitHub-333333?logo=github"></a>
 </p>
@@ -38,24 +38,30 @@ English | <a href="README_zh-hans.md">中文</a>
 
 ## Changelog
 
+[24/02/08] 🔥 Very Big Update! We supported [Image Chart](https://geektechstudio.feishu.cn/wiki/LZFxwTuegiXxPGkhXcpcBUEXnHb)、[Audio Chart](https://geektechstudio.feishu.cn/wiki/SU6mwcVNbixMf1k95KbcZHDCnJe)、Mutil-Experiments Chart and a series of comprehensive optimizations and improvements! Please be sure to upgrade to the latest version via `pip install -U swanlab`.
+
 [24/01/25] 😄 We supported a new Config/Summary table component that supports parameter search. Additionally, we've used new fonts and color schemes.
 
 [24/01/23] 🚨 We use SQLite database and Peewee to replace the previous basic configuration information reading and writing solution by [#114](https://github.com/SwanHubX/SwanLab/issues/114). It's a major change that is highly beneficial for the future of the project, but the drawback is that it's not compatible with old versions (swanlab<=v0.1.4) of log data files. Therefore, if you need to visualize log files generated by older versions, please use [transfer script](script/transfer_logfile_0.1.4.py)
 
 Additionally, we supported export the experiment list as `CSV`, new environment record items `Run path` and `logdir`, added interactive quick copy function, and new API `swanlab.config`.
 
-[24/01/14] 🔥 We supported a new UI, tracking additional environment information, including command, git commit/branch and memory. Additionally, we've added a `logdir` API, allowing developers to set the directory for log files.
-
 [Full Changelog](https://github.com/SwanHubX/SwanLab/releases)
 
 <br>
 
 ## Key Function
 
+- **🚀 Multimedia charts**: log Image/Audio/Video/Text/object3D...
+
+<div align="center">
+  <img src="readme_files/mutilmedia-chart.gif" width="600">
+</div>
+
 - **🧪 Experiments GridView**: compare your key metrics for inspiration faster
 
 <div align="center">
-  <img src="readme_files/experiments-gridView.gif" width="600">
+  <img src="readme_files/experiments-table.png" width="600">
 </div>
 
 - **📊 Charts**: visualize your entire training process
@@ -113,6 +119,7 @@ import swanlab
 swanlab.init(
   # save model inputs and hyperparameters in a swanlab.config object
   config={'learning_rate': 0.01},
+  logdir="./logs",
 )
 
 # Model training code here...
@@ -124,7 +131,7 @@ for epoch in range(1, 20):
 
 3. Third, Run a Dashboard: 
 ```bash
-$ swanlab watch
+$ swanlab watch -l ./logs
 ```
 
 That's it! Open http://127.0.0.1:5092 to view a dashboard of your first SwanLab Experiment.
@@ -141,6 +148,7 @@ Learn how to use SwanLab more effectively by following these use cases:
 | [MNIST](https://github.com/SwanHubX/SwanLab-examples/tree/main/MNIST) | Handwriting recognition based on a plain net and MNIST dataset with pytroch, swanlab. |
 | [Image Classification](https://github.com/SwanHubX/SwanLab-examples/blob/main/Resnet50) | Cat and dog classification based on ResNet50 with pytorch, swanlab and gradio. [Tutorial](https://zhuanlan.zhihu.com/p/676430630). |
 | [Text Generation](https://github.com/SwanHubX/SwanLab-examples/blob/main/Word_language_model) | Text generation based on Word_language_model (RNN/LSTM/GRU/Transformer) |
+| [UIE-Finetune](https://github.com/SwanHubX/SwanLab-examples/tree/main/UIE) | how to use personal data to finetune UIE model and monitor training process through swanlab |
 
 <br>
 

diff --git a/README_zh-hans.md b/README_zh-hans.md
@@ -38,22 +38,28 @@
 
 ## 更新日志
 
+[24/02/08] 🔥 超大更新! 我们支持了[图像图表](https://geektechstudio.feishu.cn/wiki/LZFxwTuegiXxPGkhXcpcBUEXnHb)、[音频图表](https://geektechstudio.feishu.cn/wiki/SU6mwcVNbixMf1k95KbcZHDCnJe)、多实验图表以及一系列全面的优化和改进！可通过 `pip install -U swanlab` 升级到最新版本体验新特性。
+
 [24/01/25] 😄 我们发布了新的Config/Summary表格组件，支持参数搜索。此外我们还使用了新的字体和配色。
 
 [24/01/23] 🚨 我们使用SQLite数据库和Peewee库替代了之前的基础配置信息读写方案（[#114](https://github.com/SwanHubX/SwanLab/issues/114)），这是个极大有利于项目未来的改动，但缺陷是不兼容旧版本（swanlab<=v0.1.4）的日志数据文件，所以如需可视化旧版本产生的日志文件, 请使用[转换脚本](script/transfer_logfile_0.1.4.py)。与此同时，我们增加了支持导出实验列表为CSV，新的环境记录项`Run path`和`logdir`，增加了快捷复制的交互，以及新的API `swanlab.config`。
 
-[24/01/14] 🔥 我们发布了一个新的UI界面，以及跟踪更多环境信息，包括Command、git提交/分支、和机器内存。此外，我们还添加了一个`logdir` API，允许开发人员设置日志文件的目录。
-
 [完整更新日志](https://github.com/SwanHubX/SwanLab/releases)
 
 <br>
 
 ## 核心功能
 
+- **🚀 多媒体图表**: 记录训练中的图像/音频/视频/文本/3D模型...
+
+<div align="center">
+  <img src="readme_files/mutilmedia-chart.gif" width="600">
+</div>
+
 - **🧪 表格视图**: 对比关键指标，更快获得洞见
 
 <div align="center">
-  <img src="readme_files/experiments-gridView.gif" width="600">
+  <img src="readme_files/experiments-table.png" width="600">
 </div>
 
 - **📊 图表视图**: 可视化你的机器学习训练全过程
@@ -110,6 +116,7 @@ import swanlab
 swanlab.init(
   # save model inputs and hyperparameters in a swanlab.config object
   config={'learning_rate': 0.01},
+  logdir="./logs",
 )
 
 # Model training code here...
@@ -121,7 +128,7 @@ for epoch in range(1, 20):
 
 3. 第三步，开启一个SwanLab仪表板: 
 ```bash
-$ swanlab watch
+$ swanlab watch -l ./logs
 ```
 
 就是这样！打开 http://127.0.0.1:5092 ，查看你的第一个SwanLab实验的仪表板。
@@ -138,6 +145,7 @@ $ swanlab watch
 | [MNIST](https://github.com/SwanHubX/SwanLab-examples/tree/main/MNIST) | 基于神经网络的MNIST手写体识别（使用pytorch、swanlab库） |
 | [图像分类](https://github.com/SwanHubX/SwanLab-examples/blob/main/Resnet50) | ResNet50猫狗分类（使用pytorch、swanlab、Gradio库） [图文教程](https://zhuanlan.zhihu.com/p/676430630) |
 | [文本生成](https://github.com/SwanHubX/SwanLab-examples/blob/main/Word_language_model) | 基于自然语言模型的文本生成 (RNN/LSTM/GRU/Transformer) |
+| [微调UIE](https://github.com/SwanHubX/SwanLab-examples/tree/main/UIE) | 如何使用个人数据来微调UIE模型并通过swanlab监控训练过程 |
 
 <br>
 

diff --git a/readme_files/charts-1.gif b/readme_files/charts-1.gif
diff --git a/readme_files/experiments-gridView.gif b/readme_files/experiments-gridView.gif
diff --git a/readme_files/experiments-table.png b/readme_files/experiments-table.png
diff --git a/readme_files/mutilmedia-chart.gif b/readme_files/mutilmedia-chart.gif
diff --git a/script/transfer_logfile_0.1.4.py b/script/transfer_logfile_0.1.4.py
@@ -2,7 +2,6 @@
 # -*- coding: utf-8 -*-
 r"""
 @DATE: 2024-01-25 12:37:08
-@File: test/test_wandb.py
 @IDE: vscode
 @Description:
     Swanlab update script for upgrading from v0.1.4 to v0.1.5, transitioning from file system to database.

diff --git a/swanlab/data/modules/__init__.py b/swanlab/data/modules/__init__.py
@@ -2,12 +2,13 @@
 from .audio import Audio
 from .image import Image
 from .text import Text
+
+# from .video import Video
 from typing import Protocol, Union
 
 
 class FloatConvertible(Protocol):
-    def __float__(self) -> float:
-        ...
+    def __float__(self) -> float: ...
 
 
 DataType = Union[float, FloatConvertible, int, BaseType]
diff --git a/swanlab/data/modules/audio.py b/swanlab/data/modules/audio.py
@@ -32,7 +32,10 @@ class Audio(BaseType):
     """
 
     def __init__(
-        self, data_or_path: Union[str, np.ndarray, List["Audio"]], sample_rate: int = None, caption: str = None
+        self,
+        data_or_path: Union[str, np.ndarray, List["Audio"]],
+        sample_rate: int = None,
+        caption: str = None,
     ):
         """Accept a path to an audio file on a numpу array of audio data."""
         super().__init__(data_or_path)

diff --git a/swanlab/data/modules/chart.py b/swanlab/data/modules/chart.py
@@ -18,5 +18,7 @@ class Chart:
     image = "image", [list, str]
     # 音频类型，list代表一步多音频
     audio = "audio", [list, str]
-    # 文本类型，代表一步多文本
+    # 文本类型，list代表一步多文本
     text = "text", [list, str]
+    # 视频类型 list代表一步多视频
+    video = "video", [list,str]
diff --git a/swanlab/data/modules/image.py b/swanlab/data/modules/image.py
@@ -1,8 +1,9 @@
 import numpy as np
 from PIL import Image as PILImage
 from .base import BaseType
+from .utils_modules import BoundingBoxes, ImageMask
 from ..utils.file import get_file_hash_pil
-from typing import Union, List
+from typing import Union, List, Dict
 from io import BytesIO
 import os
 
@@ -26,12 +27,26 @@ def __init__(
         data_or_path: Union[str, np.ndarray, PILImage.Image, List["Image"]],
         mode: str = "RGB",
         caption: str = None,
+        # boxes: dict = None,
+        # masks: dict = None,
     ):
         super().__init__(data_or_path)
         self.image_data = None
         self.mode = mode
         self.caption = self.__convert_caption(caption)
 
+        # self.boxes = None
+        # self.boxes_total_classes = None
+        # self.masks = None
+        # self.masks_total_classes = None
+
+        # TODO: 等前端支持Boxes和Masks后再开启
+        # if boxes:
+        #     self.boxes, self.boxes_total_classes = self.__convert_boxes(boxes)
+
+        # if masks:
+        #     self.masks, self.masks_total_classes = self.__convert_masks(masks)
+
     def get_data(self):
         # 如果传入的是Image类列表
         if isinstance(self.value, list):
@@ -73,6 +88,46 @@ def __convert_caption(self, caption):
             raise TypeError("caption must be a string, int or float.")
         return caption
 
+    def __convert_boxes(self, boxes):
+        """将boxes转换为Dict[str, BoundingBoxes]类型对象, 并返回该对象和总标签"""
+        # 如果boxes的类型不是字典，则报错
+        if not isinstance(boxes, dict):
+            raise TypeError("swanlab.Image 'boxes' argument must be a dictionary")
+
+        boxes_final: Dict[str, BoundingBoxes] = {}
+        total_classes = {}
+
+        # 对于boxes中的每一个key
+        for key in boxes:
+            box_item = boxes[key]
+            if isinstance(box_item, BoundingBoxes):
+                boxes_final[key] = box_item
+            elif isinstance(box_item, dict):
+                boxes_final[key] = BoundingBoxes(box_item, key)
+            total_classes.update(boxes_final[key]._class_labels)
+
+        return boxes_final, total_classes
+
+    def __convert_masks(self, masks):
+        """将masks转换为Dict[str, ImageMask]类型对象, 并返回该对象和总标签"""
+        # 如果masks的类型不是字典，则报错
+        if not isinstance(masks, dict):
+            raise TypeError("swanlab.Image 'masks' argument must be a dictionary")
+
+        masks_final: Dict[str, ImageMask] = {}
+        total_classes = {}
+
+        # 对于masks中的每一个key
+        for key in masks:
+            mask_item = masks[key]
+            if isinstance(mask_item, ImageMask):
+                masks_final[key] = mask_item
+            elif isinstance(mask_item, dict):
+                masks_final[key] = ImageMask(mask_item, key)
+            total_classes.update(masks_final[key]._class_labels)
+
+        return masks_final, total_classes
+
     def __preprocess(self, data):
         """将不同类型的输入转换为PIL图像"""
         if isinstance(data, str):
@@ -147,13 +202,24 @@ def get_more(self, *args, **kwargs) -> dict:
         if isinstance(self.value, list):
             return self.get_more_list()
         else:
-            return (
-                {
-                    "caption": self.caption,
-                }
-                if self.caption is not None
-                else None
-            )
+            get_more_dict = {}
+
+            if self.caption is not None:
+                get_more_dict["caption"] = self.caption
+
+            # if self.boxes is not None:
+            #     get_more_dict["boxes"] = self.boxes
+
+            # if self.boxes_total_classes is not None:
+            #     get_more_dict["boxes_total_classes"] = self.boxes_total_classes
+
+            # if self.masks is not None:
+            #     get_more_dict["masks"] = self.masks
+
+            # if self.masks_total_classes is not None:
+            #     get_more_dict["masks_total_classes"] = self.masks_total_classes
+
+            return get_more_dict if get_more_dict else None
 
     def get_namespace(self, *args, **kwargs) -> str:
         """设定分组名"""

diff --git a/swanlab/data/modules/utils_modules/__init__.py b/swanlab/data/modules/utils_modules/__init__.py
@@ -0,0 +1,2 @@
+from .bounding_boxes import BoundingBoxes
+from .image_mask import ImageMask
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		from .bounding_boxes import BoundingBoxes
		from .image_mask import ImageMask